Dinosaurs are Forever

Friday, June 6, 2014

using kgraft with ubuntu

New live kernel patching projects have hit LKML recently [1][2], and I've taken the opportunity to test drive kGraft with the Ubuntu kernel. This post documents how to get a sample patch working.

A Simple Example

First, I had to take the patches from [3] and apply them against the ubuntu-utopic kernel, which is based on 3.15-rc8 as of this post. They cherry-picked cleanly and the branch I'm using is stored here [4]. In addition to applying the patches I had to also enable CONFIG_KGRAFT. A pre-built test kernel can be downloaded here [5].

Next, I created a test VM and installed the test kernel, headers, and build dependencies into that VM and rebooted. Now after a successful reboot, we need to produce an actual patch to test. I've created a github project [6] with the sample patch; to make it easy to clone and get started.

sudo apt-get install git build-essential

git clone https://github.com/arges/kgraft-examples.git
cd kgraft-examples
make

The code in kgraft_patcher.c is the example found in samples/kgraft [7]. Now we can build it easily using the Makefile I have in my project by typing make.

Next, the module needs to be inserted using the following:

sudo insmod ./kgraft_patcher.ko

Run the following to see if the module loaded properly:

lsmod | grep kgraft

You'll notice some messages printed with the following:

[  211.762563] kgraft_patcher: module verification failed: signature and/or  required key missing - tainting kernel
[  216.800080] kgr failed after timeout (30), still in degraded mode
[  246.880146] kgr failed after timeout (30), still in degraded mode
[  276.960211] kgr failed after timeout (30), still in degraded mode

This means that not all processes have entered the kernel and may not have a "new universe" flag set. Run the following to see which processes still needs to be updated.

cat /proc/*/kgr_in_progress

In order to get all processes to enter the kernel sometimes a signal needs to be sent to get the process to enter the kernel.

An example of this is found in the kgraft-examples [6] called 'hurryup.sh':

#!/bin/bash
for p in $(ls /proc/ | grep '^[0-9]'); do
  if [[ -e /proc/$p/kgr_in_progress ]]; then
    if [[ `sudo cat /proc/$p/kgr_in_progress` -eq 1 ]]; then
     echo $p;
     sudo kill -SIGCONT $p
    fi
  fi
done

Here is checks for all processes that have 'kgr_in_progress' set and sends a SIGCONT signal to that process.

I've noticed that I had to also send a SIGSTOP followed by a SIGCONT to finally get everything synced up.

Eventually you'll see:

[ 1600.480233] kgr succeeded

Now your kernel is running the new patch without rebooting!

References

Monday, March 24, 2014

netfilter hooks to snoop packets

I've written a short netfilter hook to help snoop some outgoing packets. You can view the source code here [1]. I read a few articles [2] [3] [4] to get an idea of how to put things together.

Wednesday, March 5, 2014

trusty release party

Come celebrate the release [1] of the next Ubuntu LTS 'Trusty Tahr' 14.04 in Austin!

Wednesday, April, 16th, 5pm @ the Flying Saucer

So majestic!

creating a local vm easily and automatically

This is the nifty little snippet I use to create a quick and dirty VM for testing. Just apt-get install vmbuilder and run the following:

NAME="ubuntu"
vmbuilder kvm ubuntu --arch 'amd64' --suite 'precise' \
--rootsize '8096' \
--mem '1024' \
--components 'main,universe' \
--addpkg vim  \
--addpkg openssh-server  \
--addpkg bash-completion  \
--user 'ubuntu'  --pass 'ubuntu' \
-d /var/lib/libvirt/images/${NAME} \
--libvirt qemu:///system \
-o \
-v --debug \
--hostname ${NAME}

If you need additional help consult:
https://help.ubuntu.com/12.04/serverguide/jeos-and-vmbuilder.html

using LTS HWE kernels with MAAS nodes

Once you get machines setup using MAAS [1], you may want to be able to install an alternative kernel when starting a machine. Here's how to do it.

This is assuming we have already commissioned a node, but have not started it.

If we are using the _normal_ installer with MAAS do the following:
Edit /etc/maas/preseeds/preseed_master on your maas-server

Add the second line as shown:

d-i base-installer/kernel/image string linux-server
d-i base-installer/kernel/override-image string linux-generic-lts-saucy

Start the node,
Now when you boot you should be using the 3.11 series kernel.

Monday, February 17, 2014

creating a local vm domain from an ubuntu cloud image

It is very convenient to be able to quickly spin up a local VM in order to do some sort of bug verification or testing. Generally I can use a cloud provider to spin up an instance, but occasionally you need the flexibility of a local VM in order to setup more complex networking or other specific environments.

This link [1] was extremely helpful in figuring our how to setup a ready to boot image, but I noticed a few issues. First, on Precise I didn't have the cloud-localds tool, and second there wasn't a very simple way to import the image into libvirt.

The cloud-localds problem was solved easily by branching the latest copy and using it locally. Virt-install can be easily used to create a domain based on the image.

Below is a snippet to be run after using Scott's original script:

# install cloud image
sudo virt-install --connect qemu:///system \
  --ram 1024 -n ubuntu --os-type=linux --os-variant=ubuntuprecise \
  --disk path=./disk.img,device=disk,bus=virtio,format=qcow2 \
  --disk path=./my-seed.img,bus=virtio,format=raw \
  --vcpus=1 --vnc --noautoconsole --import

http://ubuntu-smoser.blogspot.com/2013/02/using-ubuntu-cloud-images-without-cloud.html

Monday, December 16, 2013

automatic bisection of qemu with console and monitor interaction

I had some fun debugging a performance issue after VM migration on v1.0 [1]. Because this was fixed in newer versions, a bisect was in order. Doing this manually becomes very tedious so writing some scripts to use 'git bisect run' really helps. Here I'll document how I did this in case others find it useful when trying to bisect issues in qemu.

First step is getting the problem reproducible using the command line. In addition you will also need to output anything that be used to determine if the test case passes or fails using standard console output. This makes it easy to run the test case with expect. You will need to follow [2] on how to setup your VM to use serial output.

Next modify the expect script and ensure it works by just running it by itself. Identify which versions pass or fail (a coarse bisect). Once you can get between two release tags you should start a bisect between those tags.

Next run:

git bisect start
git bisect good <good tag>
git bisect bad <bad tag>
git bisect run ../bisect-run

I ran into a few gotchas that may be bugs that really need fixing. Occasionally when running in '-noconsole' mode, I wouldn't see any prompt for a very long time. When re-running with '-vga std', I'd see that it was waiting at GRUB. You may need to modify timeouts such that you don't hit those issues without a VGA console.

Overall, you can find these scripts here [3].

Hopefully they will evolve a bit once its used more and more.