using ktest.pl with ubuntu

Bisecting the kernel is one of those tasks that's time-consuming and error prone. Ktest.pl is a script that lives in the linux kernel source tree [1] that helps to automate this process. The script is extremely extensible and as such takes times to understand which variables need to be set and where. In this post, I'll go over how to perform a kernel bisection using a VM as the target machine. In this example I'm using 'ubuntu' as the VM name.

First ensure you have all dependencies correctly setup:

sudo apt-get install libvirt-bin qemu-kvm cpu-checker virtinst uvtool git
sudo apt-get build-dep linux-image-`uname -r`

Ensure kvm works:

kvm-ok

In this example we are using uvtool to create VMs using cloud images, but you could just as easily use a preseed install or a manual install via an ISO.

First sync the cloud image:

uvt-simplestreams-libvirt sync release=trusty arch=amd64

Clone the necessary git directory:

git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git linux.git

Copy ktest.pl outside of the linux kernel (since bisecting it also changes the script, this way it remains constant):

cd linux
cp tools/testing/ktest/ktest.pl
cp -r tools/testing/ktest/examples/include ..
cd ..

Create directories for script:

mkdir configs build
mkdir configs/ubuntu build/ubuntu

Get an appropriate config for the bisect you are using and ensure it can reasonable 'make oldconfig' with the kernel version you are using. For example, if we are bisecting v3.4 kernels, we can use an Ubuntu v3.5 series kernel config and yes '' | make oldconfig to ensure it is very close. Put this config into configs/ubuntu/config-min. For convenience I have a put a config that works here for this example:
http://people.canonical.com/~arges/amd64-config.flavour.generic

Create the VM, ensure you have ssh keys setup on your local machine first:

uvt-kvm create ubuntu release=trusty arch=amd64 --password ubuntu --unsafe-caching

Ensure the VM can be ssh'ed to via 'ssh ubuntu@ubuntu':

echo "$(uvt-kvm ip ubuntu) ubuntu" | sudo tee -a /etc/hosts

SSH into VM with ssh ubuntu@ubuntu.

Set up the initial target kernel to boot on the VM:

sudo cp /boot/vmlinuz-`uname -r` /boot/vmlinuz-test
sudo cp /boot/initrd.img-`uname -r` /boot/initrd.img-test

Ensure SUBMENUs are disabled on the VM, as the grub2 detection script in ktest.pl fails with submenus, and update grub.

echo "GRUB_DISABLE_SUBMENU=y" | sudo tee -a /etc/default/grub
sudo update-grub

Ensure we have a serial console on the VM with /etc/init/ttyS0.conf, and ensure that agetty automatically logs in as root. If you ran with the above script you can do the following:

sudo sed -i 's/exec \/sbin\/getty/exec \/sbin\/getty -a root/' /etc/init/ttyS0.conf

Ensure that /root/.ssh/authorized_keys on the VM contains the host keys so that ssh root@ubuntu works automatically. If you are using the above commands you can do:

sudo sed -i 's/^.*ssh-rsa/ssh-rsa/g' /root/.ssh/authorized_keys

Finally add a test case to /home/ubuntu/test.sh inside of the ubuntu VM. Ensure it is executable.

#!/bin/bash
# Make a unique string
STRING=$(cat /dev/urandom | tr -dc 'a-zA-Z0-9' | head -c 32)
> /var/log/syslog
echo $STRING > /dev/kmsg
# Wait for things to settle down...
sleep 5
grep $STRING /var/log/syslog
# This should return 0.

Now exit out of the machine and create the following configuration file for ktest.pl called ubuntu.conf. This will bisect from v3.4 (good) to v3.5-rc1 (bad), and run the test case that we put into the VM.

# Setup default machine
MACHINE = ubuntu

# Use virsh to read the serial console of the guest
CONSOLE = virsh console ${MACHINE}
CLOSE_CONSOLE_SIGNAL = KILL

# Include defaults from upstream
INCLUDE include/defaults.conf
DEFAULTS OVERRIDE

# Make sure we load up our machine to speed up builds
BUILD_OPTIONS = -j8

# This is required for restarting VMs
POWER_CYCLE = virsh destroy ${MACHINE}; sleep 5; virsh start ${MACHINE}

# Use the defaults that update-grub spits out
GRUB_FILE = /boot/grub/grub.cfg
GRUB_MENU = Ubuntu, with Linux test
GRUB_REBOOT = grub-reboot
REBOOT_TYPE = grub2

DEFAULTS

# Do a simple bisect
TEST_START
RUN_TEST = ${SSH} /home/ubuntu/test.sh
TEST_TYPE = bisect
BISECT_GOOD = v3.4
BISECT_BAD = v3.5-rc1
CHECKOUT = origin/master
BISECT_TYPE = test
TEST = ${RUN_TEST}
BISECT_CHECK = 1

Now we should be ready to run the bisection (this will take many, many hours depending on the speed of your machine):

./ktest.pl ubuntu.conf

1. http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/tools/testing/ktest?id=HEAD