Showing posts with label kernel. Show all posts
Showing posts with label kernel. Show all posts

Tuesday, July 1, 2014

using ktest.pl with ubuntu

Bisecting the kernel is one of those tasks that's time-consuming and error prone. Ktest.pl is a script that lives in the linux kernel source tree [1] that helps to automate this process. The script is extremely extensible and as such takes times to understand which variables need to be set and where. In this post, I'll go over how to perform a kernel bisection using a VM as the target machine. In this example I'm using 'ubuntu' as the VM name.

First ensure you have all dependencies correctly setup:
sudo apt-get install libvirt-bin qemu-kvm cpu-checker virtinst uvtool git
sudo apt-get build-dep linux-image-`uname -r`

Ensure kvm works:
kvm-ok

In this example we are using uvtool to create VMs using cloud images, but you could just as easily use a preseed install or a manual install via an ISO.
First sync the cloud image:
uvt-simplestreams-libvirt sync release=trusty arch=amd64

Clone the necessary git directory:
git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git linux.git

Copy ktest.pl outside of the linux kernel (since bisecting it also changes the script, this way it remains constant):
cd linux
cp tools/testing/ktest/ktest.pl
cp -r tools/testing/ktest/examples/include ..
cd ..

Create directories for script:
mkdir configs build
mkdir configs/ubuntu build/ubuntu

Get an appropriate config for the bisect you are using and ensure it can reasonable 'make oldconfig' with the kernel version you are using. For example, if we are bisecting v3.4 kernels, we can use an Ubuntu v3.5 series kernel config and yes '' | make oldconfig to ensure it is very close. Put this config into configs/ubuntu/config-min. For convenience I have a put a config that works here for this example:
http://people.canonical.com/~arges/amd64-config.flavour.generic

Create the VM, ensure you have ssh keys setup on your local machine first:
uvt-kvm create ubuntu release=trusty arch=amd64 --password ubuntu --unsafe-caching

Ensure the VM can be ssh'ed to via 'ssh ubuntu@ubuntu':
echo "$(uvt-kvm ip ubuntu) ubuntu" | sudo tee -a /etc/hosts

SSH into VM with ssh ubuntu@ubuntu.

Set up the initial target kernel to boot on the VM:
sudo cp /boot/vmlinuz-`uname -r` /boot/vmlinuz-test
sudo cp /boot/initrd.img-`uname -r` /boot/initrd.img-test

Ensure SUBMENUs are disabled on the VM, as the grub2 detection script in ktest.pl fails with submenus, and update grub.
echo "GRUB_DISABLE_SUBMENU=y" | sudo tee -a /etc/default/grub
sudo update-grub

Ensure we have a serial console on the VM with /etc/init/ttyS0.conf, and ensure that agetty automatically logs in as root. If you ran with the above script you can do the following:
sudo sed -i 's/exec \/sbin\/getty/exec \/sbin\/getty -a root/' /etc/init/ttyS0.conf

Ensure that /root/.ssh/authorized_keys on the VM contains the host keys so that ssh root@ubuntu works automatically. If you are using the above commands you can do:
sudo sed -i 's/^.*ssh-rsa/ssh-rsa/g' /root/.ssh/authorized_keys

Finally add a test case to /home/ubuntu/test.sh inside of the ubuntu VM. Ensure it is executable.
#!/bin/bash
# Make a unique string
STRING=$(cat /dev/urandom | tr -dc 'a-zA-Z0-9' | head -c 32)
> /var/log/syslog
echo $STRING > /dev/kmsg
# Wait for things to settle down...
sleep 5
grep $STRING /var/log/syslog
# This should return 0.

Now exit out of the machine and create the following configuration file for ktest.pl called ubuntu.conf. This will bisect from v3.4 (good) to v3.5-rc1 (bad), and run the test case that we put into the VM.
# Setup default machine
MACHINE = ubuntu

# Use virsh to read the serial console of the guest
CONSOLE = virsh console ${MACHINE}
CLOSE_CONSOLE_SIGNAL = KILL

# Include defaults from upstream
INCLUDE include/defaults.conf
DEFAULTS OVERRIDE

# Make sure we load up our machine to speed up builds
BUILD_OPTIONS = -j8

# This is required for restarting VMs
POWER_CYCLE = virsh destroy ${MACHINE}; sleep 5; virsh start ${MACHINE}

# Use the defaults that update-grub spits out
GRUB_FILE = /boot/grub/grub.cfg
GRUB_MENU = Ubuntu, with Linux test
GRUB_REBOOT = grub-reboot
REBOOT_TYPE = grub2

DEFAULTS

# Do a simple bisect
TEST_START
RUN_TEST = ${SSH} /home/ubuntu/test.sh
TEST_TYPE = bisect
BISECT_GOOD = v3.4
BISECT_BAD = v3.5-rc1
CHECKOUT = origin/master
BISECT_TYPE = test
TEST = ${RUN_TEST}
BISECT_CHECK = 1

Now we should be ready to run the bisection (this will take many, many hours depending on the speed of your machine):
./ktest.pl ubuntu.conf
  1. http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/tools/testing/ktest?id=HEAD

Friday, June 6, 2014

using kgraft with ubuntu

New live kernel patching projects have hit LKML recently [1][2], and I've taken the opportunity to test drive kGraft with the Ubuntu kernel. This post documents how to get a sample patch working.

A Simple Example

First, I had to take the patches from [3] and apply them against the ubuntu-utopic kernel, which is based on 3.15-rc8 as of this post. They cherry-picked cleanly and the branch I'm using is stored here [4]. In addition to applying the patches I had to also enable CONFIG_KGRAFT. A pre-built test kernel can be downloaded here [5].

Next, I created a test VM and installed the test kernel, headers, and build dependencies into that VM and rebooted. Now after a successful reboot, we need to produce an actual patch to test. I've created a github project [6] with the sample patch; to make it easy to clone and get started.

sudo apt-get install git build-essential
git clone https://github.com/arges/kgraft-examples.git
cd kgraft-examples
make

The code in kgraft_patcher.c is the example found in samples/kgraft [7]. Now we can build it easily using the Makefile I have in my project by typing make.

Next, the module needs to be inserted using the following:

sudo insmod ./kgraft_patcher.ko

Run the following to see if the module loaded properly:

lsmod | grep kgraft

You'll notice some messages printed with the following:

[  211.762563] kgraft_patcher: module verification failed: signature and/or  required key missing - tainting kernel
[  216.800080] kgr failed after timeout (30), still in degraded mode
[  246.880146] kgr failed after timeout (30), still in degraded mode
[  276.960211] kgr failed after timeout (30), still in degraded mode

This means that not all processes have entered the kernel and may not have a "new universe" flag set.  Run the following to see which processes still needs to be updated.

cat /proc/*/kgr_in_progress

In order to get all processes to enter the kernel sometimes a signal needs to be sent to get the process to enter the kernel.

An example of this is found in the kgraft-examples [6] called 'hurryup.sh':

#!/bin/bash
for p in $(ls /proc/ | grep '^[0-9]'); do
  if [[ -e /proc/$p/kgr_in_progress ]]; then
    if [[ `sudo cat /proc/$p/kgr_in_progress` -eq 1 ]]; then
     echo $p;
     sudo kill -SIGCONT $p
    fi
  fi
done

Here is checks for all processes that have 'kgr_in_progress' set and sends a SIGCONT signal to that process. 

I've noticed that I had to also send a SIGSTOP followed by a SIGCONT to finally get everything synced up.

Eventually you'll see:

[ 1600.480233] kgr succeeded

Now your kernel is running the new patch without rebooting!

References

  1. https://lkml.org/lkml/2014/4/30/477
  2. https://lkml.org/lkml/2014/5/1/273
  3. https://git.kernel.org/cgit/linux/kernel/git/jirislaby/kgraft.git/
  4. http://zinc.ubuntu.com/git?p=arges/ubuntu-utopic.git;a=shortlog;h=refs/heads/kgraft-utopic
  5. http://people.canonical.com/~arges/kgraft-utopic/
  6. https://github.com/arges/kgraft-examples
  7. https://git.kernel.org/cgit/linux/kernel/git/jirislaby/kgraft.git/tree/samples/kgraft/kgraft_patcher.c?h=kgraft
  8. https://git.kernel.org/cgit/linux/kernel/git/jirislaby/kgraft.git/tree/tools/kgraft/create-stub.sh?h=kgraft

Thursday, July 18, 2013

using git send-email for sending kernel patches

Sending patches to a Linux kernel mailing list can be done easily using git-send-email. This post will help setup your environment and show you how to format the git send-email command. In addition there are some more advanced features that help when you need to send from multiple accounts.

Git Setup

First, install git and git-email. Then, setup ~/.gitconfig for your user and proper sendemail section. This shows a sendemail setup for a typical single gmail account.

[user]  
    name = Your Name
    email = user@gmail.com  
[sendemail]
    from = Your Name <user@gmail.com>
    smtpserver = smtp.gmail.com
    smtpuser = user@gmail.com
    smtpencryption = tls
    smtppass = PASSWORD
    chainreplyto = true
    smtpserverport = 587

However, it may be more useful to be able to easily send from multiple accounts. This can be accomplished using the --identify flag in git.

[user]
    name = Your Name
    email =user@gmail.com
[user "work"]
    email = user@work.com
[user "gmail"]
    email = user@gmail.com
[sendemail "work"]
    from = Your Name <user@work.com>
    smtpserver = smtp.work.com
    smtpuser = me
    smtppass = PASSWORD
    smtpencryption = tls
    smtpserverport = 587
    chainreplyto = false
[sendemail "gmail"]
    from = Your Name <user@gmail.com>
    smtpserver = smtp.gmail.com
    smtpuser = user@gmail.com
    smtppass = PASSWORD
    smtpencryption = tls
    smtpserverport = 587
    chainreplyto = false

This way when you can select your identity to fill in these values. In addition if you specify no identity you can have default fields if necessary. If you added a [sendemail] field this would be called by default. Man git-config can show you more options.

Formatting the Patch

Once we have the patch committed to the HEAD on our branch we format the patch using:

git format-patch -1

This should produce a patch like 0001-blah.patch.
Then check for formatting errors using the checkpatch script provided in the kernel repository:

./scripts/checkpatch.pl 0001-blah.patch

You should read the kernel documentation to get a better idea of what is expected.

Sending A Single Patch

Now we are ready to send a patch. The ./scripts/get_maintainer.pl in the kernel repository provides a way to specify whom needs to be CC'ed based on the maintainers file, the history of the file, and which lines of code are changed.


git send-email --to <ml_list> --cc-cmd="scripts/get_maintainer.pl -i" \
    <0001-patch.patch>

The -i means the script is interactive and you can edit the list before sending. Once you have completed the commands, the patch should be sent!

Pull Requests

If you have your patches in a public git repository, it is sometimes easier to send pull-requests for patches instead of sending.

git request-pull <hash right before your changes> \
    git://<public git repo> > request-pull.txt

Then add a 'Subject: ' line in the request-pull.txt that explains the pull request. Adding --subject doesn't seem to work for me. In additoin add any other tesxt


git send-email --identity=gmail --to=<mailing list> \
    ./request-pull.txt

With this you should have a pull request sent!

References

  1. http://git-scm.com/docs/git-send-email
  2. http://git-scm.com/docs/git-config
  3. https://www.kernel.org/doc/Documentation/SubmittingPatches
  4. http://git-scm.com/docs/gitcredentials.html