box.matto.nl

home/

Modify Debian Squeeze KVM guest to a diskless KVM system

Last edited
Fun with a diskless Debian Squeeze KVM guest system

Build a system to run diskless KVM Linux guests with a NFS mounted rootfs

This page is about creating and booting diskless KVM Debian Squeeze guests. But much of it can be used for other Linux guests too.

The diskless guest boots over PXE, gets a kernel and an initramfs over tfpd and mounts a NFS root filesystem. The ip number of the KVM guest determiinse the root filesystem.

So on the NFS server we have a directory structure like this:

/kvm
 |-/rootfs
     |-192.168.1.50
     |   |---/etc
     |   |---/bin
     |   |---/sbin
     |   |---/...
     |-192.168.1.51
         |---/etc
         |---/...

This means that several KVM guests can be booted diskless and that the mac-address specified on the start of the KVM guest determines the ip address it gets from the DHCP server and which rootfs it will mount over NFS.

The whole idea of this is to make things easy and simple (after the initial work is done :)

Install KVM with Debian Squeeze

Create a KVM disk image

qemu-img create -f qcow2 squeeze.img 4G

Install Squeeze over PXE on the KVM disk image

kvm  -vnc 192.168.1.1:1 -boot n -option-rom /usr/share/kvm/pxe-rtl8139.bin \
-net nic,model=rtl8139,vlan=0,macaddr=01:23:45:67:89:ab \ 
-net tap,ifname=tap1,script=/etc/qemu-ifup-br0 -m 1024 -hda squeeze.img

When requested what type of system to install, choose for SSH-server.

Boot from KVM disk image and install nfs-common

kvm  -vnc 192.168.1.2:1 -net nic,model=rtl8139,vlan=0,macaddr=01:23:45:67:89:ab \ 
-net tap,ifname=tap1,script=/etc/qemu-ifup-br0 -m 1024 -hda squeeze.img

apt-get install nfs-common

Remove some unneeded files

apt-get install localepurge

Mount nfs partition to move the root file system to and copy system

Now we have a working Debian Squeeze system, we are going to move it to the NFS server.

Create a new directory on your NFS server and configure /etc/exports to allow mounting of this directory.

After editting /etc/exports reload this file with:

 exportfs -rv

On the KVM guest we are now going to transfer our root filesystem to the NFS mount:

mount -tnfs -onolock 192.168.1.1:/kvm/rootfs /mnt   
cp -axv /. /mnt/.  
cp -axv /dev/. /mnt/dev/.  

Halt KVM system and tune the rootfs on the NFS server.

Stop the KVM system, you can delete the squeeze,img or keep it around for other stuff. Change some files in the rootfs on the NFS server, like /etc/fstab and /etc/network/interfaces in order to let it properly run from nfs.

Replace /etc/fstab with the following.

# <file system> <mount point>   <type>  <options>       <dump>  <pass>
proc            /proc           proc    defaults        0       0
/dev/nfs        /               nfs     defaults 0 0
none            /tmp            tmpfs   defaults 0 0
none            /var/run        tmpfs   defaults 0 0
none            /var/lock       tmpfs   defaults 0 0
none            /var/tmp        tmpfs   defaults 0 0
none            /var/lib/dhcp   tmpfs   defaults 0 0
none            /media          tmpfs   defaults 0 0
  • In the root file system comment out everything in /etc/network/interfaces or replace this file with an empty one.

Prepare initramfs

The NFS mount of the rootfs is done from the initramfs, we create our own initramfs for this.

Build a small busybox system

Download busybox and create a small system for the initramfs.

Use the busybox udhcp client in the initramfs

Put the simple.script needed for udhcp client into the /bin directory of the initramfs. Alter this script so it writes the ip address it got from the DHCP server to /etc/ip. We are going to use this in order to NFS mount the right rootfs. Here is the part of simple.script with the alternation:

        renew|bound)
                echo "Setting IP address $ip on $interface"
                ifconfig $interface $ip $NETMASK $BROADCAST
                echo "$ip" > /etc/ip

The line containing echo "$ip" > /etc/ip is not in the original simple.script but added for this.

Now we can use this in the /init script of the initramfs. Here is the relevant part of /init:

mijnip=`cat /etc/ip`
mount -tnfs -onolock 192.168.1.8:/kvm/rootfs/$mijnip /newroot

The complete contents of my initfile is at the bottom of this page.

Create and populate /lib/modules

Because we are going to run several KVM guests we could build a special kernel for this with the all the needed drivers built in. For now we take a "normal" kernel and modprobe the modules from the init file in the initramfs.

Don't forget to put the relevant files into /lib/modules in the initramfs.

Build the initramfs

Do not forget to make the /init file executable (chmod a+x init) and to setuid the busybox file (chmod a+s busybox) first.

find . | cpio -H newc -o | gzip -9 > ../initramfs.gz

Configure tftpd-hpa and dhcpd.conf on your DHCP server

Configure dhcpd.conf

Make life more simple by giving your KVM guest a fixed ip address. Here is an example for a entry in /etc/dhcp/dhcpd.conf:

host kvmguest {
        hardware ethernet 54:52:00:00:00:01;
        fixed-address 192.168.1.4;
        next-server 192.168.1.1;
        option subnet-mask 255.255.255.0;
        option broadcast-address 192.168.1.255;
        option routers 192.168.1.254;
}

It is also possible to replace the ip address in the fixed-address line by a hostname:

host kvmguest {
        hardware ethernet 54:52:00:00:00:01;
        fixed-address kvmbox.mydomain.com;
        next-server 192.168.1.1;
        option subnet-mask 255.255.255.0;
        option broadcast-address 192.168.1.255;
        option routers 192.168.1.254;
}

Configure tftpd-hpa to bypass menu

I use tftpd-hpa for my tftpd file transfers.

Normaly you set up a menu for your PXE tftp file transfers. If everythings works OK you can by pass the menu by adding a small file in your pxelinux.cfg directory. There are several options for naming this file. The most easy one is

01-54-52-00-00-00-01

where the mac-address of the KVM guest is 54:52:00:00:00:01. So simply replace the colons (:) with dashes (-) and add 01- in front of it.

But we want a number of machines boot the same kernel and initramfs, so we go for the hexidecimal representation of the ip address. This way we can choose a range of ip addresses which result in the same hexidecimal string. The easiest way to choose the ip range is by using the program gethostip. This is part of the syslinux package in Debian, so apt-get install that package.

Now we can ask the hexidecimal representation of the ip address:

gethostip 192.168.1.96
192.168.1.96 192.168.1.96 C0A80160

gethostip 192.168.1.111
192.168.1.111 192.168.1.111 C0A8016

So if we choose the range 192.168.1.96 - 192.168.1.111 we can name our config file pxelinux.cfg/C0A8016

Likewise we could choose the range 192.168.1.16 - 192.168.1.255 and use the config file pxelinux.cfg/C0A801

This is what you put inside this config file:

default linux
prompt 0

label linux
        kernel /srv/tftp/debian/kvmguest/vmlinuz
        append initrd=/srv/tftp/debian/kvmguest/kvmguest.initramfs.gz root=/dev/nfs vga=0x301 acpi=ht netboot=nfs

It is possible to increase the log-level of tftpd-hpa. The configuration of tftpd-hpa is done in a strange location, this is to be found in /etc/default/tftpd-hpa. Add a line to this file:

TFTP_OPTIONS="-vvv"

Or if there is already a TFTP_OPTIONS line in it, just add -vvv to it.

Don't forget to restart tftpd-hpa after changing the configuration.

Interesting follow ups

Make the root filesystem readonly

FreeBSD automatic mounts it's root filesystem readonly when it discoveres it is booted diskless. This is imho a great idea. For Debian mounting the root filesystem requires some tuning. These pages seems to be interesting:

init file

Below follows the contents of my /init in the initramfs:

#!/bin/busybox sh

[ -d /dev ] || mkdir -m 0755 /dev
[ -d /root ] || mkdir -m 0700 /root
[ -d /sys ] || mkdir /sys
[ -d /proc ] || mkdir /proc
[ -d /tmp ] || mkdir /tmp
[ -d /newroot ] || mkdir /newroot
mkdir -p /var/lock
mount -t sysfs -o nodev,noexec,nosuid none /sys
mount -t proc -o nodev,noexec,nosuid none /proc

modprobe 8139cp
modprobe sunrpc
modprobe auth_rpcgss
modprobe nfs_acl
modprobe fscache
modprobe lockd
modprobe nfs

echo "Configure eth0 ..."
ifconfig eth0 up
udhcpc -t 5 -q -s /bin/simple.script

echo "Sleep 5 ..."
sleep 5

echo "Mount nfs ..."
mijnip=`cat /etc/ip`
mount -tnfs -onolock 192.168.1.8:/kvm/rootfs/$mijnip /newroot
mount --move /proc /newroot/proc
mount --move /sysfs /newroot/sys

exec switch_root /newroot /sbin/init