box.matto.nl
Enjoying Open Source Software

Posted:

Building a diskless FreeBSD 10 Jail server

Building a new diskless FreeBSD 10.0 Jail server

FreeBSD Jail is a great lightweight virtualisation method.

Our project of the day is to build a new diskless system.

The goal is to have a setup on the PXE-server that will allow a diskless motherboard to run as a FreeBSD 10.0 server. The rootfs will be mounted over NFS. Also the diskless server mounts a iscsi virtual disk on which the jails are hosted.

Our diskless server will be running on an old VIA EPIA 600 MHz mini-ITX board. This means a low power processor and not that much RAM.

In the network there is a SAN that serves both NFS shares as well as iscsi drives. The root filesystem of our new diskless server will be mounted over NFS on this SAN. This makes it easy to access the rootfs from outside the server.

For our jails NFS is not a good choice because starting and running jails requires file locks. Because of this we choose to mount a iscsi drive which will hold our jails.

Steps

  • Get FreeBSD 10.0 on a virtual machine
  • Get the latest source code
  • Build server rootfs with buildworld
  • Build kernel for diskless boot with rootfs over NFS
  • Mount rootfs over NFS
  • Install world and kernel on rootfs

Administrating FreeBSD means practicing patience. We have to checkout the source code and build a new environment so several steps will take a lot of time. It helps to plan to do steps during the night...

Get FreeBSD 10.0 on a virtual machine

At the time of writing this page, FreeBSD 10.0-BETA4 was available. By using this beta we get all the new goodies FreeBSD 10 has to offer.

FreeBSD offers several possibilities to get it up and running, I downloaded the KVM qcow2 image and started from there.

This is how I started the virtual machine on my Debian kvm-server:

kvm -vnc 192.168.1.25:0  \
-net nic,model=e1000,vlan=0 \
-net tap,ifname=tap1,script=/etc/qemu-ifup-br0\
-m 2048 -hda FreeBSD-10.0-BETA4-i386-20131130-r258774.qcow2

This virtual machine will be used to build the rootfs and the kernel for our new diskless server.

The time needed to create our new diskless server is mostly determined by the duration of compiling world and compiling the kernel. So the computing power of the virtual machine has a lot of impact on the duration of building the new diskless system.

Fetch the source code

svnlite is the new cvsup

Next we get the latest source code to build FreeBSD 10.0. In the virtual machine we fetch this code with the new svnlite that is the replacement for cvsup.

svnlite checkout https://svn0.us-west.freebsd.org/base/releng/10.0/ /usr/src

Choose a mirror site close to you.

The checkout seems to take ages, so do this at the end of the day and continue with the following steps the next morning.

Build server rootfs with buildworld

First, set up /etc/src.conf.

WITHOUT_ATM=yes
WITHOUT_BIND=yes
WITHOUT_BLUETOOTH=yes
WITHOUT_FLOPPY=yes
WITHOUT_GAMES=yes
WITHOUT_INET6=yes
WITHOUT_IPX=yes
WITHOUT_PPP=yes
WITHOUT_SENDMAIL=yes
WITHOUT_SHAREDOCS=yes
WITHOUT_WIRELESS=yes
WITHOUT_ZFS=yes

The VIA EPIA motherboard does not support that much RAM, so we will not use ZFS. I have not yet played with IPV6, so we first build without IPV6 support. After having set up /etc/src.conf we can start building the root filesystem.

Probably we could have disabled more parts of the build to create a smaller rootfs, but this way we have something to start with that can be optimized in a later phase.

mount server:/path/to/nfsshare /mnt
setenv NFSROOTDIR /mnt
cd /usr/src
make buildworld
make installworld DESTDIR=${NFSROOTDIR}

Build kernel for diskless boot with rootfs over NFS

Next we have to build a kernel that allows diskless booting.

Create /usr/src/sys/i386/conf/MYKERNEL as a config file for the configuration of the kernel. 

make buildkernel KERNCONF=MYKERNEL
make installkernel KERNCONF=MYKERNEL DESTDIR=${NFSROOTDIR}
make distribution DESTDIR=${NFSROOTDIR}

Prepare new environment

Go into the newly created environment, set password and create a new user, that can be used for ssh login.

chroot ${NFSROOTDIR}
passwd
adduser
echo 'sshd_enable="YES"' > /etc/rc.conf
echo 'defaultrouter="192.168.1.254"' >>  /etc/rc.conf
mkdir /iscsi
mkdir -p conf/base
mkdir -p conf/base/iscsi
tar -c -v -f conf/base/etc.cpio.gz --format cpio --gzip etc
tar -c -v -f conf/base/var.cpio.gz --format cpio --gzip var
exit

The /conf/base/var and /conf/base/etc directories become read-write mounted as /var and /etc on a memory file system in our diskless system.

The conf/base/iscsi directory becomes read-write mounted as /iscsi on our diskless system. We need this to be able to mount an iscsi-drive on which we can install our jails.

Prepare PXE server

On the PXE/DHCP server we now have to set up an entry for our diskless server. I use the isc-dhcp-server on a Debian-box. This is what is needed in /etc/dhcp/dhcpd.conf:

host disklessfreebsdbox {
    hardware ethernet 12:34:56:78:90:ab;
    fixed-address 192.168.1.9;
    filename "freebsd10/boot/pxeboot";
    next-server 192.168.1.2;
    option domain-name "tradesystem.nl";
    option domain-name-servers 192.168.1.1;
    option subnet-mask 255.255.255.0;
    option root-path "192.168.1.2:/rootfs/freebsd10";
}

You have to change this according to your local situation.

Here is some explanation on the entry above:

  • hardware ethernet: the mac address of your diskless system
  • fixed-address: this is the ip address to be asigned to your diskless system
  • filename "freebsd10/boot/pxeboot": the location of the file named "pxeboot" which was installed during the execution of the make commands above, the path is relative to the tftpboot-path on the PXE-server
  • next-server 192.168.1.2: the ip-address of the NFS-server
  • option domain-name-servers 192.168.1.1: your DNS-server (mine is local)
  • option subnet-mask 255.255.255.0: the netmask for your diskless system
  • option root-path "192.168.1.2:/rootfs/freebsd10": the location of the root filesystem which was installed during the execution of the make commands above, that is going to be mounted over NFS by the diskless system.

Boot diskless

After these preparations are done we can do a test boot of the system. Solve any problems before going to the next part.

Enter your fresh diskless system and have a look around. Notice that the root filesystem is mounted read-ony. This is done by the script inside /etc/rc.initdiskless, this is part of the standard FreeBSD environment. This is why we need the /conf/base directory on the NFS-server. The contents of this directory will be mounted in read-write mode on a memory filesystem.

Install ezjail

Stop the diskless system. In the rootfilesystem we create a /etc/iscsi.conf file and run the tar command again to re-create the cpio format etc.cpio.gz file.
After that chroot again into the root filesystem and install ezjail.

cd ${NFSROOTDIR}
cd etc
cp /etc/resolv.conf .
chroot ${NFSROOTDIR}
pkg install ezjail

Now the pkg repository catalog will be initialized. Have a cup of tea, it might take a while.

The total amount of data on the ${NFSROOTDIR} for our diskless system will now be grown to about 520 Mb.

Prepare ezjail

Setup iscsi, mount iscsi drive

Go into the diskless system.

Use the created /etc/iscsi.conf file to connect to the iscsi-server and mount the iscsi-share on /iscsi on the diskless system.

mkdir -p /iscsi/jails/etc/ezjail

On the virtual machine create a symlink in the NFS-share

chroot ${NFSROOTDIR}
cd /usr/local/etc
rmdir ezjail
ln -s /iscsi/jails/etc/ezjail .

In /usr/local/etc/ezjail.conf set the ezjail_jaildir to the /iscsi directory

ezjail_jaildir=/iscsi/jails

This way when we create jails in the diskless server, these will be created on the iscsi-drive, which is the only re-writable partition on a harddisk.

Initialize ezjail

ezjail-admin install

After this, create a first jail to test

ezjail-admin create -f example example.com 10.0.0.2

Make sure you have created an alias on your network interface. This can be done on the virtual machine in the rc.conf on the ${NFSROOTDIR}.

Using the system

One of the quirks of FreeBSD jails is that for every jail that is created a file named /etc/fstab.<jailname> is created, where <jailname> is the name of the jail with periods replaced by underscores.

In our diskless system the /etc directory lives in a memory filesystem. This means any run-time changes are lost on shutdown of the system. I have not found a way to change this path. So we have to conserve these files on a writable harddisk.

What i do is copy the /etc/fstab.* files to /iscsi/jails/etc/ and after a reboot and a remount of /iscsi copy the files back to /etc. It is ugly, but it works...

Resources

To build this server I used a tons of info from the internet. Below are some very helpful pages.

Tags:

⇽ Raspberry Pi as PXE, TFTP, DHCP and NFS server NetBSD on a Raspberry Pi ⇾