box.matto.nl
Enjoying Open Source Software

Posted:

Jails with nullfs mount of base system on FreeBSD 10 without buildworld

FreeBSD jails are one ot the features that make FreeBSD awesome.

Jails are mostly used for enhanced security but they help whenever working in silos offers an advantage. Very often one is advised not to mix packages build through the ports system with binary packages installed by pkg. With jails it is easy to keep them separated. Also jails can be useful in testing some application or some setup. et ceterea, there can be many more reasons to setup jails.

This is a setup without ezjail.

Of course working with ezjail is simple but setting up everything by hand helps to learn a little.

The system I did this on does not have a lot of computing power so I choose for a way to do this without building world.

The use of nullfs mounts (comparable with bind mounts on Linux) saves diskspace and because that part is mounted readonly in the jail adds some security.

I started with a fresh install of FreeBSD 10 i386.

Most of the stuff below I did according to the very good FreeBSD handbook and some other resources I found on the internet.

Install cpdup

If you have a fresh FreeBSD10 system, you may need to install cpdup.

pkg install cpdup

General overview

The jail system to create has the following layout:

home
|-- j
|   |-- mroot
|   |-- skel
|   |-- myjail
|   `-- mysecondjail
|-- js
|   |-- myjail
|   `-- mysecondjail
  • /home/j/mroot: this holds the base system that will be mounted readonly over nullfs
  • /home/j/skel: this is a template for the read-write part of the jail
  • /home/j/myjail: this is where the base system gets mounted for the jail 'myjail'
  • /home/js/myjail: this is where the read-write part of the jail lives

The directory /home/j/mroot holds the base system. For every jail this will be mounted readonly over nullfs. So this part will only exists once on your harddisk.

The /home/j/skel contains the variable files that differs from jail to jail, like /etc/rc.conf.

Setting up the directory for a new jail consists of two parts:

  • creating a directory in /home/j/ in which the base system will be mounted readonly
  • copy the /home/j/skel to a directory in /home/js/, this will holds the variable files like /etc/rc.conf of the jail and the home directories of the users. Also it contains jail specific stuff, like jail specific applications.

Also, a new jail requires adding two lines to /etc/fstab, adding an instance to /etc/jail.conf and some fine tuning.

Prepare base system

The base system will be mounted read-only into the jail.

We populate the base system from the install CDROM, so we don't have to do a buildworld.

mkdir -p /home/j/mroot
setenv DESTDIR /home/j/mroot
mount_cd9660 /dev/cd0 /mnt
tar -xf /mnt/usr/freebsd-dist/base.txz -C $DESTDIR
tar -xf /mnt/usr/freebsd-dist/doc.txz -C $DESTDIR
tar -xf /mnt/usr/freebsd-dist/ports.txz -C $DESTDIR
cd 
umount /mnt
cp /etc/resolv.conf $DESTDIR/etc/
chroot $DESTDIR
tzsetup
pkg install cpdup

With this last step (pkg install cpdup) we force the system to install the pkg utility. Later on in the process this will not be as easy as this, because we are going to move directories like /etc out of the base system.

Prepare template for the read-write part of the jail

First setup the directory for the template:

mkdir /home/j/skel /home/j/skel/home /home/j/skel/usr-X11R6 /home/j/skel/distfiles

Now populate it by moving parts of the base system into it:

cd /home/j/mroot
mv etc /home/j/skel
mv usr/local /home/j/skel/usr-local
mv tmp /home/j/skel
mv var /home/j/skel
mv root /home/j/skel

And create symlinks back into the base system:

cd /home/j/mroot
mkdir s
ln -s s/etc etc
ln -s s/home home
ln -s s/root root
ln -s /s/usr-local usr/local
ln -s /s/usr-X11R6 usr/X11R6
ln -s /s/distfiles usr/ports/distfiles
ln -s s/tmp tmp
ln -s s/var var

Setup make.conf:

echo "WRKDIRPREFIX?= /s/portbuild" >> /home/j/skel/etc/make.conf

Creating the directories for a new jail

Two directories are created for the new jail:

  • /home/j/myjail
  • /home/js/myjail

The /home/j/myjail will be populated by a nullfs mount, the /home/js/myjail will get real files by copying the skel directory to it.

Populate /home/js/myjail

cpdup /home/j/skel /home/js/myjail

Edit /etc/fstab

On the host add the following lines to /etc/fstab:

/home/j/mroot   /home/j/myjail  nullfs  ro      0       0
/home/js/myjail /home/j/myjail/s        nullfs  rw      0       0

Edit /etc/jail.conf

For each jail a block has to be added to /etc/jail.conf:

imjail {
    host.hostname = "myjail";
    path = "/home/j/myjail";
    ip4.addr += "192.168.1.100/32";
    devfs_ruleset = 4;
    allow.raw_sockets = 0;
    exec.clean;
    exec.system_user = "root";
    exec.jail_user = "root";
    exec.start = "/bin/sh /etc/rc";
    exec.stop = "/bin/sh /etc/rc.shutdown";
    exec.consolelog = "/var/log/jail_myjail_console.log";
    mount.devfs;
    allow.set_hostname = 0;
    allow.sysvipc = 0;
}

devfs and FreeBSD 10

The release of FreeBSD 10 brought some challenges around devs. I have not found a great solution for this on the internet. The best is to manually mount devfs into each jail directory before starting the jail services. 

mount -t devfs /dev /home/j/myjail/dev

This can for example be done from /etc/rc.local

Last steps

Mount the new directories for the jail, setup root password, add a user, set the hostname and make sure sshd will only listen to the ip address of the jail:

mount -a
mount
chroot /home/j/myjail
passwd
adduser
vi /etc/rc.conf
vi /etc/ssh/sshd_config
exit

Start the jail

services jail start myjail

Mount jail on a remote NFS share

Currently I am experimenting with mounting the read write part of the jail from a remote NFS server. My NFS server is a small system running Debian.

First, add some lines to /etc/rc.conf:

rpcbind_enable="YES"
rpc_statd_enable="YES"
rpc_lockd_enable="YES"

To be sure, I did a reboot of the FreeBSD system, maybe just a restart of rpc.bind is enough. After this, I mounted a remote NFS directory to /home/jnfs, followed by:

mkdir /home/j/nfsjail
cpdup /home/j/skel /home/jnfs/nfsjail
mount_nullfs -o ro /home/j/mroot /home/j/nfsjail/
mount_nullfs -o rw /home/jnfs/nfsjail /home/j/nfsjail/s/
mount -t devfs devfs /home/j/nfsjail/dev/

From there on, edited sshd_config in the nfsjail directory to only listen to the ip address of the jail, set a hostname in the rc.conf and started the jail.

Entered the jail with jexec /bin/sh.

The ultimate test was to set the root password and add a user. Both were possible without complaints about not being able to lock the password file.

Now that this seems to work, I will try to run a few jails this way and see if this can run stable. The idea behind this is to be able to run a FreeBSD jail server diskless without having to run the jails from iSCSI, see Building a diskless FreeBSD jail server

Resources

There is a lot of good and useful information on the internet. The following resources were very helpful:

Tags:

⇽ NetBSD on a Raspberry Pi FreeBSD 11 on Raspberry Pi ⇾