Pi-BSD

Bastille Jail Management on Raspberry Pi

by

Continuing my series on creating a Raspberry Pi running FreeBSD for a small home server, I’ve decided to take the plunge and utilize jails. Initially, I wasn’t going to go this route because the Pi isn’t going to do all that much, but as I started to install unbound as a dns server and ad-blocker, I realized the isolation of a jail would be really helpful. I might change my mind about this setup and jails make it easy to blow away a bunch of changes and start over (there are other benefits, including avoiding dependency issues and added security).

A jail is basically a form of virtualization which allows you to create a “system within a system.” Varying degrees of virtualization, from complete virtual machines such as KVM in Linux to svelte portable Docker containers, have been all the rage for the last ten years. With jails, FreeBSD has been on top of the virtualization craze before it was a craze.

As a novice, my favorite thing about jails is that I can tinker around and if I make a mess and break things I can just blow everything up without doing a full reinstall. By running each major application in its own jail, you know it (and its dependencies) won’t conflict with your other major applications and any tinkering you do won’t break what’s already working. Also, with ZFS it’s easy to snapshot jails. Unlike traditional backup methods, this is pretty instantaneous, local, and because of the way ZFS works, doesn’t take up a bunch of space. That way you can take a snapshot before upgrading

I initially planned on using iocage to create a jail, but when researching this article I came across Bastille and I’ve decided to use that for jail management instead. Bastille is lighter and comes with a bunch of templates which should make installing new services much easier.

1. Install & Enable Bastille

Make sure your system and all port/pkgs are up to date. If you’re using ports go to /usr/ports/sysutils/bastille and type the following:

make -C /usr/ports/sysutils/bastille install clean

If you’re using pkg type the following:

pkg install bastille

Now you need to enable Bastille by adding the following line to /etc/rc.conf:

bastille_enable="YES"

You will need to restart or type service bastille start for this to take effect, but wait until after the next two steps to do this.

2. Configure Bastille

I’m going to use my ZFS mirror to store my jails, which doesn’t actually store the operating system running my Pi. Your configuration needs may be different and this guide should explain what you need regardless of your setup, but keep in mind that if you’re following this guide step-by-step you’ll need to start with setting up the ZFS mirror.

Our Bastille configuration file can be found here: /usr/local/etc/bastille/bastille.conf. Open the file in your favorite text editor.

Following the official Bastille startup guide, I changed a couple lines. First, the timezone:

## default timezone
bastille_tzdata="America/Detroit"   

Next, I enable ZFS and utilize my zpool. Note that in my setup the SD card runs the OS using the UFS filesystem while my zpool consists of two mirrored SSDs. It is possible to do a couple other things with the Pi. Whatever you do, I highly recommend you use ZFS with your jails because you will find it can be a lifesaver. With snapshots, you can quickly and easily snapshot your jails before making any major changes and then revert to the snapshot should something go wrong. Sure, you could run a backup each time, but that’s cumbersome and slow. Check out the introductory post where I review other possible configurations.

## ZFS options
bastille_zfs_enable="YES"
bastille_zfs_zpool="nexus"  

Finally, I tell Bastille to install the jails on the external SSDs. I had to do a bit of fiddling around to get this to work right. When I directed all the paths to the SSDs it created some errors when Bastille tried to read files in the etc folder on the SD card (fstab, localtime, crontab). The configuration below worked to keep the jails on the SSDs and the necessary config stuff on the SD card. I think the key is for the bastille_prefix to be on the SD card (where the host system is) and then you can specify the rest to be on your external drives.

## default paths
bastille_prefix="/usr/local/bastille"
bastille_backupsdir="/nexus0/bastille/backups"
bastille_cachedir="/nexus0/bastille/cache"
bastille_jailsdir="/nexus0/bastille/jails"
bastille_releasesdir="${bastille_prefix}/releases"
bastille_templatesdir="${bastille_prefix}/templates"
bastille_logsdir="/var/log/bastille"

Note: I missed this step the first time I set it up so when I created my jail it slapped it on the SD card and had created all my Bastille files in /usr/local/bastille/. For some reason I got “operation not permitted” when I tried to just move this folder over to nexus0, so I had to use mkdir bastille, mkdir backups, etc. to create all these folders individually. Then I had to set the permissions: [email protected]:/nexus0/bastille # chmod 0750 /nexus0/bastille . This shouldn’t be a problem if you follow this guide step-by-step (I think).

3. Configure Jail Networking

Networking with jails can be tricky. This setup just follows the Bastille documentation and should be fine since everything is on a local network. If it doesn’t work with some of the jail services I install I’ll update this section later with a setup that can accommodate everything.

First, edit /etc/rc.conf and add the following lines:

cloned_interfaces="lo1"
ifconfig_lo1_name="bastille0"

Now bring up the cloned device:

[email protected]:/ # service netif cloneup

What you have done is “cloned” your local device, which is really just a sort of virtual network device on the machine. Using typing ifconfig will help explain things:

[email protected]:/etc # ifconfig
genet0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=68000b<RXCSUM,TXCSUM,VLAN_MTU,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
	ether dc:a8:32:c8:28:4d
	inet 192.168.1.2 netmask 0xffffff00 broadcast 192.168.1.255
	media: Ethernet autoselect (100baseTX <full-duplex>)
	status: active
	nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
	options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
	inet6 ::1 prefixlen 128
	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
	inet 127.0.0.1 netmask 0xff000000
	groups: lo
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
bastille0: flags=8008<LOOPBACK,MULTICAST> metric 0 mtu 16384
	options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
	groups: lo
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

Tidbit for learners

Let’s dissect this a bit. I am not an expert so I won’t cover everything here (if you are and I got something wrong, feel free to e-mail me a correction), but understanding some networking 101 is helpful. First, genet0 is your physical ethernet device. If you have wifi enabled then you will see another device listed, too. Where you see ether dc:a8:32:c8:28:4d, that tells us this is an ethernet device and the MAC address. Where you see inet 192.168.1.2, that tells you the device IP address. This is not your public IP address on the internet because you have only one (usually) and you cannot assign it to every device on your local network. Instead, using DHCP, your router assigns local addresses to each device on the network.

I told my router to always assign the MAC address dc:a8:32:c8:28:4d the IP address 192.168.1.2. Generally, a router will always assign the same IP address to a device once it learns the MAC address, but it helps to specify it. On my router, if I don’t specify the IP address, it will forget it if there’s a power failure or if I unplug the router. If your ISP provides you with an IPv6 address instead of IPv4, then you actually receive a whole bundle of IPv6 addresses instead of a single address, meaning that your router will assign unique public IP addresses to all your devices.

While there’s a local network inside your home, your operating system can run a sort of local network within itself. This is our loopback device, lo0. We “clone” this so Bastille can then assign local IP addresses in our jails. So we don’t mix these up with the local addresses assigned by the router, they will fall under the 10.0.0.0 block. Like the 192.168.x.x addresses, these are reserved as local addresses.

3. Configure FreeBSD Networking

The next thing we have to do is setup the pf firewall to act as a sort of liaison between our jail network, loopback device, and physical network. The first time I setup jails this is the part that really tripped me up. I’m going to keep this simple and just use pf to direct the jail networking traffic, but in the future I might write an article describing how to set it up as a real firewall.

Pf is installed on FreeBSD by default, so you just have to add the following line to /etc/rc.conf:

pf_enable="YES"

Now we need to create a firewall configuration file for pf. This file is /etc/pf.conf. You will probably have to create it. Put the following text into the file (exclude the lines stylized with strike-through):

ext_if="genet0"

set block-policy return
scrub in on $ext_if all fragment reassemble
set skip on lo

table <jails> persist
nat on $ext_if from <jails> to any -> ($ext_if:0)
rdr-anchor "rdr/*"

block in all
pass out quick keep state
antispoof for $ext_if inet
pass in inet proto tcp from any to any port ssh flags S/SA keep state

Now restart.

Tidbit for learners

The above is copied from the official Bastille site, with the only alteration being the name of our ethernet device genet0, as pulled from ifconfig above. I’m still learning pf and there’s a lot to it so I’ll just explain some essentials here.

Pf is hierarchical in two ways. First, there are seven major sections of the configuration, macros, tables, options, normalization, queueing, translation, filtering, and these must be in order (starting at the top of the document with options). Second, as we move down the document, later rules overrule any previous rules. For instance, I could make a rule block all, which, by itself would be very bad because it would block all traffic. However, I could then make subsequent rules that would function as exceptions to the rule.

In our first section, options, we set variables. In this case, ext_if defines our external interface (our ethernet port). We call the variable with the $, so $ext_if below is equal to genet0.

In the next section, we just have some standard pf stuff. set block-policy return just tells the firewall what to do with rejected packets. By default this is set to drop. I do not know if Bastille requires this change or if it’s simply the preference of the creator. I removed the next line, which corrects fragmented packets. This is usually a good idea, but it can interfere with our NFS server (if you want this option, it is possible to run pf inside of your jails and set it there…I think). set skip on lo simply says the rules don’t apply to the loopback device.

The next section is where the real magic happens. “NAT” stands for “network address translation.” Basically, we route incoming packets on ethernet (ext_if) to the local address of the jail and vice versa.

The final four lines just add some basic security and efficiency. Note that the last line ensures that SSH is available (so you haven’t locked yourself out in case you don’t use a monitor). I removed the block in all line because it locks out NFS. If this was a public facing server, I would want to leave it and then make a more specific rule to allow NFS (like what this configuration does with SSH). For the sake of security, I’ll probably extend this firewall configuration in the future.

4. Bootstrap a FreeBSD version and Create Jail

This step just assigns a version of FreeBSD to Bastille so you can use it to create your jails. Basically, this is your “template” and the jails are deviations from this template. This is what makes jails so lightweight and powerful. I use the word “template” in quotation marks because I don’t want it to be confused with the actual Bastille template we will apply in our next steps.

Right now 13.1 is the latest stable FreeBSD version. In the future you may want to replace that with your current version.

bastille bootstrap 13.1-RELEASE update

Now we create our jail:

[email protected]:~ # bastille create test-jail 13.1-RELEASE 10.5.5.5

Tidbit for learners

This should be pretty obvious, but bastille is the command that summons Bastille, create tells it to create a new jail, test-jail is the name of this jail (it can be anything you like), and 10.5.5.5 is the IP address assigned to the jail. As mentioned before, the IP address can be anything you want as long as it fits the format of 10.x.x.x or 172.16.x.x (with the ‘x’ being any 1-3 digit numbers you want). You can also do 192.168.x.x, but that may conflict with addresses your router has assigned.

5. Test Jail

Bastille allows you to to run commands, modify your rc.conf file, or start/stop/restart services without entering the jails using the bastille commands cmd, sysrc, and service respectively. For example, to see the contents of your jail /usr/local/etc folder, you could type:

[email protected]:~ # bastille cmd test-jail ls /usr/local/etc

Personally, I prefer to just “enter” the jail and treat it like I’ve booted into a different operating system. To do this, use the console command:

[email protected]:/ # bastille console test-jail

When successful, you will now see that you are logged in under your jail:

[email protected]:~ #

You can now test your networking:

[email protected]:/ # fetch https://www.freebsd.org/images/beastie.png

This should download the beastie.png image and you should see if in whatever folder you’re in using the ls command.

It should be noted that not all of your networking is fully functional. For example:

[email protected]:~ # ping 1.1.1.1
ping: ssend socket: Operation not permitted

If don’t need these network services, don’t add them as they make your jail less secure. If you do, add the following line to the end of /etc/pf.conf (on the host system, the file we modified in step 3):

pass inet proto icmp icmp-type {echoreq unreach}

Then edit your jail.conf file. Mine is located here (replace red text with the appropriate values): /nexus0/bastille/jails/test-jail/jail.conf. Add the following text somewhere on a new line inside the curly brackets:

allow.raw_sockets;

Then restart the jail:

[email protected]:/ # service bastille restart

More Adventures With Raspberry Pi. . .

F

Back to Top