Raspberry Pi in a Jail

Bastille Jail Management on Raspberry Pi

by

Continuing my series on creating a Raspberry Pi running FreeBSD for a small home server, I’ve decided to take the plunge and utilize jails managed by Bastille. Initially, I wasn’t going to go this route because the Pi isn’t going to do all that much, but as I started to install unbound as a dns server and ad-blocker, I realized the isolation of a jail would be really helpful. I might change my mind about this setup and jails make it easy to blow away a bunch of changes and start over (there are other benefits, including avoiding dependency issues and added security).

A jail is basically a form of virtualization which allows you to create a “system within a system.” Varying degrees of virtualization, from complete virtual machines such as KVM in Linux to svelte portable Docker containers, have been all the rage for the last ten years. With jails, FreeBSD has been on top of the virtualization craze before it was a craze.

Verbose (article contains many explanations and tangential information)

  1. Home Server with Raspberry Pi & FreeBSD
  2. Home NAS Using a Raspberry Pi
  3. Bastille Jail Management on Raspberry Pi
  4. Pi-Hole with FreeBSD using AdGuard Home

As a novice, my favorite thing about jails is that I can tinker around and if I make a mess and break things I can just blow everything up without doing a full reinstall. By running each major application in its own jail, you know it (and its dependencies) won’t conflict with your other major applications and any tinkering you do won’t break what’s already working. Also, with ZFS it’s easy to snapshot jails. Unlike traditional backup methods, this is pretty instantaneous, local, and because of the way ZFS works, doesn’t take up a bunch of space. That way you can take a snapshot before upgrading

I initially planned on using iocage to create a jail, but when researching this article I came across Bastille and I’ve decided to use that for jail management instead. Bastille is lighter and comes with a bunch of templates which should make installing new services much easier.

Necessary Equipment

Links are affiliate links through Amazon. If you’re going to buy these items, use the links below to support this site. Prices tend to fluctuate so I just listed what I paid. They should be approximately right. For the Raspberry Pi, check out RPI locator. They are extremely hard to find right now (2022). When they’re available for the suggested retail price on Amazon again I’ll add a link.

$35 (1) Raspberry Pi 4
Although you could technically do this on an earlier model, I would advise against it. The Pi 4 isn’t fast, but it’s significantly faster than the previous models. Plus, it’s the only model to include USB 3.0 ports. Note that $35 will get you the 1GB RAM model. I use a 2GB ($45) and they go up to 8GB for $75. I might upgrade to an 8GB since I plan on throwing a lot at this thing, but so far the 2GB has held up well.

$18 (1) Raspberry Pi 4 Accessories
You’ll need to buy your power supply and case for the Raspberry Pi separate. This kit gives you those and some heat sinks and a fan. If you do everything in this series at least the heat sinks would be a good idea.

$8 (1) Micro SD Card
The primary drive for your Raspberry Pi (sounds scary, I know). Since I am just using the SD card for the OS and some other basic software (Wireguard, Pi-Hole), I went with a 32GB because it was cheap.

$60 (2) 2.5″ SATA SSDs
These are your standard laptop hard drives. You could go with old-school spinning drives and it probably wouldn’t drastically change the performance since this is getting routed through USB and then the Pi, which probably bottlenecks things. You can get these pretty cheap, such as the PNY drives I use. Better drives do perform better and probably last longer. If I’m happy with this setup I may upgrade in the future.

You may see some guides online using USB thumb drives for a similar setup. I highly discourage you from doing this because those drives are not designed for the constant read-write operations.

$20 (2) 2.5″ Drive Cases (SATA to USB 3.0)
These are cheap and easy to find. The SSKs I use cost about $10 a piece. I picked them because of the aluminum housing. There are all sorts of other options you could go with, but these are about as minimal as you can get.

~$141 Total
You may need to spend a little more for a power cable and case, or any other Pi accessories you may desire. You may also need an adapter or reader to use the SD card on another computer to install the operating system.

1. Install & Enable Bastille

Make sure your system and all port/pkgs are up to date. If you’re using ports go to /usr/ports/sysutils/bastille and type the following:

make -C /usr/ports/sysutils/bastille install clean

If you’re using pkg type the following:

pkg install bastille

Now you need to enable Bastille by adding the following line to /etc/rc.conf:

bastille_enable="YES"

You will need to restart or type service bastille start for this to take effect, but wait until after the next two steps to do this.

2. Configure Bastille

I’m going to use my ZFS mirror to store my jails, which doesn’t actually store the operating system running my Pi. Your configuration needs may be different and this guide should explain what you need regardless of your setup, but keep in mind that if you’re following this guide step-by-step you’ll need to start with setting up the ZFS mirror.

Our Bastille configuration file can be found here: /usr/local/etc/bastille/bastille.conf. Open the file in your favorite text editor.

Following the official Bastille startup guide, I changed a couple lines. First, the timezone:

## default timezone
bastille_tzdata="America/Detroit"   

Next, I enable ZFS and utilize my zpool. Note that in my setup the SD card runs the OS using the UFS filesystem while my zpool consists of two mirrored SSDs. It is possible to do a couple other things with the Pi. Whatever you do, I highly recommend you use ZFS with your jails because you will find it can be a lifesaver. With snapshots, you can quickly and easily snapshot your jails before making any major changes and then revert to the snapshot should something go wrong. Sure, you could run a backup each time, but that’s cumbersome and slow. Check out the introductory post where I review other possible configurations.

## ZFS options
bastille_zfs_enable="YES"
bastille_zfs_zpool="nexus"  

Finally, I tell Bastille to install the jails on the external SSDs. I had to do a bit of fiddling around to get this to work right. When I directed all the paths to the SSDs it created some errors when Bastille tried to read files in the etc folder on the SD card (fstab, localtime, crontab). The configuration below worked to keep the jails on the SSDs and the necessary config stuff on the SD card. I think the key is for the bastille_prefix to be on the SD card (where the host system is) and then you can specify the rest to be on your external drives.

## default paths
bastille_prefix="/usr/local/bastille"
bastille_backupsdir="/nexus0/bastille/backups"
bastille_cachedir="/nexus0/bastille/cache"
bastille_jailsdir="/nexus0/bastille/jails"
bastille_releasesdir="${bastille_prefix}/releases"
bastille_templatesdir="${bastille_prefix}/templates"
bastille_logsdir="/var/log/bastille"

Note: I missed this step the first time I set it up so when I created my jail it slapped it on the SD card and had created all my Bastille files in /usr/local/bastille/. For some reason I got “operation not permitted” when I tried to just move this folder over to nexus0, so I had to use mkdir bastille, mkdir backups, etc. to create all these folders individually. Then I had to set the permissions: [email protected]:/nexus0/bastille # chmod 0750 /nexus0/bastille . This shouldn’t be a problem if you follow this guide step-by-step (I think).

3. Configure Jail Networking

Networking with jails can be tricky, but Bastille makes it a little bit easier. This setup just follows the Bastille documentation and should be fine since everything is on a local network.

First, edit /etc/rc.conf and add the following lines:

cloned_interfaces="lo1"
ifconfig_lo1_name="bastille0"

Now bring up the cloned device:

[email protected]:/ # service netif cloneup

What you have done is “cloned” your local device. This is really just a sort of virtual network device on the machine. Using typing ifconfig will help explain things:

[email protected]:/etc # ifconfig
genet0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=68000b<RXCSUM,TXCSUM,VLAN_MTU,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
	ether dc:a8:32:c8:28:4d
	inet 192.168.1.2 netmask 0xffffff00 broadcast 192.168.1.255
	media: Ethernet autoselect (100baseTX <full-duplex>)
	status: active
	nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
	options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
	inet6 ::1 prefixlen 128
	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
	inet 127.0.0.1 netmask 0xff000000
	groups: lo
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
bastille0: flags=8008<LOOPBACK,MULTICAST> metric 0 mtu 16384
	options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
	groups: lo
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

Tidbit for learners

Let’s dissect this a bit. I am not an expert so I won’t cover everything here (if you are and I got something wrong, feel free to e-mail me a correction), but understanding some networking 101 is helpful. First, genet0 is your physical ethernet device. If you have wifi enabled then you will see another device listed, too. Where you see ether dc:a8:32:c8:28:4d, that tells us this is an ethernet device and the MAC address. Where you see inet 192.168.1.2, that tells you the device IP address. This is not your public IP address on the internet because you have only one (usually) and you cannot assign it to every device on your local network. Instead, using DHCP, your router assigns local addresses to each device on the network.

I told my router to always assign the MAC address dc:a8:32:c8:28:4d the IP address 192.168.1.2. Generally, a router will always assign the same IP address to a device once it learns the MAC address, but it helps to specify it. On my router, if I don’t specify the IP address, it will forget it if there’s a power failure or if I unplug the router. If your ISP provides you with an IPv6 address instead of IPv4, then you actually receive a whole bundle of IPv6 addresses instead of a single address, meaning that your router will assign unique public IP addresses to all your devices.

While there’s a local network inside your home, your operating system can run a sort of local network within itself. This is our loopback device, lo0. We “clone” this so Bastille can then assign local IP addresses in our jails. So we don’t mix these up with the local addresses assigned by the router, they will fall under the 10.0.0.0 block. Like the 192.168.x.x addresses, these are reserved as local addresses.

3. Configure FreeBSD Networking

The next thing we have to do is setup the pf firewall to act as a sort of liaison between our jail network, loopback device, and physical network. The first time I setup jails this is the part that really tripped me up. I’m going to keep this simple and just use pf to direct the jail networking traffic, but in the future I might write an article describing how to set it up as a real firewall.

Pf is installed on FreeBSD by default, so you just have to add the following line to /etc/rc.conf:

pf_enable="YES"

Now we need to create a firewall configuration file for pf. This file is /etc/pf.conf. You will probably have to create it. Put the following text into the file (exclude the lines stylized with strike-through):

ext_if="genet0"

set block-policy return
scrub in on $ext_if all fragment reassemble
set skip on lo

table <jails> persist
nat on $ext_if from <jails> to any -> ($ext_if:0)
rdr-anchor "rdr/*"

block in all
pass out quick keep state
antispoof for $ext_if inet
pass in inet proto tcp from any to any port ssh flags S/SA keep state

Now restart.

Tidbit for learners

The above is copied from the official Bastille site, with the only alteration being the name of our ethernet device genet0, as pulled from ifconfig above. I’m still learning pf and there’s a lot to it so I’ll just explain some essentials here.

Pf is hierarchical in two ways. First, there are seven major sections of the configuration, macros, tables, options, normalization, queueing, translation, filtering, and these must be in order (starting at the top of the document with options). Second, as we move down the document, later rules overrule any previous rules. For instance, I could make a rule block all, which, by itself would be very bad because it would block all traffic. However, I could then make subsequent rules that would function as exceptions to the rule.

In our first section, options, we set variables. In this case, ext_if defines our external interface (our ethernet port). We call the variable with the $, so $ext_if below is equal to genet0.

In the next section, we just have some standard pf stuff. set block-policy return just tells the firewall what to do with rejected packets. By default this is set to drop. I do not know if Bastille requires this change or if it’s simply the preference of the creator. I removed the next line, which corrects fragmented packets. This is usually a good idea, but it can interfere with our NFS server (if you want this option, it is possible to run pf inside of your jails and set it there…I think). set skip on lo simply says the rules don’t apply to the loopback device.

The next section is where the real magic happens. “NAT” stands for “network address translation.” Basically, we route incoming packets on ethernet (ext_if) to the local address of the jail and vice versa.

The final four lines just add some basic security and efficiency. Note that the last line ensures that SSH is available (so you haven’t locked yourself out in case you don’t use a monitor). I removed the block in all line because it locks out NFS. If this was a public facing server, I would want to leave it and then make a more specific rule to allow NFS (like what this configuration does with SSH). For the sake of security, I’ll probably extend this firewall configuration in the future.

4. Bootstrap a FreeBSD version and Create Jail

This step just assigns a version of FreeBSD to Bastille so you can use it to create your jails. Basically, this is your “template” and the jails are deviations from this template. This is what makes jails so lightweight and powerful. I use the word “template” in quotation marks to avoid confusing it with the actual Bastille template we’ll apply later.

Right now 13.1 is the latest stable FreeBSD version. In the future you may want to replace that with your current version.

bastille bootstrap 13.1-RELEASE update

Now we create our jail:

[email protected]:~ # bastille create test-jail 13.1-RELEASE 10.5.5.5

Tidbit for learners

This should be pretty obvious, but bastille is the command that summons Bastille, create tells it to create a new jail, test-jail is the name of this jail (it can be anything you like), and 10.5.5.5 is the IP address assigned to the jail. As mentioned before, the IP address can be anything you want as long as it fits the format of 10.x.x.x or 172.16.x.x (with the ‘x’ being any 1-3 digit numbers you want). You can also do 192.168.x.x, but that may conflict with addresses your router has assigned.

5. Test Jail

Bastille allows you to to run commands, modify your rc.conf file, or start/stop/restart services without entering the jails using the bastille commands cmd, sysrc, and service respectively. For example, to see the contents of your jail /usr/local/etc folder, you could type:

[email protected]:~ # bastille cmd test-jail ls /usr/local/etc

Personally, I prefer to just “enter” the jail and treat it like I’ve booted into a different operating system. To do this, use the console command:

[email protected]:/ # bastille console test-jail

When successful, you will now see that you are logged in under your jail:

[email protected]:~ #

You can now test your networking:

[email protected]:/ # fetch https://www.freebsd.org/images/beastie.png

This should download the beastie.png image and you should see if in whatever folder you’re in using the ls command.

It should be noted that not all of your networking is fully functional. For example:

[email protected]:~ # ping 1.1.1.1
ping: ssend socket: Operation not permitted

If don’t need these network services, don’t add them as they make your jail less secure. If you do, add the following line to the end of /etc/pf.conf (on the host system):

pass inet proto icmp icmp-type {echoreq unreach}

Then edit your jail.conf file. Mine is located here (replace red text with the appropriate values): /nexus0/bastille/jails/test-jail/jail.conf. Add the following text somewhere on a new line inside the curly brackets:

allow.raw_sockets;

Then restart the jail:

[email protected]:/ # service bastille restart

Conclusion

You can now use Bastille to create jails and run applications in your jails. You can isolate each major application in a jail, but I find it easiest to isolate tasks. For example, I would consider a webserver a task. You could isolate each part of the webserver—Apache, SQL, etc.—but that makes things messy. As you will see in upcoming articles, there are huge advantages to having each task isolated in its own jail. Personally, I find FreeBSD jails to be much easier to work with than Linux containers like Docker, especially using Bastille.

F

Back to Top