Set up ezjail

Follow this doc:

clone lo0 to lo1 in /etc/rc.conf:

cloned_interfaces="lo1"

To create it without reboot: service netif cloneup.

Install ezjail

cd /usr/ports/sysutils/ezjail & make install clean

Enable ezjail in rc.conf: ezjail_enable="YES"

Start ezjail: service ezjail start.

To setup the base environment: ezjail-admin install -p.

Copy host's resolv.conf to jail's template so each newly created jail is able to resolve domain names:

host> cp /etc/resolv.conf /usr/jails/newjail/etc/

Networking

Conventional way

Each jail must be asigned an IP. Traditionally in ezjail, this is done as part of jail creation:

ezjail-admin create dnsjail 'lo1|127.0.1.1,em0|192.168.1.50'

This would assign dnsjail a private IP 127.0.1.1 on lo1, and an aliased IP 192.168.1.50. The latter is an alias IP the host OS creates. You can see it with ifconfig em0 in the host. Also the host is accessible via this IP in its LAN. For more information about IP aliasing, see virtual host.

Then in the jail, edit hosts to change 127.0.0.1 to 127.0.1.1 and add the jail's hostname to each entry. This is essential for it to access internet.

But this is hard to manage, each network-talking jail needs an IP and you need to configure router for each of these IPs. Rather, it would be good if the jails can do networking through host's IP address. That's what the next section is about.

Networking through host's IP

Inspired by this post. It's done through NAT. You still need a pool of IPs but they don't need to be aliases to host's IP.

In /etc/rc.conf, add:

cloned_interfaces="lo1"
ipv4_addrs_lo1="192.168.60.1-9/29"

Note the range 192.168.60.1 ~ 192.168.60.9. I previously used 192.168.0.1-9 and I lost network connection to my host.

Now restart netif: host> service netif restart. And you should see the newly created IPs.

em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=4219b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,WOL_MAGIC,VLAN_HWTSO>
    ether ...
    inet 192.168.0.7 netmask 0xffffff00 broadcast 192.168.0.255 
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
    media: Ethernet autoselect (100baseTX <full-duplex>)
    status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
    options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
    inet6 ::1 prefixlen 128 
    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2 
    inet 127.0.0.1 netmask 0xff000000 
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
    groups: lo 
lo1: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
    options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
    inet 192.168.60.1 netmask 0xfffffff8 
    inet 192.168.60.2 netmask 0xffffffff 
    inet 192.168.60.3 netmask 0xffffffff 
    inet 192.168.60.4 netmask 0xffffffff 
    inet 192.168.60.5 netmask 0xffffffff 
    inet 192.168.60.6 netmask 0xffffffff 
    inet 192.168.60.7 netmask 0xffffffff 
    inet 192.168.60.8 netmask 0xffffffff 
    inet 192.168.60.9 netmask 0xffffffff 
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
    groups: lo

Now we use pf to map traffics to and from jails.

Enable pf by adding to rc.conf: pf_enable="YES". Edit /etc/pf.conf:

# Public IP address
IP_PUB="<host's public IP>"

# Packet normalization
scrub in all

# Allow outbound connections from within the jails
nat on em0 from lo1:network to any -> (em0)

# webserver jail at 192.168.60.2
rdr on em0 proto tcp from any to $IP_PUB port 443 -> 192.168.60.2
rdr on em0 proto tcp from any to $IP_PUB port 80 -> 192.168.60.2
# .. or map jail's host's 80 to jail's 8080:
# rdr on em0 proto tcp from any to $IP_PUB port 80 -> 192.168.60.2 port 8080

# mailserver jail at 192.168.60.3
rdr on em0 proto tcp from any to $IP_PUB port 25 -> 192.168.60.3  
rdr on em0 proto tcp from any to $IP_PUB port 587 -> 192.168.60.3  
rdr on em0 proto tcp from any to $IP_PUB port 143 -> 192.168.60.3  
rdr on em0 proto tcp from any to $IP_PUB port 993 -> 192.168.60.3

Start pf: host> service pf start

Now creating a jail is simply:

ezjail-admin create <jail_name> <IP>

where IP is one of those of lo1's newly created, e.g., 192.168.60.2.

We can set up another interface lo2 but without configuring NAT for its network, in that case the jail is restricted to LAN access only:

cloned_interfaces="lo1 lo2"
ipv4_addrs_lo1="192.168.60.1-9/29"  # Set up NAT for them
ipv4_addrs_lo2="192.168.70.1-9/29"  # Don't set up NAT for them

I observed start of jail with LAN-only access is slower, maybe due to services requiring internet timed out during start.

Jail accessing file system outside of jail

This can be done by nullfs_mount. Basically by mounting a part of the host file system under the jail's root:

mkdir /usr/jails/<jail_name>/data
mount -t nullfs -o ro /data /usr/jails/<jail_name>/data

Alternatively, add this to the jail-specific fstab at: /etc/fstab.<jail_name>:

/data /usr/jails/<jail_name>/data nullfs ro

However there's a [bug I'm currently investigating][nullfs_issue], where the mount yields inconsistent subfolders.

Working with jails:

  • List: jls [-v]
  • Start/stop: ezjail-admin <start|stop|restart> <jail_name>
  • delete: ezjail-admin delete [-w] <jail_name>
  • ezjail's per-jail configuration is in directory /usr/local/etc/ezjail
  • ezjail's per-jail root is in /usr/jails/<jail_name> directory. Here you can modify jail's settings that's created at creation, e.g., the IP.
  • In jails you can't use ping to test network connection, instead, use telnet google.com 80.
  • Root of a jail is at /usr/jails/<jail_name>
  • Jail-specific fstab are at /etc/fstab.<jail_name>

Accessing mounted file systems from inside jail

Host file system can be mounted to jail by modifying /etc/fstab.<jail_name>. Inside the jail, the mounted file system has the same ACL as in the host. But the owner/group are shown as IDs. If a user inside jail (e.g., www) wants to access the file system, we must first create a user/group inside jail with the corresponding owner/group ID. And give the target jail user the corresponding ACL inside the jail. Example:

# /tmp/foo/ has 333:8000
jail> pw groupadd foo_group -g 8000  # create foo_group with ID 8000 inside jail
jail> pw groupmod www -m foo_group   # add jail user `www` to foo_group

References

Networking troubleshooting

Routing rable

host> netstat -r
Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
default            192.168.0.1        UGS         em0
hostname-bsd       link#2             UH          lo0
192.168.0.0/24     link#1             U           em0
192.168.0.7        link#1             UHS         lo0
192.168.60.1       link#3             UH          lo1
192.168.60.2       link#3             UH          lo1
192.168.60.3       link#3             UH          lo1
192.168.60.4       link#3             UH          lo1
192.168.60.5       link#3             UH          lo1
192.168.60.6       link#3             UH          lo1
192.168.60.7       link#3             UH          lo1
192.168.60.8       link#3             UH          lo1
192.168.60.9       link#3             UH          lo1

Internet6:
Destination        Gateway            Flags     Netif Expire
::/96              hostname-bsd       UGRS        lo0
hostname-bsd       link#2             UH          lo0
::ffff:0.0.0.0/96  hostname-bsd       UGRS        lo0
fe80::/10          hostname-bsd       UGRS        lo0
fe80::%lo0/64      link#2             U           lo0
fe80::1%lo0        link#2             UHS         lo0
ff02::/16          hostname-bsd       UGRS        lo0