FreeBSD BHyve Setup Note
bhyve is FreeBSD's hypervisor. The native setup guide is at here. However, there exists a higher level wrapper called vm-bhyve available. I'm using that.
I'm following the installation guide in README. In a summary:
pkg install vm-bhyve
A few notes:
I created a zfs dataset
zroot/vm for storing the vms, and set
I followed the guide to name the vm switch
public. And I can see a network
vm-public was created for me:
vm-public: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
Choice of Linux distro and ISO
I initially want to use CentOS. However CentOS by default uses graphic installation, but I wasn't able to get that working (bhyeve seems to support VNC for UEFI graphic but I didn't get that working).
I had better luck with Ubuntu server. It offers text mode installation. And I was able to start and install it:
vm iso http://releases.ubuntu.com/18.04.4/ubuntu-18.04.4-live-server-amd64.iso
The security update at the end of Ubuntu installation did fail, so I chose "cancel update and reboot".
After reboot, Ubuntu server booted into grub prompt. This SO post is helpful. It means grub can't find root partition. So in grub prompt:
This started ubuntu normally. Then issue command to fix grub:
However after reboot, it stuck in grub prompt again...*
[WORKING SOLUTION] it was due to Ubuntu Server installs boot to 2nd partition, but with default bhyve grub looks for boot in 1st partition. See here.
The fix is to add:
into the VM's conf (
Now the VM can start normally.
Here's the VM config that works for Ubuntu Server 18.04.4:
Trying out using cloud images. The ubuntu server minimal is only 100MB+
pkg install qemu-utils # needed for using cloud images
Cloud image doesn't allow password login. Luckily vm-bhyve also supports cloud init. So you can inject SSH public keys:
pkg install cdrkit-genisoimage # required by cloud init
Cloud init is two part:
- the cloud-init service running inside the guest machine, reading data sources (e.g., from a mounted ISO system "seed.iso" to read init config.
- the preparation of the "seed.iso" to "inject" the init config. This is done on the host, before booting the guest VM for the first time.
At the moment, vm-bhyve doesn't support all the cloud-init functions. Its
support is basically upon
vm create command:
- it reads the cloud-init related configs it understands (very limited)
- dump them into
- create the
genisoimage -output ./seed.iso -volid cidata -joliet -rock .cloud-init/*
And cloud-init will be triggered the first time the VM is booted.
So to circumvent vm-bhyve's limitation, I:
- create a new VM
- create a
.cloud-initfolder with proper config files as I desire
- create the
- boot VM to trigger cloud-init
Cloud-init doc is hard to read, but can be found here. Specifically:
- user-data spec
- network config: https://cloudinit.readthedocs.io/en/latest/topics/network-config.html
To allow user password login, use a
passwd field is generated on another Linux machine, using
mkpasswd --method=SHA-512 --rounds=4096. See here.
A few tips:
- on FreeBSD, the content of the
seed.isocan be easily inspected by
tar xzf seed.iso(careful this dumps the content to pwd, so might do this in a new folder)
- the cloud-init operation can be seen on the boot message from the guest console, when the VM is booted for the first time. Subsequent boots doesn't seem to trigger cloud-init.
Some Networking Learning
ifconfig bridge create [bridge-name]
- Bridge basically connects multiple networks at MAC layer, i.e., forwarding traffic to the medium of the connected interfaces.