This is a full automated build of a virtual LXD cluster using KVM and my vmstart script.
Ref: https://linuxcontainers.org/lxd/docs/master/preseed
vmstart needs just qemu/KVM and some kernel modules support, it is very lightweight: 34K!
Click here to show the options
- –path or -p: This is the first thing to do to set/reset the full path of the directory that you want to use for this project.
- –start or -s: To start the VMs.
- –kill or -k: To kill all the running VMs ( same as –off or –stop )
- –check or -c: To check hostname/IP of each VM
- –vsock or -v: To execute commands on the VMs with vsock.
- –nic or -n: To add a nic to a VM.
- –cpu or -cpu: To hotplug a CPU.
- –mem or -mem: To hotplug RAM.
- –disk or -d: To hotplug disks.
- –usb or -u: To attach an usb device.
- –migr or -mg: To migrate VMs between nodes.
- –data or -db: To check the nodes/VMs database.
- –img or -i: To create an image of the running VM.
- –cmd or -cm: To read a list of commands from a cmd file and pass them to a VM.
- –help or -h: This help.
- –help-full: The extend help ( same as -H).
I have compiled LXD against the MUSL libraries and imported binaries and a couple of libraries on a Linux Alpine VM.
I have also added a small binary and a script to support accessing through vsock and a script to read the qemu_fw entries at boot.
This customised image can be downloaded here: https://bollati/info/tiny/alpine_lxd.img
The new scripts and binaries are:
/etc/local.d/vmstart.start
/usr/bin/vsocksrv
/usr/bin/vsock_cli
/usr/bin/vsock_srv
lxd stuff on /usr/local/bin/ and /usr/lib/
vmstart can read a simple text file containing one or more instructions per line and execute either on a VM or on the host, to let it execute on the host just put "- " at the start.
The instructions that should be executed locally will be execute first despite the order on the text file.
To deploy the VM cluster we need just the alpine image
To deploy the LXD cluster we need:
- Two .cmd files that contain the instructions for vmstart
- Two preseeded file: One for the first node and another one for the other nodes
- A small set of rules for the firewall if we want the containers access outside
alpine-lxd.img
lxdcluster1.cmd
lxdcluster2.cmd
nat.nft
preseed_nodes
preseed_vm1
The .cmd files being pure text are easy to interpreter so I not reporting here the sequence of commands but just the files content.
First set of commands
Click here to show lxdcluster1 content
- - export NODES="5"
- - echo -e "alpine-lxd.img\n5\n"| vmstart -s
- - echo -e "\n\t\t wating 10 seconds for the VMs to boot\n"
- - sleep 10
- - clear
- - scp -i id_vm preseed_vm1 10.0.3.1:/tmp/
- -sshe 10.0.3.1 "sh -c '/usr/local/bin/lxd –group wheel 2>/dev/null &>/dev/null &'"
- - sleep 6
- -sshe 10.0.3.1 "sh -c 'cat /tmp/preseed_vm1 | /usr/local/bin/lxd init –preseed 2>/dev/null &>/dev/null &'"
- - echo -e "\n Please start udhcpd on host, attached to br0 range 10.0.30.2 10.0.30.240"
- - sshe -f 10.0.3.1 'sleep 1 && ip a flush dev eth0 && ip r a default via 10.0.3.254 &>/dev/null &'
- - (sleep 10 && scp -T -i id_vm 10.0.3.1:/var/lib/lxd/cluster.crt . && for i in $(seq 2 "$NODES"); do echo -e "$i\n lxdcluster2.cmd\n" | vmstart –cmd;done 2>/dev/null &>/dev/null &)
- - echo -e "\n\t You can watch the background installation progress with: \n\n\t watch sshe '10.0.3.1 /usr/local/bin/lxc cluster ls' \n"
- - echo -e "\n\t A tmux splitted window with the cluster and instances status should be accessible with tmux a \n"
- service cgmanager start
- /usr/local/bin/lxc profile set default cluster.evacuate=migrate
- /usr/local/bin/lxc cluster ls
- ip link add mcvlan1 link eth0 type macvlan mode bridge
- ip a a 10.0.3.1/24 dev mcvlan1
- ip l set dev mcvlan1 up
- ip r a default via 10.0.3.254
Second set of commands
Click here to show lxdcluster2 content
- - for i in $(seq 2 "$NODES"); do scp -q -i id_vm {preseed_nodes,cluster.crt} 10.0.3."$i":/tmp/ ; done
- - if ! pidof tmux ; then tmux new -ds LXD 'sshe 10.0.3.1'; tmux split-window -v 'sshe 10.0.3.1' ;tmux set -g pane-border-status top ;tmux select-pane -t0 -T "Cluster Status"; tmux select-pane -t1 -T "Instances Status" ;tmux send -t0 'watch -n6 lxc cluster ls' ENTER ; tmux send -t1 'watch -n6 lxc ls' ENTER ;tmux split-window -h 'sshe 10.0.3.1';tmux select-pane -t2 -T "Operations";tmux send -t2 clear ENTER; fi
- service cgmanager start
- (/usr/local/bin/lxd –group wheel 2>/dev/null &>/dev/null &)
- sleep 8
- cp /tmp/cluster.crt /var/lib/lxd/
- cat /tmp/preseed_nodes | sed -e "s/NAME/"node
uname -n |grep -o [0-9]"/; s/IP/"ip a |grep -o 10.0.3.[0-9]"/" |/usr/local/bin/lxd init –preseed
- /usr/local/bin/lxc cluster ls
First preseeded file ( first node )
Click here to show preseed_vm1 content
- cluster:
- enabled: true
- server_name: NAME
- server_address: IP:8443
- cluster_address: 10.0.3.1:8443
- cluster_certificate_path: /var/lib/lxd/cluster.crt
- cluster_password: my_password
- member_config:
- - entity: storage-pool
- name: default
- driver: dir
- profiles:
- - config: {}
Second preseeded file ( all the other nodes )
Click here to show preseed_nodes content
- cluster:
- enabled: true
- server_name: NAME
- server_address: IP:8443
- cluster_address: 10.0.3.1:8443
- cluster_certificate_path: /var/lib/lxd/cluster.crt
- cluster_password: lxd1961
- member_config:
- - entity: storage-pool
- name: default
- driver: dir
- profiles:
- - config: {}
vmstart needs to be initialised to set the working directory, the database and the ssh keys:
vmstart -p Insert node name => done
Insert node IP => done
- Close with CTRL+c when it asks for the image
- Now we launch vmstart again with –cmd to execute the first set of commands:
vmstart --cmd Enter the VM number => 1
Found the following cmd files: lxdcluster1.cmd lxdcluster2.cmd
If everything has worked fine you should see the firs node on the cluster:
+-------+-----------------------+----------+--------------+----------------+-------------+--------+-------------------+
| NAME | URL | DATABASE | ARCHITECTURE | FAILURE DOMAIN | DESCRIPTION | STATE | MESSAGE |
+-------+-----------------------+----------+--------------+----------------+-------------+--------+-------------------+
| node1 | https://10.0.3.1:8443 | YES | x86_64 | default | | ONLINE | Fully operational |
+-------+-----------------------+----------+--------------+----------------+-------------+--------+-------------------+
On the output there is a suggestion to launch a dhcp server to provide an IP to the containers, udhcpd comes with busybox and can be set on the fly without touching the system dhcp server.
If you have tmux a splitted window with the cluster and instances informations are displayed.
There is also another Operation window to play with the cluster.
See below how the cluster looks with the LXD dashboard ( https://lxdware.com/ )