The cluster has 3 nodes for mon e storage.

Every node has one disk for OS and three disks for storage

• Creating the switch and attaching a tap device:

vde_switch -n 20 -mod 660 -group users -daemon --mgmt /tmp/swmgmt0 -s /run/vde.ctl0

vde_plug2tap -d -s /run/vde.ctl0 tap0

Disk creation for the node

qemu-img create -f qcow2 ubuntu-18.04.2-orig.qcow2 10G

After installing ubuntu on the VM I have installed Ceph ( apt-get -y install ceph-deploy ceph-common ceph-md ) before cloning to created the other nodes.

• OS installation on a VM

qemu-system-x86_64 -name ubuntu-18.04  -m 2G -cpu host -smp sockets=1,cores=2,threads=1  --enable-kvm -cdrom ubuntu-18.04.2-live-server-amd64.iso -drive file=ubuntu-18.04.2-orig.qcow2,if=virtio -device virtio-net-pci,netdev=net0,mac=92:b7:41:5f:a3:a1 -netdev vde ,id=net0,sock=/run/vde.ctl0   -monitor unix:/tmp/mon,server,nowait -vga qxl -spice port=5940,password=051ec34c  -daemonize

Once the OS has been installed on a VM, prepare it for leaving just a minimum adaptation once cloned for the other nodes.

• Change the IP ( vim /etc/netplan/50-cloud-init.yaml )
• Assign a default route ( vim /etc/netplan/50-cloud-init.yaml )
• Add DNS addresses ( echo -e "nameserver 195.186.4.111\nnameserver 195.185.1.11" > /etc/resolv.conf )
• Change the hostname ( /etc/hostname )
• Create an /etc/hosts with each node ip/name
• Assign a password to root ( echo -e "mypwd\nmypwd" |passwd ; echo sd-ceph-1 >/etc/hostname )
• Allow root ssh access
• Create ssh keys ( ssh-keygen -t ecdsa )
• Copy the host ssh keys to this VM ( ssh-copy-id 192.168.0.111)
• Install ntp ( apt-get -y install ntp )
• Install ceph ( apt-get -y install ceph-deploy ceph-common ceph-mds )
• apt-get remove cloud-init

Node creation

• After the disk creation it can be cloned to make the other nodes:

qemu-img convert -O qcow2 ubuntu-18.04.2-orig.qcow2 ubuntu-18.04.2-1.qcow2

qemu-img convert -O qcow2 ubuntu-18.04.2-orig.qcow2 ubuntu-18.04.2-2.qcow2

qemu-img convert -O qcow2 ubuntu-18.04.2-orig.qcow2 ubuntu-18.04.2-3.qcow2

qemu-img convert -O qcow2 ubuntu-18.04.2-orig.qcow2 ubuntu-18.04.2-host.qcow2

Additional disk creation within nodes:

qemu-img create -f qcow2 ceph1-disk1 20G
qemu-img create -f qcow2 ceph1-disk2 20G
qemu-img create -f qcow2 ceph1-disk3 20G
qemu-img create -f qcow2 ceph2-disk1 20G
qemu-img create -f qcow2 ceph2-disk2 20G
qemu-img create -f qcow2 ceph2-disk3 20G
qemu-img create -f qcow2 ceph3-disk1 20G
qemu-img create -f qcow2 ceph3-disk2 20G
qemu-img create -f qcow2 ceph3-disk3 20G

Starting the nodes one by one ( start the "testing VM" first and "ceph1" last ) , updating IP/hostname and rebooting:

qemu-system-x86_64 -name host -m 1G -cpu host -smp sockets=1,cores=1,threads=1 --enable-kvm -drive file=ubuntu-18.04.2-host.qcow2,if=virtio  -device virtio-net-pci,netdev=net0,mac=92:b7:41:5f:a3:a4 -netdev vde,id=net0,sock=/run/vde.ctl0 -monitor unix:/tmp/mon4,server,nowait -display none -daemonize

• Testing VM will take .110.

ssh 192.168.0.111 'echo testing-vm > /etc/hostname; sed -i s/192.168.0.111/192.168.0.110/ /etc/netplan/50-cloud-init.yaml ;sync;reboot'

To check

ssh 192.168.0.110 'uname -n;ip a |grep "/24";ip r|grep default'

qemu-system-x86_64 -name ceph2 -m 1G -cpu host -smp sockets=1,cores=1,threads=1 --enable-kvm -drive file=ubuntu-18.04.2-2.qcow2,if=virtio -drive file=ceph2-disk1.qcow2,if=virtio -drive file=ceph2-disk2.qcow2,if=virtio -drive file=ceph2-disk3.qcow2,if=virtio -device virtio-net-pci,netdev=net0,mac=92:b7:41:5f:a3:a2 -netdev vde,id=net0,sock=/run/vde.ctl0 -monitor unix:/tmp/mon2,server,nowait -display none -daemonize

• ceph2 will take .112

ssh 192.168.0.111 'echo sd-ceph-2 > /etc/hostname; sed -i s/192.168.0.111/192.168.0.112/ /etc/netplan/50-cloud-init.yaml ;sync;reboot'

qemu-system-x86_64 -name ceph3 -m 1G -cpu host -smp sockets=1,cores=1,threads=1 --enable-kvm -drive file=ubuntu-18.04.2-3.qcow2,if=virtio -drive file=ceph3-disk1.qcow2,if=virtio -drive file=ceph3-disk2.qcow2,if=virtio -drive file=ceph3-disk3.qcow2,if=virtio -device virtio-net-pci,netdev=net0,mac=92:b7:41:5f:a3:a3 -netdev vde,id=net0,sock=/run/vde.ctl0 -monitor unix:/tmp/mon3,server,nowait -display none -daemonize

• ceph3 will take .113

ssh 192.168.0.111 'echo sd-ceph-3 > /etc/hostname; sed -i s/192.168.0.111/192.168.0.113/ /etc/netplan/50-cloud-init.yaml ;sync;reboot'

• node ceph1 does not need any adjustment

qemu-system-x86_64 -name ceph1 -m 1G -cpu host -smp sockets=1,cores=1,threads=1 --enable-kvm -drive file=ubuntu-18.04.2-1.qcow2,if=virtio -drive file=ceph1-disk1.qcow2,if=virtio -drive file=ceph1-disk2.qcow2,if=virtio -drive file=ceph1-disk3.qcow2,if=virtio -device virtio-net-pci,netdev=net0,mac=92:b7:41:5f:a3:a1 -netdev vde,id=net0,sock=/run/vde.ctl0 -monitor unix:/tmp/mon1,server,nowait -display none -daemonize

node ceph1 does not need any adjustment

• For any problem accessing the nodes via ssh, start them with a spice server:

del "-display none"

add " -vga qxl -spice port=5940,password=xxxxxx"

Log to node sd-ceph-1

• copy the ssh keys on the other hosts

ssh-copy-id sd-ceph-2

ssh-copy-id sd-ceph-3

Deploy and install ceph

It is important to deploy from this directory

cd /etc/ceph

Add this line at the end of ceph.conf

public_network = 192.168.0.0/24

ceph-deploy new sd-ceph-1 sd-ceph-2 sd-ceph-3

ceph-deploy install sd-ceph-1 sd-ceph-2 sd-ceph-3

ceph-deploy mon create-initial

• Create OSDs in the nodes

sd-ceph-1

ceph-deploy osd create sd-ceph-1:vdb

ceph-deploy osd create sd-ceph-1:vdc

ceph-deploy osd create sd-ceph-1:vdd

sd-ceph-2

ceph-deploy osd create sd-ceph-2:vdb

ceph-deploy osd create sd-ceph-2:vdc

ceph-deploy osd create sd-ceph-2:vdd

sd-ceph-3

ceph-deploy osd create sd-ceph-3:vdb

ceph-deploy osd create sd-ceph-3:vdc

ceph-deploy osd create sd-ceph-3:vdd

• Test it with:

ceph -w detail

ceph health detail

ceph osd tree

If there are some OSDs down

e.g. OSD 7 is down

systemctl start -l ceph-osd@7

• Create the mgr:

ceph-deploy --overwrite-conf mgr create sd-ceph-1

• Create the metadata server

ceph-deploy mds create sd-ceph-2

ceph-deploy mds create sd-ceph-3

Create a pool

ceph osd pool create massi 128

• Check with

ceph osd lspools

Delete a pool

ceph tell mon.\* injectargs '--mon-allow-pool-delete=true'

ceph osd pool delete massi massi --yes-i-really-really-mean-it

For any problems with the conf file

cp /etc/ceph/ceph.conf /etc/ceph/ceph.conf-orig

then

ceph-deploy --overwrite-conf mds create sd-ceph-1

• Get the settings

ceph osd pool get rdb pg_num

ceph osd pool get rdb pgp_num

• Create a cephfs according to the previous settings

ceph osd pool create cephfs_data 128

ceph osd pool create cephfs_metadata 128

ceph fs new cephfs cephfs_metadata cephfs_data

• checking

ceph fs ls

ceph mds stat

ceph osd lspools

ceph df detail

Mounting the cephfs on another host

• Start the host:

qemu-system-x86_64 -name host -m 1G -cpu host -smp sockets=1,cores=1,threads=1 --enable-kvm -drive file=ubuntu-18.04.2-host.qcow2,if=virtio  -device virtio-net-pci,netdev=net0,mac=92:b7:41:5f:a3:a4 -netdev vde,id=net0,sock=/run/vde.ctl0 -monitor unix:/tmp/mon4,server,nowait -display none -daemonize

• set the hostname

echo bionic > /etc/hostname

• set a static ip:

edit /etc/netplan/50-cloud-init.yaml

• enable ssh root access

apt-get remove cloud-init

apt-get -y install ceph-deploy ceph-common ceph-mds

• Create a new directory to mount the cephfs:

mkdir /media/ceph

chmod 777 /media/ceph

• get the ceph key from node1:

ssh 192.168.0.111 'cat /etc/ceph/ceph.client.admin.keyring'

• mount the filesystem:

mount.ceph 192.168.0.111,192.168.0.112,192.168.0.113:/ /media/ceph/ -o name=admin,secret=AQBfWEBdN1RaLhAAjDx1MzhXGwfuh/0l32ZQog==

• check:

df -h /media/ceph/

Safely shutdown the cluster:

• From the manager node:

ceph osd set noout; ceph osd set nobackfill; ceph osd set norecover

Adapt the below script to poweroff all the nodes

for i in {2..3}; do ssh 192.168.0.11$i 'poweroff';sleep 10;done

Safely start all the cluster nodes:

• From the host where the VMs are running start storage/monitors
• From the manager node:

ceph osd unset noout; ceph osd unset nobackfill; ceph osd unset norecover