nft two nics firewall

max
linux network 
                                V S O K
+---------+-------------------------+-------------------------+----------------------+
|         |                         |                         |                      |
|         |                         |                         |                      |
|         |CID 3                    | CID 4                   | CID 5                | CID 6
|   +-----v----------+       +------v---------+     +---------v------+     +---------v------+
|   |                |       |                |     |                |     |                |
|   |     V M 1      |       |     V M 2      |     |     V M 3      |     |     V M 4      |
|   |                |       |                |     |                |     |                |
|   |                |       |                |     |     httpd      |     |                |
|   |                |       |                |     |                |     |                |
|   +----------------+       +----------------+     +----------------+     +----------------+
|       eth1|       |eth0            |eth0                 |eth0                  |eth0
|   10.0.5.1|       |10.0.2.1        |10.0.2.2             |10.0.2.3              |10.0.2.4
|  +----------------------------------------/run/vde.ctl0------------------------------------+
|  |                                                                                          |
|  |                                       V D E SWITCH                                       |
|  |                                                                                          |
|  +---------------------------------------------+------------------------------------+------+
|                                                |                                    |
|                                           tap0 |10.0.5.254                          |
|                                       +----------------+                            v
|                                       |                |                         swmgmt0
|     V S O K                           |    H O S T     |
+---------------------------------------+                +-------->  router
                                        |                |eth1
                                        |                |192.168.0.100
                                        +----------------+

VM1 is a gateway/firewall where eth0 is connected with the rest of the VMs and eth1 with the outside

  • VM1, VM2, VM3 and VM4 start with eth0 connected to the VDE switch ( vmstart2 script ) on the network 10.0.2.1/24
  • second nic ( eth1 ) hotplugged to VM1 via qemu monitor and placed on a different subnet ( IP 10.0.5.1 )
  • VM1 has ssh listening on port 2228 and drops connections for outside on port 22
  • VM1 is connected with the rest of the VMs through ssh port 22
  • VM3 has an https server that has to be reached from outside ( DNAT )
  • All the VMs should reach the ouside ( SNAT )
  • ncat is listening on port 2222 ( VM2 ), 2223 ( VM3) , 2224 ( VM4) to test dnat
  • Host network interface tap0 10.0.5.254
  • For testing and easy settings a vsock connection between the host and all the VMs enables access to all the VMs without passing through a network. In this way the VMs can be accessed regardless of any IP/firewall settings.
  • All the settings are pushed from the host through ssh
  • The counters are external and resettable
  • Enabling ip forward on VM1

sshe 10.0.5.1 'echo 1 > /proc/sys/net/ipv4/ip_forward'

  • Hotplugging the new nic to VM1

echo -e "netdev_add vde,id=net1,sock=/run/vde.ctl0 \n" | ncat -U /tmp/mon.1 2>/dev/null

echo -e "device_add virtio-net-pci,netdev=net1,id=virtio-net-pci.1 \n" | ncat -U /tmp/mon.1 2>/dev/null

  • Assigning the IP 10.0.5.1 to the new NIC

sshe 10.0.2.1 'ip a a 10.0.5.1/24 dev eth1;ip l set dev eth1 up

  • Adding port 2228 to ssh server

sshe 10.0.2.1 'echo -e "Port 22 \nPort 2228" >> /etc/ssh/sshd_config

  • Restarting the ssh server on VM1

vsock_cli 3 1961 -cmd '/etc/init.d/S50sshd restart'

  • Setting the default route to the first VM on the other VMs

for i in $(seq 2 4); do sshe 10.0.2.$i 'ip r del default 2>/dev/null ; ip r a default via 10.0.2.1';done

  • Starting some fake services on the VMs to test DNAT

sshe 10.0.2.2 "ncat -l -k -p 2222 -c 'uname -n; ip a |grep eth0$ ;exit 0' > /dev/null & "

sshe 10.0.2.3 "ncat -l -k -p 2223 -c 'uname -n; ip a |grep eth0$ ;exit 0' > /dev/null & "

sshe 10.0.2.4 "ncat -l -k -p 2224 -c 'uname -n; ip a |grep eth0$ ;exit 0' > /dev/null & "

  • Assigning the IP to the tap0 interface that is connected to the VM1

ip a flush dev tap0

ip a a 10.0.5.254/24 dev tap0

ip l set dev tap0 up

sshe is just ssh using the keys

sshe 10.0.5.1 -p 2228'
nft flush ruleset ;
nft define LAN  = { 10.0.2.1/24 } ;
nft add table ip filter ;
nft add counter ip filter global_counter ;
nft add counter ip filter ssh_counter ;
nft add counter ip filter http_counter ;
nft add chain ip filter input { type filter hook input priority 0\; policy accept\; } ;
nft add chain ip filter input_eth0  ;
nft add chain ip filter input_eth1  ;
nft add rule ip filter input iifname eth0 jump input_eth0 ;
nft add rule ip filter input iifname eth1 jump input_eth1 ;
nft add rule filter input ct state established,related accept ; 
nft add rule filter input iif lo accept ;
nft add rule filter input ct state invalid drop ;
nft add rule filter input  reject with icmp type port-unreachable  ;
nft add rule ip filter input tcp dport 8  accept  ;
nft add rule filter input_eth0 tcp dport http ct state new log prefix \"HTTP_IN \" counter name "http_counter" counter name "global_counter" accept ;
nft add rule filter input_eth0 ip saddr { 10.0.5.254, 10.0.2.2, 10.0.2.4 }  tcp dport 2228 ct state new log prefix \"FW-ACCESS-eth0 \"  accept ;
nft add rule filter input_eth0 tcp dport ssh ct state new  accept ;
nft add rule filter input_eth0 ip protocol icmp icmp type echo-request limit rate 1/second accept ;
nft add rule filter input_eth1 tcp dport {139, 445} drop ;
nft add rule filter input_eth1 tcp dport 80 accept ;
nft add rule filter input_eth1 tcp dport ssh log prefix \"SSH_DROPPED \" drop;
nft add rule filter input_eth1 ip saddr { 10.0.5.254, 10.0.2.2, 10.0.2.4 } tcp dport 2228 ct state new counter name "ssh_counter" counter name "global_counter" log prefix \"FW-ACC \" accept;
nft add rule filter input_eth1 ip protocol icmp icmp type echo-request limit rate 1/second accept ;
nft add chain ip filter forward { type filter hook forward priority 0\; policy accept\;}  ;
nft add chain ip filter output { type filter hook output priority 0\; policy accept\;}  ;
nft add rule ip filter output tcp dport 8  accept  ;
nft add table nat  ;
nft add counter nat NAT_counter ;
nft add chain nat prerouting { type nat hook prerouting priority 0 \; }  ;
nft add chain nat postrouting { type nat hook postrouting priority 100 \; }  ;
nft add rule nat postrouting ip saddr 10.0.2.0/24 oif eth1 snat 10.0.5.1 ;
nft add rule nat prerouting iif eth1 tcp dport { 80, 443, 8080 } ct state new log prefix \"HTTP_FWD \" counter name NAT_counter dnat 10.0.2.3:80 ;
nft add rule nat prerouting iif eth1 tcp dport 2222 counter name NAT_counter dnat 10.0.2.2 ;
nft add rule nat prerouting iif eth1 tcp dport 2223 counter name NAT_counter dnat 10.0.2.3 ;
nft add rule nat prerouting iif eth1 tcp dport 2224 counter name NAT_counter dnat 10.0.2.4 ;
nft add chain  ip filter input { policy drop\;  }
'
  • Creating an index.html file and starting an httpd server on VM3

vsock_cli 5 1961 -cmd 'echo -e "$(uname -n) $(ip a |grep eth0$)" > index.html; httpd'

Testing:

  • Connecting to the web server on VM3 ( 10.0.2.3)

wget -S -O - http://10.0.5.1 2>/dev/null

vm3 inet 10.0.2.3/24 scope global eth0

  • Connecting the fake services on VM2, VM3, VM4

for i in $(seq 2 4); do ncat --recv-only 10.0.5.1 222$i; done

vm2

inet 10.0.2.2/24 scope global eth0

vm3

inet 10.0.2.3/24 scope global eth0

vm4

inet 10.0.2.4/24 scope global eth0