Is the best way to prevent hosts in an isolated network from communicating with the host at all to just bring down the network's assigned interface on the host? I just need the VMs in the network to communicate with each other and not be able to communicate with the host whatsoever.
I tried deleting the bridge from the XML file in virsh but it just returned upon the network being brought back up.
Posted May 22, 2021, 9:55 PM UTC · edited
I know in Oracle VirtualBox you can create a host-only adaptor or something and I think that network was able to achieve it, although now that I think about it the host would at least get its own ip as well, but idk if that discards your usecase.
Maybe you can add a network and then nmcli connection down from host, can that work?
I haven't tried this myself but there's plenty of relevant discussion on the web. Have you seen this for example? https://unix.stackexchange.com/questions/551105/kvm-isolation-between-different-nat-networks
Are the VMs all running on the same host? If so, you can create a network bridge on the host without connection to any physical device, and then add a tuntap interface (software/emulated network card) for each VM, and they can each communicate over the bridge. I don't know if qemu allows you to pass multiple VMs the same tuntap interface, but you could try it.
My setup bridges the virtual NIC to the physical NIC, so yours would be different in that you wouldn't connect the physical NIC. Anyway, I added this to the VM configuration qemu arguments (not sure about virsh):
-netdev tap,id=netid,ifname=taplan,script=no,downscript=no
-device e1000,netdev=netid,mac=52:54:00:01:02:03
This is how I set up the tuntap virtual device ("taplan"), the bridge ("br0"), and connected it to the physical network device on the host ("enp4s0"). Also note that I turned off the network-manager in favor of my manual configuration below (I'm on Debian - your config locations may differ).
# cat /etc/network/interfaces.d/taplan
auto taplan
iface taplan inet manual
up ip tuntap add taplan mode tap
post-up ip link set taplan master br0
post-up ip addr flush dev taplan
#post-up ip link set address 52:54:00:01:02:03 dev taplan
post-up ip link set address 52:54:00:00:00:00 dev br0
post-up ip link set dev taplan up
down ip link del dev taplan
# cat /etc/network/interfaces.d/br0
auto br0
iface br0 inet static
address 192.168.0.100/24
broadcast 192.168.0.255
gateway 192.168.0.1
bridge_ports enp4s0 taplan # connects physical device enp4s0 to virtual nic
bridge_stp off
bridge_fd 0
bridge_maxwait 0
post-up ip link set address 52:54:00:00:00:00 dev br0
post-up ip addr flush dev enp4s0
State when no VM is on:
# ip a
2: enp4s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master br0 state UP group default qlen 1000
link/ether b4:2e:99:f0:a7:03 brd ff:ff:ff:ff:ff:ff
3: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 52:54:00:00:00:00 brd ff:ff:ff:ff:ff:ff
inet 192.168.0.100/24 brd 192.168.0.255 scope global br0
valid_lft forever preferred_lft forever
4: taplan: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast master br0 state DOWN group default qlen 1000
link/ether da:a9:3a:20:c5:0b brd ff:ff:ff:ff:ff:ff
State when the VM is on and connected:
2: enp4s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master br0 state UP group default qlen 1000
link/ether b4:2e:99:f0:a7:03 brd ff:ff:ff:ff:ff:ff
3: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 52:54:00:00:00:00 brd ff:ff:ff:ff:ff:ff
inet 192.168.0.100/24 brd 192.168.0.255 scope global br0
valid_lft forever preferred_lft forever
4: taplan: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master br0 state UP group default qlen 1000
link/ether da:a9:3a:20:c5:0b brd ff:ff:ff:ff:ff:ff
Great advanced question. I hope someone with more knowledge will jump into the conversation here, but I suspect you should go straight to the KVM/qemu and maybe libvirt mailing lists, because that's where the expertise is. Reddit is basically teenager daycare. The days of the top sub being r/programming are long long long gone, and this sub is tiny anyway.
Isolating VMs in qemu/kvm on linux is difficult. The network stack wasn't designed for it, but there's been a lot of work in the last ten years or so to make it better.
I think it's still best practice to use a separate physical network interface for host/hypervisor management. It's just really really hard to isolate the host's bridge interface. If your host has an IP interface on the same bridge VMs are on, there's going to be some crazy way those VMs will be able to mess with it. Just use a separate physical interface.
Making your host system NOT be a router (example: net.ipv4.ip_forward = 0) really helps, but that limits you to certain scenarios.
Personally I hate libvirt's built in network filtering. It's garbage. So I wrote my own scripts via /etc/libvirt/hooks. Unfortunately I can't share that code publicly.
Actually re-reading your post, you sound kind of novice. You need to be putting these VMs into a bridge, and that bridge should be isolated on the hypervisor. You need to learn the brctl command, notably show and showmacs