Source
r/linuxadmin•
phaeew

LibVirt network configuration conflicts with Docker?

I’ve tried the typical GoogleFu and came up with a bunch of things, none of which helped...

I have a Debian server which was running Wheezy, dist-upgraded to Jessie, running a bunch of VMs through the KVM/QEMU approach. This sever has been running for 6+ years just fine.

Last weekend I put Docker on it to move services out of VMs and learn about containers. The Docker container I setup is working perfectly but I learned that my br0 interface isn’t passing external traffic.

  • guest ping router, fails
  • guest pings other guest, fails
  • guest pings hosts, works
  • host pings guest, works
  • host pings router, works

The above is true for static assigned IPs but because one guest provides DHCP for others, any with DHCP can’t communicate on ipv4 at all due to lacking an IP.

Anyway... is this something any of you have seen? I’ve rebuilt the bridge and reconfigured VMs to no avail.

I then upgraded to Stretch thinking that Docker or a dependency may have assumed I was on the current stable. This didn’t help but it did show me that I had 2 gateways in my interface definition file so I fixed that. No errors or warnings on boot now... still no traffic.

Edit to add resolution:

iptables -A FORWARD -p tcp —tcp-flags SYN,RST SYN -j TCPMSS —clamp-mss-to-pmtu

Docker messes up my iptables. As soon as I put this rule back, bridged traffic started flowing.

Posted Jan 28, 2018, 5:27 PM UTC · edited

2414
Guirlande•

Just a quick thought but have you checked your FW? I'm quite sure both well rewrite the FW chains after loading.

2
phaeewOP•

By FW, you’re just referring to iptables? There’s no new secret stretchy firewall?

I was under the impression that bridged VM nics didn’t need any additional iptables rules... from my searching, it seemed to only be necessary for port forwarding to isolated virtual networks...

If I’m wrong, this is definitely the problem. It drew my focus because firewall problems explain exactly why the middle player could see everything but nothing got through.

I also checked the /proc forwarding rules and those were all bad after installing docker so I fixed them but it didn’t help.

1
RulerOf•

Docker does screw around with iptables when you install it. It's a really big problem especially on systems that use firewall-cmd, like RHEL7 and its derivatives.

I ended up having to remove docker's rules from my system and manage firewall behavior myself. I found this page to be a helpful explanation of what was going on.

3
phaeewOP•

Mother of god! It was exactly as simple as I expected...

iptables -A FORWARD -p tcp —tcp-flags SYN,RST SYN -j TCPMSS —clamp-mss-to-pmtu

After running that, it all works.

Now I just need to make sure Docker doesn’t break it on boot... which I think is going to happen.

3
RulerOf•

After running that, it all works.

Yay.

Mother of god! It was exactly as simple as I expected...

iptables -A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu

I'm glad you think iptables is simple... I'll go hide behind firewall-cmd now.

Now I just need to make sure Docker doesn’t break it on boot... which I think is going to happen.

So long as your iptables rules are persistent, I would suggest telling docker to stop molesting your chains via /usr/bin/dockerd --iptables=false in whatever autostart system Debian uses, rebooting the server, and then adding whatever iptables rules you need to add in order to restore docker container connectivity to the outside world—probably a MASQUERADE rule on docker0.

2
antiphase_•

If you’re having to clamp TCP MSS it points at an MTU problem somewhere else on your network, or you have a shit ISP and you should be doing this on your edge router. Can also be cause by breaking Path MTU Discovery by inappropriate filtering of ICMP.

2
RaunchyBushrabbit•

Docker defines one or more virtual networks straight out of the box, one of these might conflict with a network you already have in use, check the default socker networks and netmasks against your current setup.

Source: had have this happen at several customers.

2
natermer•

You generally don't want to mix docker with a VM host.

Docker messes up my iptables. As soon as I put this rule back, bridged traffic started flowing.

Docker depends on port forwarding in the firewall to expose hosted services to the internet. It wants to manage those things automatically. So unless you put a lot of effort into it fixing things for docker will break your VMs and visa versa.

I would get rid of Docker on the physical system and put it into one of the virtual machines. This is the easy approach and is much easier to maintain.

Personally I find docker and virtual machines as complimentary technologies.

The point of these things is to make installing and managing software on Linux easier. Running docker on a VM is easier then running it on a physical host. Especially if you have a mixture of applications that benefit from the docker approach and others that do not.

...............

As a general approach...

Pro tip for great success:

Keep the OS that runs directly on the physical host as absolutely minimal as possible. It should be treated much more like a network appliance then a Unix server. Consider any time you have to log in and/or run sudo as a failure in system engineering. Keep as much as the complexity and configuration as possible in Virtual Machines.

Virtual machines are easier to manage then physicals and it's much easier to manage OS images and automation like Ansible for virtual machines then it is for physicals. You can blow away and recreate VMs extremely quickly... it's not so easy for physical host OSes.

The overhead is negligible and unless you are doing something that requires massive amount of I/O performance for local drives (or whatever) then you won't see any real measurable difference in most real-world applications.

The except for this is development systems and personal workstations.

2
phaeewOP•

This is the conclusion I’m slowly coming to :)

I’m one of those learn-by-doing sorts. I went to “every service aside from V host runs on a VM” paradigm a decade ago and have stuck with it until this foray into containers.

Since I waited so long, I expected Docker to be more like LibVirt or VMWare. Though in docker’s defense, I’d never run libvirt and virtualbox on the same system.

Your sage advice is about to be followed!

The one thing I was hoping to speed up with docker was a file transcode process that is just awful at the moment. The VM barely hits 70% processor usage and can’t maintain it for long because of the poor IO. Mostly due to the age of the system but also because of the VM configuration and guest OS.

I was hoping to reduce the layers between the transcode process and the hardware... may make sense to just offload it to a purpose-built box with SSDs and skip the container/VM entirely for performance.

1
natermer•

If you want you can dedicate a full disk to a VM if you have something that requires really good disk performance. There are different ways to do it that reduce the overhead.

2
Nate8199•

I made a VM just for a docker host to avoid this. Also check out portainer, made docker way nicer to manage.

2
phaeewOP•

Brilliant!

1
frebib•

Docker adds the DOCKER-USER chain which is the first rule in the FORWARD chain. Add your custom forward rules to that table instead of FORWARD

2
phaeewOP•

Thanks for the additional details. This post has been unimaginably helpful, above and beyond the “fix my problem” resolution that I found early. All the “right way” alternatives are so useful!

1
keror0•

Soultion: Stop docker from adding rules to iptables

1