Docker container zombie- cannot be killed
I've been struggling for a couple weeks with weird docker behavior. I originaly noticted uncorrectable errors constantly jumping on an SSD, and thought maybe i had a failing drive and some corruption, but now i've replaced the drive, rebuilt cache, and still have had docker crash/corrupt on me.
I think I'm finally catching the problem early enough to see more data. Previously, i wouldn't notice it until every container was crashed, unable to restart, and i had to delete the docker vdisk, then could rebuild and restart. Today, i finally caught that qbittorrent crashed into this state while other containers were still working, so I was able to check the logs and try a few more things instead of nuke and rebuild, since i wasnt suddenly missing all my services.
The container logs are pages and pages of Http::RequestParser::ParseResult Http::RequestParser::doParse(const QByteArray&) message body parsing error
bool Http::RequestParser::parsePostMessage(const QByteArray&) unknown content type: "application/x-binary"
spaming endlessly, until
Catching signal: SIGTERM
Exiting cleanly
Except that last part is a lie, the container still shows running:
docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
92849540237d lscr.io/linuxserver/qbittorrent:4.5.5 "/init" 6 days ago Up 6 days qbittorrent
i cannot kill it, rm it, or do anything with it- it gives me the following error :
root@Vault:~# docker kill 92849540237d
Error response from daemon: Cannot kill container: 92849540237d: tried to kill container, but did not receive an exit event
I've even tried shutting down the whole docker service and restarting it- it gives me an error that 'the docker will not die' but then appears to stop the service. But on restart, the same container is there again.
I'm baffled where to go from here. I can delete the vdisk image and start over- again, but would suspect it will crash again, since this is the third such crash since i replaced the disk i originally thought to be the problem.
I've googled around, but haven't had any lucky hits- looks like the error in the logs might be api related, but nothing to tell me why it would freeze the container so hard it persists through service stops. I've tried enabling only critical dockers. It seems like it'll crash within 48 hours if i bring my full containers i was running before this all began, but even running a basic stack (*arrs, qbit, ABS) it seems like i can't make it a full week before things start breaking.
Any ideas what else i can pick through before nuking and rebuilding? Or what i can monitor to try to catch when the issue first starts, before a container locks up?
Posted Nov 2, 2023, 1:29 AM UTC