FTB Evolution keeps dropping conenction on LAN

I am currently playing FTB Evolution, base in The Void having progressed so far with the pack.

The setup: Client is Kubuntu 24.04 with Prism Launcher having 6/8GB RAM allocated (tried both), i5 13th Gen Laptop
Server: LAN server on another Debian 13 Workstation, allocated 12GB RAM, i5 8500U. Minecraft run in itzg's Docker image. It's fully solo, I just want to delegate the heavy server work to another machine to not overload my laptop.

I basically expose the minecraft port so the client can connect to the server properly, within LAN. However today I am facing an issue where my client just randomly time out. Basically interaction halted for like 5-7s on client side, the world looks frozen on client side, the game itself is not frozen. After that it kicks me simply for Time out. This happens very randomly, maybe once per 30min, sometimes also twice in 5min.

Server log also does not report anything about "Cant keep up" either. Any message sent during the "frozen" phase from the client can be seen immediately from server log. The modpack is also using so much ram that server always hover around 12GB while client at 10GB.

Output of spark healthreport is as following. eth0 tx is always all time high while rx is always low like that. How can I possibly try to debug this further? My theory is that maybe the packet size is too big and caused some kind of hiccup in the network.

/spark healthreport --network

> TPS from last 5s, 10s, 1m, 5m, 15m:
    20.0, *20.0, 20.0, 20.0, 20.0
> Tick durations (min/med/95%ile/max ms) from last 10s, 1m:
    7.1/20.5/33.7/54.0; 6.7/21.4/31.7/60.2
> CPU usage from last 10s, 1m, 15m:
    11%, 11%, 13%  (system)
    10%, 10%, 12%  (process)
> Memory usage:
    2.8 GB / 12.0 GB   (23%)
    [┃┃┃┃┃┃┃┃┃┃┃┃┃┃╻╻╻╻╻╻╻╻╻╻╻╻╻╻╻╻╻╻╻╻╻╻╻╻╻╻╻╻╻╻╻╻╻╻╻╻╻╻╻╻╻╻╻╻╻╻]
> Network usage: (system, last 15m)
    36.0 bytes/s / 0 pps (lo rx)
    36.0 bytes/s / 0 pps (lo tx)
    2.3 KB/s / 31 pps (eth0 rx)
    470.8 KB/s / 32 pps (eth0 tx)
> Disk usage:
    101.5 GB / 451.8 GB   (22%)
    [┃┃┃┃┃┃┃┃┃┃┃┃┃╻╻╻╻╻╻╻╻╻╻╻╻╻╻╻╻╻╻╻╻╻╻╻╻╻╻╻╻╻╻╻╻╻╻╻╻╻╻╻╻╻╻╻╻╻╻╻]