What is the bottleneck that causes me framedrops in lossless and high-quality video?

In both my own FFV1 files and some high-quality HEVC files, I get many frame drops, particularly with 4K video, but also with smaller video if it is high-quality enough. I'm not sure what causes them, however, as my disk usage goes to zero after they cache, yet they still persist, and my CPU usage never actually reaches 100%. Some reference FFV1 files with this issue of my own making (so therefore I am authorised to distribute them):

https://happysmash27.me/Upload/Renders/Lossless_Video/Goatsquirrel_rotate3.mkv (3.8 GB)

https://happysmash27.me/Upload/Renders/Lossless_Video/Provo_de_Fajro.mkv (299MB)

What could be causing this bottleneck? Nothing in particular seems to stand out as causing it.