Playing "any" video format in Godot via FFmpeg
Hi everyone.
I haven't really shown this anywhere yet, so quick intro first. I've been building my own rpg/visual novel engine on top of Godot for about a year. I wanted to make something between an RPG and a visual novel, and the tools I tried didn't fit or were frustrating, so I started building my own and it grew into a whole engine. Most of it is data driven and authored through a custom editor/IDE, so I get to spend time writing the actual game instead of constantly wiring up systems. Though, in truth, ive ended up spending way more time on this than id have spend on just building the vertical slice i was initially planning lol.
I'll be posting bits of it as I go, but I wanted to start with one piece I think is useful on its own even if you're not building anything this big: getting video to actually play in Godot.
The problem
Out of the box Godot only plays Theora .ogv. I wanted animated backgrounds, video CGs, moving portraits, that kind of thing, and the ogv route means converting everything with a fussy encoder, files come out big and rough, no transparency, and gifs basically don't work. I didn't want to build a whole game around that, so I wrote my own video playback on top of FFmpeg.
The main idea: a video is just a texture that changes
Godot's VideoStreamPlayer is its own thing and it's stuck on Theora (something about h264/h265 licensing). But really a video is just a new image every frame. So if I can get each frame out as an RGBA image in memory, I can put it on a normal TextureRect, the same node you'd use for a still image. That one idea is what the whole approach hangs on: anywhere the engine already shows an image, it can show a video or a gif instead, and nothing downstream has to change.
So the real problem becomes "turn any file into a stream of RGBA frames", and that's what FFmpeg handles.
Why FFmpeg, and why I call it directly
FFmpeg can decode pretty much anything, which is the whole reason to use it. The question was how to actually talk to it. You can run ffmpeg.exe as a separate program and read what it spits out, which works, but then you're piping frames between processes and dealing with temp files and lag. So instead I built a little C library that links FFmpeg's decode stuff directly, and load that from C# when the game starts. Every frame comes back in memory ready to go, nothing hits the disk.
The tradeoff is you ship the FFmpeg dlls yourself and write a bit of C glue, and you have to load those dlls in the right order by hand, because they depend on each other and nothing works that out for you. That was the first thing that got me, took embarrassingly long to figure out.
Getting a frame on screen
The decode side is fairly standard: open the file, find the video stream, pull frames out. The one catch is video frames come out in YUV and Godot wants RGBA, so there's a colour conversion step in the middle (swscale does it). Then I wrap that RGBA in a Godot image and upload it as a texture each frame.
Playing it at the right speed
If you just decode in a loop it runs as fast as the cpu can go, so the video plays fast. So I drive it off the clip's actual framerate instead: each frame I work out how much time has passed, how many frames that is, and jump to the right one. The annoying bit is if the game stutters for a second you suddenly have a pile of frames to catch up on, and if you blindly decode all of them you turn one stutter into a full freeze. So I cap it at a few catch-up frames per tick and drop the rest. There's a speed option on top so you can slow a clip down or speed it up on purpose.
Audio
Godot doesn't give you a "here's some raw audio, play it" function. What it does have is AudioStreamGenerator, which lets you push your own samples into the mixer. So for clips with sound I decode the audio separately, resample it to what Godot wants (stereo float at 48000), and feed it into the generator in time with the video. Getting the two to line up, since they decode independently, took a while.
Keeping it out of the way when writing
None of this is stuff I want to think about while actually making the game. So it all hides behind one thing I call a media reference: a file path plus a couple options (loop, muted, speed, a fallback image). When the engine loads one it looks at the file and picks how to handle it. Plain png or jpg goes through Godot's normal loading. A folder of numbered frames becomes an image sequence. Anything else, gif, webp, mp4, webm, goes through FFmpeg. From my side I just point a field at a file and forget about it. On the IDE side, i just select what ever file i want, image, video, docent matter. The controls for that type of media are then automatically shown.
Image from the IDE, this is the timeline editor for dialogue etc.
Nice side effect is every field that used to take an image now takes video for free, backgrounds, CGs, character art, portraits, even the map image. And if a file fails to decode it falls back to the still image you set, so a bad file gives you a frozen frame instead of crashing the scene.
Where it's at
Its Windows only right now, the FFmpeg builds I bundle are win64. Audio and video are on separate decode handles at the moment which I want to clean up. It's tied into my engine's media layer for now, but the general shape of it (little native FFmpeg lib, frames to RGBA textures, a media ref in front) would drop into any Godot project fine. Its also likely quite buggy, but works from all the test i have thrown at it so far which granted, isn't that many.
Posted Jun 2, 2026, 1:14 PM UTC