Hey everyone,

I’ve been deep into experimenting with an AI animation tool called Elser AI, and honestly, it’s been a wild ride. I thought it’d be fun to give you a peek into what’s happening behind the scenes and share the process. This isn’t a promotional post, just a dev log for anyone curious about AI video tools, homegrown pipelines, or the chaos that happens when you try to turn a simple idea into an animated story.

At the start, it sounded so easy: type in an idea, get a short animated video in your desired style. But what it turned into was a full-on production pipeline. A single prompt turns into a script, then into a rough storyboard, followed by scenes with character designs, backgrounds, and key moments. From there, the system generates everything in different anime-inspired styles, animates stills using T2V (Text-to-Video) and I2V (Image-to-Video) models, and even adds voice using custom TTS and voice cloning. Then, it all lands on a timeline where you can tweak pacing, reorder shots, and add subtitles.

The “magic” of it all is mostly the unsexy stuff: cleaning up prompts, managing model routes, fixing glitches, smoothing transitions, and trying to make the entire system feel like a seamless tool instead of a patchwork of ten different ones.

The dream of a single model doing everything? Yeah, that didn’t last long. Elser AI routes each task to the most specialized engine. For visuals, I rely on different models like Flux Context Pro for clean lines, Seedream 4.0 for cinematic lighting, and GPT Image One for quick drafts. Animation-wise, I jump between Sora Pro for stable shots, Kling 2.1 for more dynamic movement, and Seedance Lite when I need to quickly test scene flow. Audio is powered by custom TTS and voice cloning, with a layer of lip-sync magic trying to keep everything in sync and believable.

Building something usable for real people? That’s where things got *weird*. Character consistency was a big hurdle even with advanced models, characters kept changing appearance between shots. So, I had to build in a feature that locks traits and ensures they stay consistent across the whole video. Switching styles from anime to cartoony, Pixar-esque to sketchy was another challenge. I ended up creating a style library that automatically adjusts the model’s settings and rewrites prompts to keep things consistent. Then, of course, there’s the classic AI video stuff: jittery movement, random color shifts, lighting that randomly changes. Fixing that required extra checks, guided keyframes, and a bunch of little tweaks to make it all work smoothly.

And voice? TTS just wasn’t cutting it, so I added a step to generate emotional cues before feeding them into the voice models. The result? It actually feels like acting now, instead of sounding like a GPS navigation system.

And let’s not forget compute costs. Video models burn through GPU power if you let them. So, we’ve limited the heavy models to final renders, forcing drafts onto lighter models to keep things manageable. And, of course, not everyone wants to mess with seeds, samplers, or CFGs, so the tool defaults to reasonable settings while still offering an “advanced mode” for those who want to dive deep.

I’ve got a small waitlist open for folks who want to try out the early version and break things (no pressure, though). I’m looking for feedback from creators interested in AI video, anime-style projects, original characters, or unconventional storytelling formats. And if you’re building your own pipeline, I’d love to hear how you’re making it work, what’s working for you, and the weird problems you’ve run into.

Happy to dive deeper into any part of this if anyone’s curious!

Behind the Scenes of Building an AI Animation Tool: The Elser AI Journey