I’m building Jarvis: a local voice + gesture copilot for managing coding agents. Does the concept make sense?

I’ve been building a side project called Jarvis. The idea is a local-first desktop copilot that helps me supervise multiple AI coding agents without constantly switching windows or babysitting terminals.

https://reddit.com/link/1uqqvwc/video/eyrpi62ixzbh1/player

The basic concept:

• Talk to Jarvis by voice.
• Jarvis can read agent output, send prompts, approve or interrupt runs, and spawn new agents in tmux.
• It keeps lightweight shared project memory through an MCP server, so one agent can hand off context to another.
• Speech-to-text and text-to-speech run locally.
• The LLM “brain” can be swapped between providers.
• There’s also a HUD: a desktop interface with a central orb, transcript, agent tiles, mic/link state, system audio, and an ASCII webcam mirror for gesture feedback.
• Gesture control can gate the mic or approve/interrupt agents, so it is not always listening.

The design goal is not “AI assistant that does everything.” It is more like an operations layer for local coding agents: voice for intent, gestures for quick approvals, tmux for real process control, and a visual HUD so you can tell what is happening at a glance.

I’m curious what people think about both the concept and the interface direction.

A few questions I’m wrestling with:

• Would you actually want a voice/gesture layer for supervising coding agents, or is that too much ceremony?
• Does a HUD make this feel clearer, or does it risk feeling gimmicky?
• Would you trust a local assistant that can drive terminals if destructive commands require explicit approval?
• Is the “agent orchestration” angle more interesting than the voice assistant angle?
• What would make this useful enough to try for your own workflow?

I’m still treating it as an experiment, but it has reached the point where the shape of the product is becoming clearer. Would appreciate blunt feedback on whether this is compelling, overbuilt, confusing, or something you’d want to use.