Yet another terminal assistant - this time with a local, offline, small language model

I've seen quite a few posts here from people who built terminal assistants to convert natural English to command line commands using LLMs. While this is cool, I felt that it could be improved. I didn't like the idea that it relies on third part LLMs, with API calls, and lacking security.

I built my own tool called Zest. It is a small model/app that translates natural language directly into command line commands and runs fully locally with no API calls, no cloud dependency, no need for a GPU. There is a confirmation step before running commands, and guardrails against running destructive commands.

This is not to replace your workflow entirely. It's for when you forgot a command, need help with difficult or long commands, need some help when you're offline, or are not a frequent command line such as myself or my peers (data analyst/scientists/engineers).

What I did

• Fine tuned a different small Qwen models (Unsloth) using QLoRA.
• Around 100k high quality Instruction-Command line pairs
• Data was rated, augmented, and synthesised using LLMs and manual review
• Trained on Google Colab using an A100 GPU.
• Applied DPO data for aligning the model outputs.
• Model was tested on internal and external benchmarks
• The model was packaged up (Github Link below) into a .dmg

Preparing the data was the hardest and longest part of the development and took about 6 weeks to generated roughly ~100k high quality Instruction - Command Line pairs, which are kept in a private repo.

This software's code is partially AI-generated. The infra repo was partially Claude generated, with the dmg packaging logic and some of the back end logic done by AI. I'm an ML Engineer so backend is not my thing.

While it fulfilling my needs, I'm looking for some people to help me test it so please DM me if this is interesting for you.

Link:

Github: https://github.com/spicy-lemonade/zest-cli-infra