I built an open-source agent that predicts OOMs before they crash your training

Got tired of coming back to dead Colab sessions and cryptic OOM errors 40 epochs in. So I built something to fix it.

Bannin is a lightweight Python agent that monitors your training environment and streams everything to a web dashboard. You set it up from your browser, install the agent on whatever machine you're training on, and then check in from anywhere.

The OOM prediction:

It tracks memory growth rate using linear regression on a rolling window. If RAM or GPU memory is trending toward full, it tells you the estimated time until crash with a confidence score — while there's still time to checkpoint or kill the run. Not after the fact.

Remote stop/kill:

You can stop a training run directly from the dashboard. It sends SIGINT first so your checkpoint callbacks fire, waits 30 seconds, then SIGKILL if the process hangs. No need to SSH in or dig up your terminal.

What else it tracks:

- Live CPU/RAM/GPU/disk with history

- Training progress bars (hooks into tqdm, detects epoch/step patterns from stdout)

- Detected training scripts (auto-finds python processes running common training patterns)

- Active alerts with configurable thresholds

Works on Colab, Kaggle, local machines, and cloud VMs. Just pip install and point it at the relay.

Try it: https://bannin.vercel.app

Open source: github.com/Lakshman-P4/Bannin.dev

Curious what metrics would actually be useful to people during long training runs. What do you find yourself checking manually that should be automated? Feedback form on github repo:)