Benchmarking Local Models for Physical Edge Actions. How do you test tool calling when the API is flaky hardware?

Most discussions here about local agents focus purely on software sandboxes, like web scraping, code execution, or standard RAG workflows. If a small model messes up a tool call, you simply catch the exception and retry. There is minimal risk, and you get instant rollbacks.

We have been trying to optimize small open source models and quantized vision LLMs to interact directly with the physical world. To test this without building a massive industrial robotics stack, we put together a lightweight desktop testing rig combining an NVIDIA Jetson for local vision inference, a Raspberry Pi 5 as the edge gateway, and standard Espressif ESP32 boards alongside HooRii Technology modules to bridge our local thread and matter sensor clusters.

This is where local inference gets incredibly messy, and software centric benchmarks start to fail. The inputs themselves are deeply flawed: local speech to text engines often output incomplete text, and raw, low resolution camera frames easily confuse quantized multimodal models.

Furthermore, we face massive context and latency constraints;. When we tested standard Wi-Fi via ESP32, connection jitter frequently broke the LLM's exception handling loop. Moving to a dedicated local Thread mesh via Hoori stabilized the local statesync, but running real time control loops is still an uphill battle when local token throughput directly dictates the system's reaction time.

It feels like we are missing aa standardization for testing local models against unpredictable physical environments. Testing strictly on simulated environments or clean synthetic data does not show how a typical 8B model actually behaves when its physical action space breaks down.

For those running open source models for embodied AI or Iot control, how are you benchmarking your model's function calling robustness against noisy, real world sensor data? Are you relying enrichment on software mocks, or do you have a similar physical desktop setup to see exactly where the model's logic breaks down?