Scaling a real-time market data engine with Go 1.24 and Redis

Good Morning everyone, spent the last few months building a Market Intelligence Engine (MIE) called Limpio Terminal. We recently moved the core pipeline to Go 1.24 , and I wanted to share some notes on how we structured the services to handle concurrent streams from 7 different exchanges without hitting major bottlenecks.

https://imgur.com/a/fNEd7rd - ^(I've attached a screenshot of the architecture for your convenience (please don't delete the post))

To keep the ingestion pipeline stable, we decoupled the system into three specialized services:

1. Collector: Manages WebSocket connections to ^(Binance, OKX, Bybit, Kraken, Gate.i0, Bitget, and KuCoin.) It handles tick normalization and uses a logic Candle Forge to aggregate raw events into 1h bars.
2. Calculator: The heavy lifter. It listens to Redis Pub/Sub and processes indicator calculations in batches of 50-100 pairs using a parallel worker pool (usually 4-8 workers)
3. API: An isolated gateway that only reads from storage. It doesn't touch the exchanges, which ensures the data flow remains unaffected by external requests

Latency was a primary concern. In our current distributed setup, we’re seeing an average latency of ~250 ms over a 1,500km distance. It’s consistent enough for our 1h aggregation logic. We export all internal metrics like `calculator_batch_processing_time_ms` to Prometheus to monitor worker pool performance in real-time.

To keep the footprint small, we went with a hybrid storage strategy:

1. Redis: Handles ticker snapshots and the active candle window for the calculator.

2. TimescaleDB : Used only for 1h candles. By using hypertables and enabling compression for data older than 24h, we fit a full year of history for 1000 trading pairs into just a few gigabytes.

If anyone is interested, I can share the white paper. I'm really looking for different perspectives on this setup, so feel free to ask any questions or share your thoughts below!