Design a Log Collection & Analysis System (Interview-Ready Blueprint)

In interviews, you must turn “collect logs” into a scalable pipeline with clear trade-offs.

Requirements: ingest logs from many services, query by service/time/level, full-text search, basic metrics (e.g., error counts), and archival. Non-functional: scale with volume, high availability, near real-time (seconds), configurable retention, eventual consistency.

Sizing drives design. With 100 services × 100 instances × 500 logs/min × 1KB, you’re at ~5M logs/min (~4.8GB/min) ≈ 6.7TB/day; peak can be 2×. That implies hot storage for 7 days (~47TB) and cold storage for 90 days (~603TB).

Architecture: producers → ingestion API/agent → Kafka (buffer + durability) → consumers. Consumers write recent logs to Cassandra (partition by service_id + time bucket), index to Elasticsearch for search, and pre-aggregate metrics (error_count per service per hour). Archive older data to S3 (compressed, partitioned by service/date). API layer routes: Cassandra for recent range scans, Elasticsearch for text search, S3 for restore.

Trade-offs: Kafka adds complexity but prevents spike-induced loss; Cassandra beats RDBMS for write-heavy time-series; Elasticsearch is powerful but expensive—don’t keep everything hot.

#SystemDesign #DataEngineering #InterviewPrep