Built a tiny Go-based news harvester to understand how aggregators actually work

I’ve been curious for a while about how news apps actually pull stories from so many places and keep everything clean. Not the flashy ML stuff, just the basic crawling, enrichment, and pushing events around.

So I tried building that piece myself on weekends. Ended up with a small Go service I’m calling Samvad News Harvester. It reads a list of providers from YAML/JSON, crawls their feeds or sitemaps, extracts whatever metadata it can, skips duplicates, and then publishes everything as JSON to either webhooks or message queues. Nothing fancy, but the parts fit together nicely.

Repo: https://github.com/samvad-hq/samvad-news-harvester

I’m mostly doing this to understand how a full news aggregator might look under the hood. While building the harvester I ended up learning a lot more Go than I expected, especially around keeping things pluggable. Providers, Publishers, Dedupe layers, all slots you can swap without touching the core loop.

Next thing I want to explore is the ranking side: categories, clustering, freshness, maybe some light recommendations. Mostly out of curiosity.

If this sounds interesting, drop a star on Github. And if you’ve worked on similar systems or have opinions about feed/ranking pipelines, I’d love to hear them.