I built an automated pipeline to filter and summarize daily AI research (because keeping up manually became impossible)

I don’t post on Reddit much, but I’ve been quietly working on a full-stack project for a while and wanted to share it with people who might actually find it useful (or who can tear apart my architecture).

Like a lot of you, I was finding it exhausting to keep track of the daily firehose of AI papers hitting arXiv and HuggingFace. Most of it is noise, and figuring out what is actually relevant to my job takes too much time. So, I built an AI Research Aggregator to automate the entire process.

You can check out the live production version here:https://latest-ai-updates.vercel.app/

What it does

It is a full-stack intelligence platform that automatically ingests, enriches, and serves a persona-aware feed.

  • Zero-Noise Ingestion: A daily automated cron job fetches the latest AI content from arXiv and HuggingFace. It filters off-topic papers at ingestion time, before a row is ever created in the database.
  • AI Enrichment: It enriches each article asynchronously with LLM-generated career insights. This includes actionable data like why it matters, skills to learn, FAANG-style interview questions, and market demand.
  • Persona-Aware Feeds: The platform personalizes the presentation to the user's role: Student, Professional, or Researcher.
  • Automated Learn Cards: It generates original learning cards from openly-licensed curricula, which are then validated by an LLM judge.
  • Live-Search Chatbot: The platform features a grounded chatbot backed by live Tavily web search.

The Tech Stack

If anyone is curious about the plumbing, here is how I put it together:

  • Frontend: Next.js 15 App Router, React 19, TypeScript, and Tailwind CSS.
  • Backend & DB: Vercel Node.js runtime, Neon PostgreSQL (Serverless), and Prisma ORM.
  • AI Integration: OpenRouter managing four model tiers (including Gemini 2.5) and Tavily API for live search.
  • Auth: NextAuth v5.

The Hardest Part

Getting the background pipeline runner to work reliably on Vercel's serverless infrastructure without hitting the maximum duration limits. I had to build a chunked state machine where each chunk does a bounded batch, persists progress, releases its lock, and dispatches the next chunk over HTTP.

I’m still iterating on the topic affinity model, which learns from explicit feedback to re-rank the feed, but it is fully functional.

I’d love to hear your thoughts, especially if you have experience managing async AI enrichment pipelines or handling rate limits gracefully. Let me know what you think of the live version!

Posted Jul 23, 2026, 10:12 AM UTC

10
Loading comments…