Core idea

Split the system into 5 planes that can scale independently:

1. Ingestion plane (crawlers / connectors / streams)
2. Coordination plane (frontier + scheduling + dedupe + policies)
3. Memory plane (content store + vector store + graph store)
4. Computation plane (extract/normalize/link/learn)
5. Access plane (query, agents, MCP, authz/audit)

1) Ingestion plane

Workers do “fetch/render/extract” (Playwright-style), plus connectors (docs, git, email, etc.).

Key requirements:

• Identity: every ingest event has who/what/where provenance.
• Content-addressing: store blobs by sha256(content) so dedupe is natural.
• Politeness + legality: robots, rate limits, allow/deny lists, consent scopes.

Output is an append-only event:

IngestEvent { source, url, fetched_at, content_hash, raw_blob_ref, metadata... }

2) Coordination plane (the “planetary” part)

This is where most “crawlers connected together” efforts fail.

Frontier as a CRDT

Use a distributed frontier represented as CRDT sets/maps:

• Seen(url_hash) (observed)
• Fetched(url_hash, version) (completed)
• Pending(url_hash, priority, constraints) (queue-ish)
• Denied(url_hash, reason) (policy)

Nodes can gossip these states without a single leader. You still use regional schedulers for efficiency, but state converges.

Partitioning

Partition by something stable:

• (host, path_prefix) or (url_hash mod N) Use rendezvous hashing so partitions move smoothly as nodes join/leave.

Dedupe

Three layers:

• URL canonicalization + hashing
• Content hash (exact)
• Similarity hash (near-dup) for templated pages

Policies

Every fetch must pass:

• scope (who is allowed to ingest)
• robots/terms constraints
• per-domain budgets
• “sensitive zones” (PII, paywalls, medical, etc.)

Policies are data, versioned, and enforced at the edge.

3) Memory plane: blobs + vectors + graph

You want all three because each answers different questions.

A) Blob store (truth)

• Object store (S3/GCS) for raw HTML, screenshots, PDFs, etc.
• Immutable; references only.

B) Vector memory (semantic)

Sharded ANN index (HNSW / IVF / ScaNN-like) per tenant/scope.

• Supports dual embeddings (organize vs retrieve) like your Chonk description.
• Supports time decay and context windows.

C) Graph memory (meaning + provenance)

A graph DB (or graph-on-KV) storing:

• Nodes: URL, Document, Entity, Claim, Concept, Agent, User, Task
• Edges: LINKS_TO, MENTIONS, DERIVES_FROM, CONTRADICTS, SUPPORTS, SAME_AS, AUTHORED_BY, OBSERVED_AT

Graph is where “living” happens:

• Every ingest creates/updates edges
• Every extraction adds typed entities/claims
• Every query leaves a trace and can create new “task edges” (what was needed, what was missing)

Provenance is mandatory: every edge points to evidence (blob refs + offsets).

4) Computation plane (continuous refinement)

This is the “fabric” feeling: the system keeps improving itself.

Pipelines (streaming):

1. Normalize (canonical URL, metadata, language)
2. Extract (markdown/text, entities, tables, structured fields)
3. Link (entity resolution + doc similarity + citations)
4. Index (vectors + graph updates)
5. Learn (priority models, domain models, quality scoring)

Crucial: recompute is cheap because blobs are immutable and everything is evented.

5) Access plane (query + agents)

Queries you must support

• “Find docs semantically” (vector)
• “Explain relationship / lineage” (graph)
• “What changed since X?” (event log + graph diffs)
• “Show evidence” (blob refs)

Agent interface

Expose tools via MCP:

• search_vector(scope, q)
• graph_traverse(scope, pattern)
• fetch_evidence(blob_ref, range)
• propose_frontier_tasks(missing_info)
• schedule_ingest(tasks)

Agents should never directly fetch the open web; they request tasks through policy gates.

The hard parts (aka what makes it real)

1) Trust + consent

This fabric can’t be “planetary” unless it’s multi-tenant + permissioned.

• Per-tenant partitions and keys
• Audit logs for every access
• Signed provenance for ingests
• Explicit consent scopes for private connectors

2) Quality control

Add a quality score on every doc and edge:

• source reputation
• extraction confidence
• contradiction density
• staleness
• evidence completeness

3) Safety / abuse resistance

You need guardrails so it can’t become a harassment/doxxing machine:

• PII detection + redaction pipelines
• denylist categories
• rate-limited query patterns
• “evidence-required” for claims surfaced

A build plan that doesn’t melt you

Phase 1 (weeks): “Regional fabric”

• Single region scheduler + queue
• Blob store + vector store + minimal graph
• Provenance everywhere
• Policies enforced at ingest

Phase 2: “Multi-region”

• Partition frontier by host/hash
• Regional schedulers + shared CRDT state
• Cross-region replication for graph + vectors (event log based)

Phase 3: “Self-healing living memory”

• Continuous entity resolution
• Claim graph + contradiction edges
• Re-crawl triggers based on freshness + uncertainty

If you want to make this yours (not a generic architecture), tell me one constraint and I’ll pin the design to it without questions:

• Open web vs private knowledge focus?
• Single user vs org multi-tenant?
• Edge devices participating or only datacenters?

Or I can just pick the most practical: multi-tenant private-first (connectors) + opt-in web crawl, because that’s how you get “planetary” without building a legal nightmare.