file-observer: deterministic, mostly-stdlib file observation with a reproducible JSON manifest

I build data-adjacent tooling as a hobby, and I kept needing the same boring thing: point something at a directory and get back a precise, reproducible description of every file — type, size, structure, embedded metadata — as JSON I could diff and check into a pipeline. So I wrote it, and it's grown into something I use enough that it's worth sharing.

What My Project Does

file-observer walks a directory and emits a deterministic JSON manifest: identity, filesystem metadata, checksum, MIME analysis, structural signatures, and format-specific fields. The load-bearing property is the boring one — same bytes in, byte-identical manifest out, every run. There's a test that scans with 1 worker and with N workers and fails CI if the two manifests differ, so parallelism can never introduce drift.

A few things that keep it lean and honest:

- Mostly stdlib. The format specialists — PDF, OOXML (docx/xlsx/pptx), images with EXIF, MP4/MOV, MP3, email, and more — are written on `struct` / `zipfile` / `olefile` rather than heavy dependencies. Core install is small.
- It observes; it doesn't interpret. No OCR, no classification, no model. It reports what's measurably in a file and stops there — and it never executes file content, so it stays a safe thing to point at untrusted input.
- Several front doors. A CLI (`fo`), a one-call Python API (`from file_observer import scan`), `--stdout` for pipelines, a committed JSON Schema for any-language consumers, and an MCP server for agent tooling.
- Bring-your-own word lists (one newer use case). Hand it a category-tagged list and it counts matches per file — the list is supplied at runtime, never shipped, never echoed back. Handy as a deterministic pre-screen in front of anything that reads untrusted files, an AI agent included.

Target Audience

Data-pipeline and cataloguing work, CI checks, provenance/inventory over a corpus — anywhere you want a stable, diffable description of a pile of files. It's on PyPI, has ~1,400 tests, and runs on Linux/macOS/Windows. Honest framing: I'm an enthusiast, not a professional, and fo is deliberately just an observer — it hands you signal and you own what to do with it. It's production-usable for what it claims (deterministic observation), not a magic file-understanding engine.

Comparison

`python-magic` / `file` tell you a file's type and stop. Apache Tika extracts rich content across formats — powerful, but it exists to pull content out, pulls in a large parser surface, and needs a JVM. file-observer sits in a different spot: dependency-light and mostly-stdlib, focused on describing a file rather than extracting its content, with output that's byte-for-byte reproducible so it fits in a pipeline or a CI gate. Where Tika answers "what's in this file," fo answers "what is this file, exactly and repeatably."

---

pip install file-observer

Repo + design write-ups: https://github.com/russalo/file-observer

It's an enthusiast project and I'm fairly isolated from people who do parsing or security for a living, so critical eyes on it — including the unflattering kind — are exactly why it's here.