Recently I built a dataset of hundreds of millions of tables, crawled from the Internet and open data providers, to train an AI tabular foundation model. Searching through the datasets is super difficult, b/c off-the-shelf tech just doesn't exist for searching through messy tables at that scale.
So I've been working on this side project, Gini. It has subsets of FRED and data.gov--I'm trying to keep the data manageably small so I can iterate faster, while still being interesting. I picked a random time slice from data.gov so there's some bias towards Pennsylvania and Virginia. But if it looks worthwhile, I can easily backfill a lot more datasets.
Currently it does a tab
51