How would you share this dataset?

Data Question

I want to share some free data (maybe regularly) to attract future users and get more insights into their needs. The first dataset I am considering sharing is Amazon sales rank data from multiple countries. Each country has roughly 1.2 to 2m products ranked. My problem is the usual Excel limit of 1m rows and bias towards CSV. I expect at least part of the users to be more familiar with Excel than Python, R or other tools.

I considered exporting the data as GZipped CSVs, sampling or limiting the dataset to 1m rows per country or split it into multiple files but it seems inelegant. If I was not worried about Excel I could use compressed JSON, which I presume would be most portable for non-Excel users. I believe Parquet and Avro are still not used by many?

What are your thoughts? Should I:

  • Slice or sample it and dump it as compressed CSV?
  • Ignore Excel users and do JSON?
  • Do both? (Still leaves me with slice vs sample on CSV for Excel. Also, I would prefer one format to keep things simple.)
  • Something else I am missing?

Thank you

---

For context, Bold Data is a new data mining and analytics company aiming to make public data accessible to improve decision-making.

Posted May 5, 2022, 8:18 AM UTC

13
Pflastersteinmetz•

If you double-click a CSV file Excel will fuck it up and your data is pure shit.

Ignore Excel

Yes.

2
bolddataOP•

Funny you write that. I did try to load the exported CSV and it did break thanks to UTF issues. You can get it imported properly but you have to go through Excel's data import steps.

How do Excel users deal with all these limitations? Are they just limited or do they have a some tricks or tools up their sleeves?

2
Pflastersteinmetz•

You can get it imported properly but you have to go through Excel's data import steps.

Excel sets itself as the default software for CSV afaik. So people double-click everything that is a CSV --> it breaks and can even corrupt the underlying CSV --> the data is gone for good.

How do Excel users deal with all these limitations?

They don't. https://www.theverge.com/2020/8/6/21355674/human-genes-rename-microsoft-excel-misreading-dates

Are they just limited or do they have a some tricks or tools up their sleeves?

No, they are just cluesless and produce shit.

1