Each user on my website will generate a lot of activity data, which needs to be tagged with its datetime and stored in a list associated with that user.
I'm not sure how to do this:
- The standard relational approach of putting every activity item in a single giant table will probably be too slow in the long-run. (Please comment if you think I'm wrong about this. I value your input.)
- Nonrelational solutions like MongoDB look OK, but if we're talking about Mongo specifically I don't think it will really cope. The documents are limited to 20mb sizes, and the storage strategies it uses aren't really optimized for storing giant lists
- I looked at HDF5 which is meant to be good for giant