What's a modern way to design storage of BI/report data?
I have a bit experience of older .NET projects and how they store data for reports and BI. But how is this done in modern application efficiently? Also any tips on how to best handling GDPR requirements when dealing with BI/report data?
Thanks!
Posted Feb 2, 2020, 7:21 PM UTC
245
Large datalakes, Kimball model.
And with gdpr we associate a user entry (sensitive or otherwise) with a guid from a different source so all entries are just keys and not on the same db. So the data in itself doesn't contain sensitive data, just links to it through "obscure" keys.
I googled Kimball model. Daaang, that's some enterprise-level kit! I'm going to have to read through it tomorrow. Looks like mandatory knowledge for anyone responsible for business data.
In my experience, if you go into any moderately sized enterprise, this is how nearly all standard BI/reporting is done (disregarding any ML algorithms) from the database side. There is a bunch of denormalized data in fact and dimension tables that is used to generate the reports.
If you're interested in some "light" reading, here's the main book on it: https://www.amazon.com/gp/product/1118530802
Thanks! This is super cool, but sounds at the same time a bit daunting for smaller projects. Any light weight alternatives?
What's your actual need here? Are you building reporting capabilities inside of your application? If so, what tool are you planning on using?
I do plenty of in app reporting directly on top of the transactional tables, but I wouldn't do that in all solutions I build. In others I've opted for a process where I refresh aggregated tables and then sit my reports on top of those.