r/aws•
paperknight-404

Idea to efficiently running data analytics on data stored in s3

database

My company is storing additional terabytes of data daily and we have infrastructure to stream the data to s3 in multiple parquet files that arw partitioned based on the timestamp epoch date and hours. We run different analysis on the data using Athena based on the manual findings that we have initially on the data.

I realised that we must have done something wrong, as the data scanned for each query is too big and hence costing us a lot of money. Secondly, the queries are slow (7-15) mins for 1 sort of analysis. Based on my researched on Athena, the problem we are facing is due to poorly partitioned data that causes the engine to scan a lot of data unnecessarily.

I wonder how under companies crunches their data efficiently and fast? Any thoughts?

Thanks in advance for your thoughts on this!

Posted Jan 10, 2024, 6:46 PM UTC

10
Loading comments…