Can someone give feedback on my data pipeline approach?
I’m using the ELT pattern
-The csv data files are stored locally
-Stored them in the AWS S3 data lake (using python script)
-Loaded them to Redshift data warehouse (using python script)
-Cleaned them (removed duplicates, inconsistent data) using SQL (should i do this with sqlalchemy?)
-Did data transformation and analysis using python/sql alchemy
-Need to think on the visualization part
-Planning to use apache workflow for orchestrating all the steps and build the pipeline DAG
Is this a good/decent approach or did i butcher it? What do u suggest i should change?
Posted May 1, 2022, 9:44 PM UTC
813