Can someone give feedback on my data pipeline approach?

Help

I’m using the ELT pattern

-The csv data files are stored locally

-Stored them in the AWS S3 data lake (using python script)

-Loaded them to Redshift data warehouse (using python script)

-Cleaned them (removed duplicates, inconsistent data) using SQL (should i do this with sqlalchemy?)

-Did data transformation and analysis using python/sql alchemy

-Need to think on the visualization part

-Planning to use apache workflow for orchestrating all the steps and build the pipeline DAG

Is this a good/decent approach or did i butcher it? What do u suggest i should change?

Posted May 1, 2022, 9:44 PM UTC

813
Loading comments…