In this video, we will develop AWS Glue ETL script using Python shell. We can now use Python scripts in AWS Glue to run small to medium-sized ETL (extract, transform, and load) workflow. Previously, AWS Glue jobs were limited to Apache Spark environment.
Python shell jobs in AWS Glue support scripts that are compatible with Python 2 and 3 and come pre-loaded with libraries such as the Boto3, Numpy, SciPy, pandas, and others. We can also, install other libraries via .whl file.
Link to medium article on the topic: blog.devgenius.io/develop-aws-glue-etl-pipeline-wi…
Link to code, GitHub: github.com/hnawaz007/pythondataanalysis/tree/main/…
Subscribe to our channel:
youtube.com/c/HaqNawaz
---------------------------------------------
Follow me on social media!
GitHub: github.com/hnawaz007
Instagram: www.instagram.com/bi_insights_inc
LinkedIn: www.linkedin.com/in/haq-nawaz/
---------------------------------------------
#Python #ETL #AWS
Topics covered in this video:
0:00 - Introduction ETL with Python shell
0:53 - Pre-Requisites
1:30 - Create Python .whl file
2:35 - Python ETL script
4:15 - Upload scripts to AWS
5:11 - AWS Glue ETL Job
6:33 - AWS Redshift table
6:49 - Execute Glue ETL Job
7:17 - Review Data & logs
コメント