Recap Preview
Why Would We Need Multiple AWS Accounts?
In AWS, it's common for organizations to use multiple AWS accounts for various reasons, from Dev, Stage, Prod accounts to accounts being dedicated to LOBs. What do you do when your Data Pipeline needs to span AWS accounts? This webinar shows how you can run a single DAG across multiple AWS accounts in a secure manner.
DAG Overview
- Astronomer Airflow Running on EKS Cluster in AWS Account for shared services (“Referred to as AWS Account 3”)
- EMR Job running in AWS dedicated to raw data processing (“AWS Account 1”)
- Athena Query run in AWS account for data query (“AWS Account 2”)
- AWS Permissions granted to Airflow using IAM Cross Account Role, no Access Keys/Secret Access Keys needed! (Although the same setup can be completed using IAM User Access Key/Secret Access Key if preferred)
