Using Airflow with Multiple AWS Accounts

Watch On Demand

Summary:

In AWS, it's common for organizations to use multiple AWS accounts for various reasons, from Dev, Stage, Prod accounts to accounts being dedicated to LOBs. What do you do when your Data Pipeline needs to span AWS accounts? This webinar will show how you can run a single DAG across multiple AWS accounts in a secure manner.

In this webinar we covered:

  • Authoring DAGs that leverage AWS services
  • Airflow Connections
  • IAM roles vs Access Keys
  • Connecting to multiple AWS accounts in a single DAG

Missed the Webinar? Sign up for the Recap

Recap Preview

Why Would We Need Multiple AWS Accounts?

In AWS, it's common for organizations to use multiple AWS accounts for various reasons, from Dev, Stage, Prod accounts to accounts being dedicated to LOBs. What do you do when your Data Pipeline needs to span AWS accounts? This webinar shows how you can run a single DAG across multiple AWS accounts in a secure manner.

DAG Overview

  1. Astronomer Airflow Running on EKS Cluster in AWS Account for shared services (“Referred to as AWS Account 3”)
  2. EMR Job running in AWS dedicated to raw data processing (“AWS Account 1”)
  3. Athena Query run in AWS account for data query (“AWS Account 2”)
  4. AWS Permissions granted to Airflow using IAM Cross Account Role, no Access Keys/Secret Access Keys needed! (Although the same setup can be completed using IAM User Access Key/Secret Access Key if preferred)

Hosted By

Tony

Tony Huinker

Director of Field Engineering

Astronomer

Viraj

Viraj Parekh

Field CTO

Astronomer