Get Started with Apache Airflow 2.0
Apache Airflow 2.0 is a momentous open-source release that many are eager to test and adopt.
If you'd like to test Airflow 2.0 on your local machine, Astronomer's open-source CLI is the fastest and easiest way to do so.
Read below for guidelines.
Note: If you're an Astronomer user looking to upgrade an Airflow Deployment on Astronomer to 2.0, refer to Upgrade to Apache Airflow 2.0 on Astronomer.
The Astronomer CLI is the easiest way to run Apache Airflow on your machine. From the CLI, you can establish a local testing environment regardless of where you'll be deploying to from there.
There are two ways to install any version of the Astronomer CLI:
Note: If you already have the Astronomer CLI installed, make sure you're running
v0.23.2or above. To check, run
$ astro version.
If you're running an earlier version, follow the install steps below to upgrade.
To use the Astronomer CLI, make sure you have Docker (v18.09 or higher) installed and running on your machine.
To install or upgrade to the latest version of the Astronomer CLI via cURL, run:
$ curl -ssl https://install.astronomer.io | sudo bash
Note: If you're a macOS user using ZSH as your shell, you may encounter an error. Learn more.
To install or upgrade to the latest version of the Astronomer CLI via Homebrew, run:
$ brew install astronomer/tap/astro
For more information, refer to the Astronomer CLI README.
First, create a new directory for your Airflow project and
cd to it:
$ mkdir <directory-name> && cd <directory-name>
In the project directory, run:
$ astro dev init
This project directory is where you'll store all files necessary to build your Airflow 2.0.0 image. It will include a pre-populated Example DAG.
Dockerfile will include reference to a Debian-based, Astronomer Certified Docker Image.
In your Dockerfile, replace the existing FROM statement with:
Feel free to refer to the Astronomer Certified 2.0.0 image source.
Now, run the following command:
$ astro dev start
This command spins up 3 Docker containers on your machine, 1 each for the Airflow Webserver, Scheduler, and Postgres components.
Note: If you’re running the Astronomer CLI with the buildkit feature enabled in Docker, you may see an error (
buildkit not supported by daemon). Check out this forum post for the suggested resolution.
To check out the Airflow 2.0 UI:
adminas both your username and password
The example DAG in your directory should be populated in the Airflow UI on your local machine.
With that, you're all set!
Note: You will NOT be able to run multiple Airflow Scheduler replicas locally. If you’re interested in testing that feature, reach out to us and we’ll help you get set up with a docker-compose override file that you can test both locally and on Astronomer.
Once you've tested Airflow 2.0 locally, refer to:
The Apache Airflow Project strongly recommends that all users interested in running Airflow 2.0 first upgrade to Airflow 1.10.14, which was built to make the migration process as easy as possible.
If you find a bug or problem in Airflow, file a GitHub issue in the Apache Airflow GitHub repo. We'll be working with open source contributors towards subsequent 2.0 releases and are committed to regularly triaging community-reported issues.
Do Airflow the easy way.