Apache Airflow 2.0 is a momentous open-source release that many are eager to test and adopt.

If you'd like to test Airflow 2.0 on your local machine, Astronomer's open-source CLI is the fastest and easiest way to do so.

Read below for guidelines.

Note: If you're an Astronomer user looking to upgrade an Airflow Deployment on Astronomer to 2.0, refer to Upgrade to Apache Airflow 2.0 on Astronomer.

Step 1: Install or Upgrade the Astronomer CLI

The Astronomer CLI is the easiest way to run Apache Airflow on your machine. From the CLI, you can establish a local testing environment regardless of where you'll be deploying to from there.

There are two ways to install any version of the Astronomer CLI:

Note: If you already have the Astronomer CLI installed, make sure you're running v0.23.2 or above. To check, run $ astro version.

If you're running an earlier version, follow the install steps below to upgrade.

Prerequisites

To use the Astronomer CLI, make sure you have Docker (v18.09 or higher) installed and running on your machine.

Install or Upgrade the CLI via cURL

To install or upgrade to the latest version of the Astronomer CLI via cURL, run:

$ curl -ssl https://install.astronomer.io | sudo bash

Note: If you're a macOS user using ZSH as your shell, you may encounter an error. Learn more.

Install or Upgrade the CLI via Homebrew

To install or upgrade to the latest version of the Astronomer CLI via Homebrew, run:

$ brew install astronomer/tap/astro

For more information, refer to the Astronomer CLI README.

Step 2: Initialize an Airflow Project

First, create a new directory for your Airflow project and cd to it:

$ mkdir <directory-name> && cd <directory-name>

In the project directory, run:

$ astro dev init

This project directory is where you'll store all files necessary to build your Airflow 2.0.0 image. It will include a pre-populated Example DAG.

Step 3: Add Airflow 2.0 to your Dockerfile

Your Dockerfile will include reference to a Debian-based, Astronomer Certified Docker Image.

In your Dockerfile, replace the existing FROM statement with:

FROM quay.io/astronomer/ap-airflow:2.0.0-2-buster-onbuild

Feel free to refer to the Astronomer Certified 2.0.0 image source.

Step 4: Start Airflow

Now, run the following command:

$ astro dev start

This command spins up 3 Docker containers on your machine, 1 each for the Airflow Webserver, Scheduler, and Postgres components.

Note: If you’re running the Astronomer CLI with the buildkit feature enabled in Docker, you may see an error (buildkit not supported by daemon). Check out this forum post for the suggested resolution.

Step 5: Access the Airflow 2.0 UI

To check out the Airflow 2.0 UI:

  1. Go to http://localhost:8080/
  2. Log in with admin as both your username and password

The example DAG in your directory should be populated in the Airflow UI on your local machine.

With that, you're all set!

Note: You will NOT be able to run multiple Airflow Scheduler replicas locally. If you’re interested in testing that feature, reach out to us and we’ll help you get set up with a docker-compose override file that you can test both locally and on Astronomer.

Resources

Once you've tested Airflow 2.0 locally, refer to:

The Apache Airflow Project strongly recommends that all users interested in running Airflow 2.0 first upgrade to Airflow 1.10.14, which was built to make the migration process as easy as possible.

If you find a bug or problem in Airflow, file a GitHub issue in the Apache Airflow GitHub repo. We'll be working with open source contributors towards subsequent 2.0 releases and are committed to regularly triaging community-reported issues.

Never miss an update from us.

Do Airflow the easy way.

Run production-grade Airflow out-of-the-box with Astronomer.