With the Astro CLI, you can run Airflow on your local machine. Follow this quickstart to build an Airflow project from the Learning Airflow template and run it in a local Airflow environment with just a few commands. At the end of the tutorial, you’ll have all of the files and components you need to develop and test Airflow dags locally.
An Astro project contains the set of files necessary to run Airflow, including dedicated folders for your dag files, plugins, and dependencies. This set of files builds an image that you can run both on your local machine with Airflow and deploy to Astro.
Use astro dev init with the --from-template flag to create the project based off of the Learning Airflow template.
This command generates all of the project files you need to run Airflow locally, including an example dag that you can run out of the box. Different templates generate different example dags. See Create an Astro project for more information about the default project structure.
Running your project locally allows you to test your dags before you deploy them to a production environment. While this step is not required for deploying and running your code on Astro, Astronomer recommends always using the Astro CLI to test locally before deploying.
learning-airflow project directory:This command builds your project and spins up 4 containers on your machine, each for a different Airflow component:
astro dev start --standalone, or set it as the default for your project with astro config set dev.mode standalone. See astro dev start for all available options.After your project builds successfully, open the Airflow UI in your web browser at https://localhost:8080/.
Find your dags in thedags directory in the Airflow UI.
In this directory, you can find an example dag, example-astronauts, which was generated with your Astro project. To provide a basic demonstration of an ETL pipeline, this dag shows a simple ETL pipeline example that queries the list of astronauts currently in space from the Open Notify API and prints a statement for each astronaut. The dag uses the TaskFlow API to define tasks in Python, and dynamic task mapping to dynamically print a statement for each astronaut.
8080 for the Airflow webserver and port 5432 for the Airflow metadata database by default. If these ports are already in use on your local computer, an error message might appear. To resolve this error message, see Run Airflow locally.Now that you have a locally running project, you can start to develop your Astro project by adding dags, dependencies, environment variables, and more. See Develop your project for more details on how to modify all aspects of your Astro project.
Most changes you make, including updates to your dag code, are applied automatically to your running environment and don’t require rebuilding your project. However, you must rebuild your project and restart your environment to apply changes from any of the following files in your Astro project:
packages.txtDockerfilerequirements.txtairflow_settings.yamlTo restart your local Airflow environment, run:
This command rebuilds your project and restarts your local Airflow environment. Alternatively, you can run astro dev stop to stop your environment without restarting, then run astro dev start when you want to restart.
After you have finished Getting Started with the CLI, you can configure your CLI to locally debug your Airflow environment, authenticate to cloud services to test your dags with data stored on the cloud, or you can learn more about developing dags with Astro.