This guide explains how to identify and resolve common Airflow DAG issues. It also includes resources to try out if you can’t find a solution to an Airflow issue. While the focus of the troubleshooting steps provided lies on local development, much of the information is also relevant for running Airflow in a production context.
Tip
Consider implementing systematic testing of your DAGs to prevent common issues. See the Test Airflow DAGs guide.
To get the most out of this guide, you should have an understanding of:
To give yourself the best possible chance of fixing a bug in Airflow, contextualize the issue by asking yourself the following questions:
Answering these questions will help you narrow down what kind of issue you’re dealing with and inform your next steps.
The 3 most common ways to run Airflow locally are using the Astro CLI, running a standalone instance, or running Airflow in Docker. This guide focuses on troubleshooting the Astro CLI, which is an open source tool for quickly running Airflow on a local machine.
The most common issues related to the Astro CLI are:
astro version to confirm that you can successfully run Astro CLI commands. If a newer version is available, consider upgrading.packages.txt and requirements.txt.astro dev logs -s to troubleshoot.To troubleshoot infrastructure issues when running Airflow on other platforms, for example in Docker, on Kubernetes using the Helm Chart or on managed services, please refer to the relevant documentation and customer support.
You can learn more about testing and troubleshooting locally with the Astro CLI in the Astro documentation.
This section covers common issues related to DAG code that you might encounter when developing.
If a DAG isn’t appearing in the Airflow UI, it’s typically because Airflow is unable to parse the DAG. If this is the case, you’ll see an Import Error in the Airflow UI.
The message in the import error can help you troubleshoot and resolve the issue.
To view import errors in your terminal, run astro dev run dags list-import-errors with the Astro CLI, or run airflow dags list-import-errors with the Airflow CLI.
If you don’t see an import error message but your DAGs still don’t appear in the UI, try these debugging steps:
Make sure all of your DAG files are located in the dags folder.
Airflow scans the dags folder for new DAGs every dag_dir_list_interval, which defaults to 5 minutes but can be modified. You can force reparsing of all files with astro dev run dags reserialize / airflow dags reserialize.
Ensure that you have permission to see the DAGs, and that the permissions on the DAG file are correct.
Run astro dev run dags list with the Astro CLI or airflow dags list with the Airflow CLI to make sure that Airflow has registered the DAG in the metadata database. If the DAG appears in the list but not in the UI, try restarting Airflow with astro dev restart.
Try restarting the Airflow scheduler with astro dev restart.
If you see an error in the Airflow UI indicating that the scheduler is not running, check the scheduler logs to see if an error in a DAG file is causing the scheduler to crash. If you are using the Astro CLI, run astro dev logs -s and then try restarting.
At the code level, ensure that each DAG:
dag_id.@dag decorator. See also Introduction to Airflow decorators.You can configure an Airflow listener as a plugin to run any Python code, either when a new import error appears (on_new_dag_import_error) or when the dag processor finds a known import error (on_existing_dag_import_error). See Airflow listeners for more information.
A frequent cause of DAG import errors is not having the necessary packages installed in your Airflow environment. You might be missing provider packages that are required for using specific operators or hooks, or you might be missing Python packages used in Airflow tasks.
In an Astro project, you can install OS-level packages by adding them to your packages.txt file. You can install Python-level packages, such as provider packages, by adding them to your requirements.txt file. If you need to install packages using a specific package manager, consider doing so by adding a bash command to your Dockerfile.
To prevent compatibility issues when new packages are released, Astronomer recommends pinning a package version to your project. For example, adding apache-airflow-provders-amazon==9.6.0 to your requirements.txt file ensures that no future releases of apache-airflow-provders-amazon causes compatibility issues. If no version is pinned, Airflow will always use the latest available version.
If you are using the Astro CLI, packages are installed in the scheduler container. You can confirm that a package is installed correctly by running:
If you have conflicting package versions or need to run multiple Python versions, you can run tasks in different environments using a few different operators:
If many Airflow tasks share a set of alternate package and version requirements a common pattern is to run them in two or more separate Airflow deployments.
If your DAGs are either not running or running differently than you intended, consider checking the following common causes:
DAGs need to be unpaused in order to run on their schedule. You can unpause a DAG by clicking the toggle on the left side of the Airflow UI or by using the Airflow CLI.
If you want all DAGs unpaused by default, you can set dags_are_paused_at_creation=False in your Airflow config.
Double check that each DAG has a unique dag_id. If two DAGs with the same id are present in one Airflow instance the scheduler will pick one at random every 30 seconds to display.
Make sure your DAG has a start_date in the past. A DAG with a start_date in the future will result in a successful DAG run with no task runs. Do not use datetime.now() as a start_date.
Test the DAG using astro dev dags test <dag_id>. With the Airflow CLI, run airflow dags test <dag_id>.
If no DAGs are running, check the state of your scheduler
using astro dev logs -s.
If your DAG is running, but not on the schedule you expected, review the DAG Schedule DAGs in Airflow guide. If you are using a custom timetable, ensure that the data interval for your DAG run does not precede the DAG start date.
This section covers common issues related to individual tasks you might encounter. If your entire DAG is not working, see the DAGs are not running correctly section above.
Tip
There were significant changes to the Airflow architecture between Airflow 2 and Airflow 3, greatly improving Airflow’s security posture and enabling new features such as remote execution. The most important impact of those changes for DAG authors is that directly accessing the metadata database from within Airflow tasks is not possible anymore. See the Upgrade from Apache Airflow® 2 to 3 guide and the Airflow release notes for more information.
It is possible for a DAG to start but its tasks to be stuck in various states or to not run in the desired order. If your tasks are not running as intended, try the following debugging methods:
start_date is in the past. A future start_date will result in a successful DAG run even though no tasks ran.scheduled or queued state, ensure your scheduler is running properly. If needed, restart the scheduler or increase scheduler resources in your Airflow infrastructure.depends_on_past parameter set to True, those newly added tasks won’t run until you set the state of prior task runs.task_queued_timeout parameter controls how long tasks can be in queued state before they are either retried or marked as failed. The default is 600 seconds.queued state, consider turning on stalled_task_timeout.Most task failure issues fall into one of 3 categories:
Failed tasks appear as red squares in the Grid view, where you can also directly access task logs.
The task logs provide information about the error that caused the failure.
To help identify and resolve task failures, you can set up error notifications. See Error Notifications in Airflow.
Task failures in newly developed DAGs with error messages such as Task exited with return code Negsignal.SIGKILL or containing a -9 error code are often caused by a lack of memory. Increase the resources for your scheduler, webserver, or pod, depending on whether you’re running the Local, Celery, or Kubernetes executors respectively.
Info
After resolving your issue you may want to rerun your DAGs or tasks, see Rerunning DAGs.
Dynamic task mapping is a powerful feature that allows you to dynamically adjust the number of tasks at runtime based on changing input parameters. It is also possible to dynamically map over task groups.
Possible causes of issues when working with dynamically mapped tasks include:
.expand() function..expand_kwargs(), you did not provide mapped parameters in the form of a List(Dict).max_map_length and is 1024 by default.max_active_tis_per_dag.When creating complex patterns with dynamically mapped tasks, we recommend first creating your DAG structure using EmptyOperators or decorated Python operators. Once the structure works as intended, you can start adding your tasks. Refer to the Create dynamic Airflow tasks guide for code examples.
Tip
It is very common that the output of an upstream operator is in a slightly different format than what you need to map over. Use
.map()to transform elements in a list using a Python function.
When you check your task logs to debug a failure, you may not see any logs. On the log page in the Airflow UI, you may see a spinning wheel, or you may just see a blank file.
Generally, logs fail to appear when a process dies in your scheduler or worker and communication is lost. The following are some debugging steps you can try:
log_fetch_timeout_sec parameter to greater than the 5 second default. This parameter controls how long the webserver waits for the initial handshake when fetching logs from the worker machines, and having extra time here can sometimes resolve issues.Typically, Airflow connections are needed to allow Airflow to communicate with external systems. Most hooks and operators expect a defined connection parameter. Because of this, improperly defined connections are one of the most common issues Airflow users have to debug when first working with their DAGs.
While the specific error associated with a poorly defined connection can vary widely, you will typically see a message with “connection” in your task logs. If you haven’t defined a connection, you’ll see a message such as 'connection_abc' is not defined.
The following are some debugging steps you can try:
<external tool>_default connection to use your connection details or define a new connection with a different name and pass the new name to the hook or operator.To find information about what parameters are required for a specific connection:
The information provided here should help you resolve the most common issues. If your issue was not covered in this guide, try the following resources:
airflow and other relevant tools you are using. Using Stack Overflow is ideal when you are unsure which tool is causing the error, since experts for different tools will be able to see your question.#newbie-questions or #troubleshooting. The Airflow slack is the best place to get answers to more complex Airflow specific questions.To get more specific answers to your question, include the following information in your question or issue: