Astronomer Certified Image Architecture
No versions of Astronomer Certified (AC) are currently supported by Astronomer. Astronomer stopped releasing new versions of AC with the release of Apache Airflow 2.4. Astronomer recommends creating all new Deployments with Astro Runtime, as well as migrating existing Deployments from AC to Astro Runtime as soon as your organization is ready. See Migrate to Runtime and Runtime image architecture.
Overview
The Astronomer Certified Docker image for Apache Airflow extends the community-developed Airflow image in a way that makes running Airflow more secure, reliable, and extensible. It is the default image for Airflow Deployments on Astronomer.
This guide provides reference information for the building blocks of Astronomer Certified, as well as information on its release and distribution.
Distribution
Astronomer Certified is distributed both as a Python wheel and a Debian-based Docker image. These distributions vary slightly in scope and dependencies.
The Python wheel and Docker image are functionally identical to open source Apache Airflow. Additionally, they both include performance and stability improvements, including critical bug fixes and security patches.
The Astronomer Certified Docker image is built from the Python wheel and incorporates additional logic that makes it easier for users to both get started and run Airflow at scale. This includes:
- A robust testing suite that covers performance benchmarking and end-to-end functionality and upgrade testing for certified environments.
- A built-in directory of example DAGs that leverage Airflow's most powerful features.
- A collection of pre-installed Airflow provider packages.
- Full compatibility with the Astronomer Platform.
Every supported version of the Astronomer Certified Python wheel is available at pip.astronomer.io. The Dockerfiles for all supported Astronomer Certified images can be found in Astronomer's ap-airflow GitHub repository.
Image Requirements
Running Airflow with the Astronomer Certified Docker image requires specific versions for key system components.
- Python: 3.6, 3.7, 3.8, 3.9
- Database: PostgreSQL (11, 12), MySQL (5.7, 8.0+)
- System Distribution: Debian 10 (Buster)
Note: While the Astronomer Certified Python Wheel supports Python versions 3.6, 3.7, and 3.8, Astronomer Certified Docker images have been tested and built only with Python 3.7. These requirements are slightly different for running only the Python wheel. For the Python wheel, you can use:
- Python: 3.6, 3.7, 3.8, 3.9
- Database: PostgreSQL (9.6, 10, 11, 12, 13), MySQL (5.7, 8+), SQLite (3.15.0+)
- System Distribution: Debian 10 (Buster)
For more information on running a Python wheel installation of Astronomer Certified, read Install on a Virtual Machine.
Environment Variables
When an Airflow service is started, it checks a file for runtime environment variables. These are equivalent to values defined in Airflow's airflow.cfg file.
If you run the Astronomer Certified Docker image without the Astronomer platform, environment variables are defined in your Dockerfile. They can be overwritten with a runtime command, such as docker run.
If you're running the Astronomer Certified Docker image with the Astronomer platform, there are a few ways you can configure environment variables. For more information, read Environment Variables.
Astronomer Certified supports the same environment variables as Apache Airflow. For a list of all configurable environment variables, read the Apache Airflow documentation.
The following table lists the essential environment variables used when running Airflow with Astronomer Certified. Only environment variables explicitly defined in the Dockerfile are listed here; all other environment variables have the same default value as they have in OSS Airflow.
| Variable | Description | Default Value |
|---|---|---|
AIRFLOW_PIP_VERSION | The version of pip that is used to install Python packages and Airflow itself. | 19.3.1 |
AIRFLOW_HOME | Filepath for your Airflow project directory. | usr/local/airflow |
AIRFLOW__WEBSERVER__BASE_URL | The URL used to access the Airflow UI. | http://localhost:8080 |
ASTRONOMER_USER | The username for your Airflow user. | astro |
ASTRONOMER_UID | The ID for your Airflow user. | 5000 |
PIP_NO_CACHE_DIR | Specifies whether to maintain copies of source files when installing via pip. | True |
PYTHON_MAJOR_MINOR_VERSION | The version of Python to use for Airflow. | 3.9 |
Provider Packages
Starting in version 2.0.0, the Astronomer Certified image includes provider packages that are utilized in some background processes, as well as packages which are commonly used by the Airflow community. The following table contains version information for each provider package installed as part of Astronomer Certified:
| Astronomer Certified | amazon | azure | celery | cncf.kubernetes | elasticsearch | ftp | http | imap | mysql | postgres | redis | slack | sqlite | ssh | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 2.0.0 | 1.0.0 | 1.1.0 | 1.0.0 | 1.0.1 | 1.0.4 | 1.0.0 | 1.0.0 | 1.0.0 | 1.0.0 | 1.0.0 | 1.0.0 | 1.0.0 | 1.0.0 | 1.0.0 | 1.0.0 |
| 2.0.2 | 1.3.0 | 1.3.0 | 1.0.1 | 1.2.0 | 1.0.4 | 1.0.1 | 2.2.0 | 1.1.1 | 1.0.1 | 1.1.0 | 1.0.1 | 1.0.1 | 3.0.0 | 1.0.2 | 1.3.0 |
| 2.1.0 | 1.4.0 | 2.0.0 | 1.0.1 | 1!1.2.1 | 1.0.4 | 1.1.0 | 3.0.0 | 2.0.0 | 1.0.1 | 1.1.0 | 1.0.2 | 1.0.1 | 3.0.0 | 1.0.2 | 1.3.0 |
| 2.1.1 | 1!2.0.0 | 1!3.0.0 | 1!2.0.0 | 1!2.0.0 | 1!2.0.1 | 1!2.0.0 | 1!4.0.0 | 1!2.0.0 | 1!2.0.0 | 1!2.0.0 | 1!2.0.0 | 1!2.0.0 | 1!4.0.0 | 1!2.0.0 | 1!2.0.0 |
| 2.1.3 | 1!2.1.0 | 1!3.1.0 | 1!2.0.0 | 1!2.0.2 | 1!2.0.2 | 1!2.0.0 | 1!5.0.0 | 1!2.0.0 | 1!2.0.0 | 1!2.1.0 | 1!2.0.0 | 1!2.0.0 | 1!4.0.0 | 1!2.0.0 | 1!2.1.0 |
| 2.1.4 | 1!2.2.0 | 1!3.1.1 | 1!2.1.0 | 1!2.0.2 | 1!2.0.3 | 1!2.0.1 | 1!5.1.0 | 1!2.0.1 | 1!2.0.1 | 1!2.1.1 | 1!2.2.0 | 1!2.0.1 | 1!4.0.1 | 1!2.0.1 | 1!2.1.1 |
| 2.2.0 | 1!2.2.0 | 1!3.2.0 | 1!2.1.0 | 1!2.0.3 | 1!2.0.3 | 1!2.0.1 | 1!6.0.0 | 1!2.0.1 | 1!2.0.1 | 1!2.1.1 | 1!2.3.0 | 1!2.0.1 | 1!4.1.0 | 1!2.0.1 | 1!2.2.0 |
| 2.2.1 | 1!2.2.0 | 1!3.2.0 | 1!2.1.0 | 1!2.0.3 | 1!2.0.3 | 1!2.0.1 | 1!6.0.0 | 1!2.0.1 | 1!2.0.1 | 1!2.1.1 | 1!2.3.0 | 1!2.0.1 | 1!4.1.0 | 1!2.0.1 | 1!2.2.0 |
| 2.2.2 | 1!2.3.0 | 1!3.3.0 | 1!2.1.0 | 1!2.1.0 | 1!2.1.0 | 1!2.0.1 | 1!6.1.0 | 1!2.0.1 | 1!2.0.1 | 1!2.1.1 | 1!2.3.0 | 1!2.0.1 | 1!4.1.0 | 1!2.0.1 | 1!2.3.0 |
| 2.2.3 | 1!2.3.0 | 1!3.3.0 | 1!2.1.0 | 1!2.1.0 | 1!2.1.0 | 1!2.0.1 | 1!6.1.0 | 1!2.0.1 | 1!2.0.1 | 1!2.1.1 | 1!2.3.0 | 1!2.0.1 | 1!4.1.0 | 1!2.0.1 | 1!2.3.0 |
| 2.2.4 | 1!3.0.0 | 1!3.6.0 | 1!2.1.0 | 1!3.0.2 | 1!2.2.0 | 1!2.0.1 | 1!6.4.0 | 1!2.0.3 | 1!2.2.0 | 1!2.2.0 | 1!3.0.0 | 1!2.0.1 | 1!4.2.0 | 1!2.1.0 | 1!2.4.0 |
System Dependencies
The Astronomer Certified Docker image includes a number of OS-level dependencies for running basic system processes. These dependencies can be installed in the Python Wheel as described in Install Packages.
- apt-utils
- build-essential
- curl
- build-essential
- default-libmysqlclient-dev
- libmariadb3
- freetds-bin
- gosu
- libffi6
- libffi-dev
- libkrb5-3
- libkrb5-dev
- libpq5
- libpq-dev
- libsasl2-2
- libsasl2-dev
- libsasl2-modules
- libssl1.1
- libssl-dev
- locales
- netcat
- rsync
- sasl2-bin
- sudo
- tini
Extras
Astronomer Certified includes a few packages that don't have a corresponding provider. These packages are used for basic system functions or optional Airflow functionality. The following list contains all extra packages built into Astronomer Certified by default:
async: Provides asynchronous workers for Gunicornpassword: Adds support for user password hashingstatsd: Adds support for sending metrics to StatsDvirtualenv: Adds support for running Python tasks in local virtual environments