Deploy to Astro with the API
While you can deploy your Apache Airflow code to Astro using the Astro GitHub integration, the Astro CLI, or by configuring a CI/CD pipeline, your organization might prefer to use the Astro API to deploy code. This is because using the Astro API has very few dependencies, making it compatible with almost all CI/CD environments and security requirements.
If the Astro API has access to your Astro project files, you can use the deploy
endpoints in the Astro API to complete either a complete project deploy, image-only deploy or dag-only deploy. You can then implement scripts to automate deploys as an alternative to using the Astro CLI or the Astro GitHub integration.
This best practice guide first walks through the steps that are necessary to deploy code to Astro using the Astro API for three different code deploy methods:
- A complete project deploy
- An image-only deploy
- A dags-only deploy
Then, the guide shows you the recommended way to combine these three automated processes to create a script with conditional logic that can automatically deploy code to Astro, depending on which types of files change in your Astro project. These examples are bash scripts that use Docker to build the image. You can also use a different scripting language, like Python, instead of bash.
Feature overview
This guide highlights the following Astro features:
- The Astro API
Deploy
endpoint to create and manage code deploys to an Astro Deployment.
Prerequisites
This guide assumes that you have:
- An API token with sufficient permissions to deploy code to Astro.
- At least one Astro Deployment.
- An Astro project that’s accessible from the machine that is making a request to the Astro API.
- Docker, or an alternative container service like Podman.
- The following values:
- Your Organization ID - See List Organizations.
- The Deployment ID - See List Deployments.
- Your Astro API token - See Create an API token.
- The path where your Astro project exists.
With dag-only deploys enabled
If you have dag-only deploys enabled, you can create scripts for complete project deploys, dag-only deploys, or image-only deploys. The following sections include a step by step description of the workflow followed by a bash script that executes the workflow.
See Deploy code to Astro for more information about the different ways to update your dags and Airflow images.
Complete project deploy
The following steps describe the different actions that the script performs to deploy a complete Astro project. Refer to What happens during a project deploy to learn the details about how a project deploy builds and deploys a new Astro image along with deploying dags.
- Make a
POST
request to theDeploy
endpoint to create a newdeploy
object. In your call, specifytype
asIMAGE_AND_DAG
. Store the values forid
,imageRepository
,imageTag
, anddagsUploadURL
that are returned in the response to use in the following steps.
This action creates an object that represents the intent to deploy code to a Deployment. See the Astro API documentation for request examples.
<image-registry-hostname>
value in the following steps with the hostname returned by the API for imageRepository
.-
Authenticate to Astronomer’s image registry using your Astro API token:
-
Build the image using the
imageRepository
andimageTag
values that you retrieved in Step 1, as well as your Astro project path. -
Push the image using the
imageRepository
andimageTag
values that you retrieved in Step 1. -
Create a tar.gz file of your Astro project dags folder:
Make sure to clean up thedags.tar.gz
file after uploading. -
Upload the tar.gz file by making a
PUT
call using thedagsUploadURL
that you retrieved in Step 1. In this call, it is mandatory to pass thex-ms-blob-type
asBlockBlob
. Then, save thex-ms-version-id
from the response header. -
Using your deploy
id
, make a request to finalize the deploy. See Astro API documentation for more information about formatting the API request.- On
Success
, your dags have successfully uploaded and ax-ms-version-id
of the dags tarball is generated in the response headers. Pass thisx-ms-version-id
in the requested body to finish your updates. - It might take a few minutes for the changes to update in your Deployment.
- On
Complete project deploy script
Dag-only deploy
The following script allows you to update only your dag files. Refer to Deploy dags to learn the details about how Astro deploys dags.
-
Make a
POST
request to theDeploy
endpoint to create a newdeploy
object. In your request, specifytype
asDAG_ONLY
. Store the values forid
anddagsUploadURL
that are returned in the response to use in the following steps.This action creates an object that represents the intent to deploy code to a Deployment. See the Astro API documentation for request usage and examples.
-
Create a tar.gz file of your Astro project dags folder:
Make sure to clean up thedags.tar.gz
file after uploading. -
Upload the tar file by making a
PUT
request using thedagsUploadURL
that you retrieved in Step 2. In this request, it is mandatory to pass thex-ms-blob-type
asBlockBlob
. Then, save thex-ms-version-id
from the response header. -
Using your deploy
id
, make a request to finalize the deploy. See Astro API documentation for more information about formatting the API request.- On
Success
, your dags have successfully uploaded and ax-ms-version-id
of the dags tarball is generated. Pass thisx-ms-version-id
in the requested body to finish your updates. - It might take a few minutes for the changes to update in your Deployment.
- On
dag-only deploy script
Image-only deploy
The following script allows you to update only your Astro project by building and deploying a new Docker image. Refer to What happens during a project deploy to learn the details about how Astro deploys image updates.
-
Make a
POST
request to theDeploy
endpoint to create a newdeploy
object. In your request, specifytype
asIMAGE_ONLY
. Store the value for theDeployID
that is returned. This action creates an object that represents the intent to deploy code to a Deployment. See the Astro API documentation for request usage and examples. -
Log in to Docker with your Astro API token.
-
Build the image using the
imageRepository
andimageTag
values that you retrieved in Step 2, as well as your Astro project path. -
Push the image using the
imageRepository
andimageTag
values that you retrieved earlier. -
Using your deploy
id
, make a request to finalize the deploy. See Astro API documentation for more information about formatting the API request.- On
Success
, your dags have successfully uploaded and ax-ms-version-id
of the dags tarball is generated. Pass thisx-ms-version-id
in the requested body to finish your updates. - It might take a few minutes for the changes to update in your Deployment.
- On
Image-only deploy script
With dag-only deploy disabled
You can only use complete project deploys if you have dag-only deploys disabled. Image-only and dag-only deploys do not work.
See Deploy code to Astro for more information about the different ways to update your dags and Airflow images.
Complete project deploy
-
Create the
Deploy
API call using theIMAGE_AND_DAG
type. This action creates an object that represents the intent to deploy code to a Deployment. -
After you create the
deploy
, retrieve theid
,imageRepository
, andimageTag
values using theGET
deploy API call, which you need in the following steps. -
Log in to Docker with your Astro API token.
-
Build the Docker image using the
imageRepository
,imageTag
, and Astro project path values that you retrieved earlier. -
Push the Docker image using the
imageRepository
andimageTag
values that you retrieved earlier. -
Finalize the deploy. See Finalize the deploy for more information about the API request.
- On
Success
, the new image has successfully uploaded. Since you didn’t update any dags in this deploy, pass the requested body as empty,({})
. - It might take a few minutes for the changes to update in your Deployment.
- On
Complete project deploy script
(Recommended) Dynamic deploys based on files changed
In addition to manually triggering project, dag, and image deploys, you can create scripts that trigger specific code deploys depending on whether there have been changes to dags or files used to build the project image.
In this section, you can read a description of the process that the following code example shows the recommended way to combine the previous scripts to trigger different code deploys depending on the files changed in your project. This example demonstrates how to combine all three types of deploys and the conditional logic for when to deploy each.
This recommended process requires that you enable dag-only deploys.
Dynamic deploys description
In the following code example, the script completes the following processes:
-
Determine whether dag files have been changed. If only dags have changed, then the script initiates a dag-only deploy.
-
Make a
POST
request to theDeploy
endpoint to create a newdeploy
object. -
If only dags have changed, the script initiates a dags-only deploy. If you changed more files, the script builds and deploys the project image, and then completes a dag-only deploy.
-
The script cleans up any tar files created during the build process.
To run the script, set the different environment variables to the values for your environment.