Develop with the CLIAdd Airflow providers and packages

Install Python packages from private sources

Python packages can be installed into your image from both public and private sources. To install packages listed on private PyPI indices or a private git-based repository, you need to complete additional configuration in your project.

Depending on where your private packages are stored, use one of the following setups to install these packages to an Astro project by customizing your Runtime image.

Deploying a custom Runtime image with a CI/CD pipeline requires additional configurations. For an example implementation, see the GitHub Actions CI/CD templates for Astro and Astro Private Cloud.

Setup

Install Python packages from private GitHub repositories

This topic provides instructions for building your Astro project with Python packages from a private GitHub repository. Although GitHub is used in these examples, the same approach works with any hosted Git repository.

Use the --build-secrets flag to pass .netrc credentials at build time. This approach lets you add private packages to your requirements.txt file without custom Dockerfile configuration.

This method requires Astro Runtime 3.1-14+, 3.0-15+, or later than 13.5.1.

Prerequisites

Step 1: Specify the private repository in your project

Add your private packages to your requirements.txt file using HTTPS URLs in the following format:

git+https://github.com/<your-github-organization-name>/<your-private-repository>.git

For example, to install mypackage1 and mypackage2 from myorganization:

git+https://github.com/myorganization/mypackage1.git
git+https://github.com/myorganization/mypackage2.git

Step 2: Configure credentials

Define the NETRC_CONTENT environment variable in your shell profile (.bashrc or .zshrc):

$export NETRC_CONTENT="machine github.com login oauth2 password $(gh auth token)"

If you don’t use the GitHub CLI, replace $(gh auth token) with a GitHub personal access token that has access to your private repositories.

Step 3: Run with build secrets

Pass the --build-secrets flag when running Astro CLI commands:

$astro dev start --build-secrets id=netrc,env=NETRC_CONTENT

To run tests:

$astro dev pytest --build-secrets id=netrc,env=NETRC_CONTENT

The --build-secrets flag securely provides the .netrc content during the Docker build without storing credentials in the image.

Deploy with GitHub Actions

To build your image in a GitHub Actions workflow, pass the .netrc content as a build secret:

1- name: Build image
2  uses: docker/build-push-action@v4
3  with:
4    context: .
5    secrets: |
6      netrc=machine github.com login oauth2 password ${{ secrets.GITHUB_TOKEN }}

Deploy with the Astronomer deploy action

To deploy using the Astronomer deploy action, pass the .netrc content through build-secrets and set the NETRC_CONTENT environment variable:

1- name: Deploy to Astro
2  uses: astronomer/deploy-action@v0.x
3  with:
4    build-secrets: id=netrc,env=NETRC_CONTENT
5    # ... deployment-id, etc.
6  env:
7    NETRC_CONTENT: "machine github.com login oauth2 password ${{ secrets.GITHUB_TOKEN }}"