Install Python packages from private sources

Install Python packages from private GitHub repositories​

This topic provides instructions for building your Astro project with Python packages from a private GitHub repository.

Although GitHub is used in this example, you should be able to complete the process with any hosted Git repository.

Prerequisites​

• The Astro CLI
• An Astro project.
• Custom Python packages that are installable with pip
• A private GitHub repository for each of your custom Python packages
• A GitHub SSH private key authorized to access your private GitHub repositories

This setup assumes that each custom Python package is hosted within its own private GitHub repository. Installing multiple custom packages from a single private GitHub repository is not supported.

Step 1: Specify the private repository in your project​

To add a Python package from a private repository to your Astro project, specify the Secure Shell (SSH) URL for the repository in a new private-requirements.txt file. Use the following format for the SSH URL:

git+ssh://git@github.com/<your-github-organization-name>/<your-private-repository>.git

For example, to install mypackage1 and mypackage2 from myorganization, add the following to your private-requirements.txt file:

git+ssh://git@github.com/myorganization/mypackage1.git
git+ssh://git@github.com/myorganization/mypackage2.git

This example assumes that the name of each of your Python packages is identical to the name of its corresponding GitHub repository. In other words,mypackage1 is both the name of the package and the name of the repository.

Step 2: Update Dockerfile​

1. (Optional) Copy and save any existing build steps in your Dockerfile.

2. Add the following to your packages.txt file:

   openssh-client
   git
   

3. In your Dockerfile, add the following instructions:

   USER root
   RUN mkdir -p -m 0700 ~/.ssh && \
       echo "github.com ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOMqqnkVzrm0SdG6UOoqKLsabgH5C9okWi0dh2l9GKJl" >> ~/.ssh/known_hosts
   
   COPY private-requirements.txt .
   RUN --mount=type=ssh,id=github pip install --no-cache-dir --requirement private-requirements.txt
   USER astro
   
   ENV PATH="/home/astro/.local/bin:$PATH"
   

   In order, these instructions:

   • Switch to root user for SSH setup and installation from private repo
   • Add the fingerprint for GitHub to known_hosts
   • Copy your private-requirements.txt file into the image
   • Install Python-level packages from your private repository as specified in your private-requirements.txt file. This securely mounts your SSH key at build time, ensuring that the key itself is not stored in the resulting Docker image filesystem or metadata.
   • Switch back to astro user
   • Add the user bin directory to PATH

Step 3: Build a custom Docker image​

1. Run the following command to automatically generate a unique image name:

   image_name=astro-$(date +%Y%m%d%H%M%S)
   

2. Run the following command to create a new Docker image from your Dockerfile. Replace <ssh-key> with your SSH private key file name.

   DOCKER_BUILDKIT=1 docker build -f Dockerfile --progress=plain --ssh=github="$HOME/.ssh/<ssh-key>" -t $image_name .
   

3. (Optional) Test your DAGs locally. See Restart your local environment.

4. Deploy the image to Astro using the Astro CLI:

   astro deploy --image-name $image_name
   

Your Astro project can now utilize Python packages from your private GitHub repository.

Install Python packages from a private PyPI index​

Installing Python packages on Astro from a private PyPI index is required for organizations that deploy a private PyPI server (private-pypi) as a secure layer between pip and a Python package storage backend, such as GitHub, AWS, or a local file system or managed service.

To complete this setup, you’ll specify your privately hosted Python packages in requirements.txt.

Prerequisites​

• An Astro project
• A private PyPI index with a corresponding username and password

Add Python packages to your Astro project​

To install a Python package from a private PyPI index, add the package name and version to the requirements.txt file of your Astro project.

If you don’t specify a version, Astro installs the latest version. Use the same syntax for installing private packages that you use when you add public packages from PyPI. Your requirements.txt file can contain both publicly accessible and private packages.

At the beginning of the requirements.txt file, add --index-url https://myuser:example.com/api/pypi/pypi/simple for Airflow 3 based projects and --extra-index-url https://myuser:example.com/api/pypi/pypi/simple for Airflow 2 based projects. Replace this example URL with your index's location.

Example requirements.txt​

--index-url https://myuser:example.com/api/pypi/pypi/simple
pandas
my-custom-package-company
apache-airflow-providers-snowflake

[global]
constraint = /etc/pip-constraints.txt

Your Astro project can now use Python packages from your private PyPi index.