Install Python packages from private sources
Python packages can be installed into your image from both public and private sources. To install packages listed on private PyPI indices or a private git-based repository, you need to complete additional configuration in your project.
Depending on where your private packages are stored, use one of the following setups to install these packages to an Astro project by customizing your Runtime image.
Setup
Private GitHub Repo
Private PyPi Index
Install Python packages from private GitHub repositories
This topic provides instructions for building your Astro project with Python packages from a private GitHub repository.
Although GitHub is used in this example, you should be able to complete the process with any hosted Git repository.
Prerequisites
- The Astro CLI
- An Astro project.
- Custom Python packages that are installable with pip
- A private GitHub repository for each of your custom Python packages
- A GitHub SSH private key authorized to access your private GitHub repositories
This setup assumes that each custom Python package is hosted within its own private GitHub repository. Installing multiple custom packages from a single private GitHub repository is not supported.
Step 1: Specify the private repository in your project
To add a Python package from a private repository to your Astro project, specify the Secure Shell (SSH) URL for the repository in a new private-requirements.txt
file. Use the following format for the SSH URL:
For example, to install mypackage1
and mypackage2
from myorganization
, add the following to your private-requirements.txt
file:
This example assumes that the name of each of your Python packages is identical to the name of its corresponding GitHub repository. In other words,mypackage1
is both the name of the package and the name of the repository.
Step 2: Update Dockerfile
-
(Optional) Copy and save any existing build steps in your
Dockerfile
. -
Add the following to your
packages.txt
file: -
In your Dockerfile, add the following instructions:
In order, these instructions:
- Switch to
root
user for SSH setup and installation from private repo - Add the fingerprint for GitHub to
known_hosts
- Copy your
private-requirements.txt
file into the image - Install Python-level packages from your private repository as specified in your
private-requirements.txt
file. This securely mounts your SSH key at build time, ensuring that the key itself is not stored in the resulting Docker image filesystem or metadata. - Switch back to
astro
user - Add the user bin directory to
PATH
- Switch to
See GitHub’s documentation for all available SSH key fingerprints.
If your repository isn’t hosted on GitHub, replace the fingerprint with one from where the package is hosted. Use ssh-keyscan
to generate the fingerprint.
Step 3: Build a custom Docker image
-
Run the following command to automatically generate a unique image name:
-
Run the following command to create a new Docker image from your
Dockerfile
. Replace<ssh-key>
with your SSH private key file name. -
(Optional) Test your dags locally. See Restart your local environment.
-
Deploy the image to Astro using the Astro CLI:
Your Astro project can now utilize Python packages from your private GitHub repository.