> ## Documentation Index
> Fetch the complete documentation index at: https://astronomer.io/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Install Python packages from private sources

Python packages can be installed into your image from both public and private sources. To install packages listed on private PyPI indices or a private git-based repository, you need to complete additional configuration in your project.

Depending on where your private packages are stored, use one of the following setups to install these packages to an Astro project by customizing your Runtime image.

<Info>Deploying a custom Runtime image with a CI/CD pipeline requires additional configurations. For an example implementation, see the GitHub Actions CI/CD templates for [Astro](/docs/astro/ci-cd-templates/github-actions-template) and [Astro Private Cloud](https://www.astronomer.io/docs/software/ci-cd#github-actions-cicd).</Info>

## Setup

<Tabs>
  <Tab title="Private GitHub Repo">
    #### Install Python packages from private GitHub repositories

    This topic provides instructions for building your Astro project with Python packages from a private GitHub repository. Although GitHub is used in these examples, the same approach works with any hosted Git repository.

    <Tabs>
      <Tab title="Using .netrc build secrets">
        Use the `--build-secrets` flag to pass `.netrc` credentials at build time. This approach lets you add private packages to your `requirements.txt` file without custom Dockerfile configuration.

        <Info>This method requires Astro Runtime 3.1-14+, 3.0-15+, or later than 13.5.1.</Info>

        #### Prerequisites

        * The [Astro CLI](/docs/cli/v1.39/overview)
        * An [Astro project](/docs/cli/v1.39/develop-project#create-an-astro-project)
        * Custom Python packages that are [installable with pip](https://packaging.python.org/en/latest/tutorials/packaging-projects/)
        * A private GitHub repository for each of your custom Python packages
        * The [GitHub CLI](https://cli.github.com/) or a [GitHub personal access token](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens)

        #### Step 1: Specify the private repository in your project

        Add your private packages to your `requirements.txt` file using HTTPS URLs in the following format:

        ```text wrap theme={null}
        git+https://github.com/<your-github-organization-name>/<your-private-repository>.git
        ```

        For example, to install `mypackage1` and `mypackage2` from `myorganization`:

        ```text wrap theme={null}
        git+https://github.com/myorganization/mypackage1.git
        git+https://github.com/myorganization/mypackage2.git
        ```

        #### Step 2: Configure credentials

        Define the `NETRC_CONTENT` environment variable in your shell profile (`.bashrc` or `.zshrc`):

        ```bash wrap theme={null}
        export NETRC_CONTENT="machine github.com login oauth2 password $(gh auth token)"
        ```

        If you don't use the GitHub CLI, replace `$(gh auth token)` with a [GitHub personal access token](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens) that has access to your private repositories.

        #### Step 3: Run with build secrets

        Pass the `--build-secrets` flag when running Astro CLI commands:

        ```bash wrap theme={null}
        astro dev start --build-secrets id=netrc,env=NETRC_CONTENT
        ```

        To run tests:

        ```bash wrap theme={null}
        astro dev pytest --build-secrets id=netrc,env=NETRC_CONTENT
        ```

        The `--build-secrets` flag securely provides the `.netrc` content during the Docker build without storing credentials in the image.

        #### Deploy with GitHub Actions

        To build your image in a GitHub Actions workflow, pass the `.netrc` content as a build secret:

        ```yaml wrap theme={null}
        - name: Build image
          uses: docker/build-push-action@v4
          with:
            context: .
            secrets: |
              netrc=machine github.com login oauth2 password ${{ secrets.GITHUB_TOKEN }}
        ```

        #### Deploy with the Astronomer deploy action

        To deploy using the [Astronomer deploy action](https://github.com/astronomer/deploy-action), pass the `.netrc` content through `build-secrets` and set the `NETRC_CONTENT` environment variable:

        ```yaml wrap theme={null}
        - name: Deploy to Astro
          uses: astronomer/deploy-action@v0.x
          with:
            build-secrets: id=netrc,env=NETRC_CONTENT
            # ... deployment-id, etc.
          env:
            NETRC_CONTENT: "machine github.com login oauth2 password ${{ secrets.GITHUB_TOKEN }}"
        ```
      </Tab>

      <Tab title="Using SSH keys">
        <Info>The following setup has been validated with only a single SSH key. You might need to modify this setup when using more than one SSH key per Docker image.</Info>

        #### Prerequisites

        * The [Astro CLI](/docs/cli/v1.39/overview)
        * An [Astro project](/docs/cli/v1.39/develop-project#create-an-astro-project)
        * Custom Python packages that are [installable with pip](https://packaging.python.org/en/latest/tutorials/packaging-projects/)
        * A private GitHub repository for each of your custom Python packages
        * A [GitHub SSH private key](https://docs.github.com/en/authentication/connecting-to-github-with-ssh/generating-a-new-ssh-key-and-adding-it-to-the-ssh-agent) authorized to access your private GitHub repositories

        This setup assumes that each custom Python package is hosted within its own private GitHub repository. Installing multiple custom packages from a single private GitHub repository is not supported.

        <Warning>If your organization enforces SAML single sign-on (SSO), you must first authorize your key to be used with that authentication method. See [Authorizing an SSH key for use with SAML single sign-on](https://docs.github.com/en/enterprise-cloud@latest/authentication/authenticating-with-saml-single-sign-on/authorizing-an-ssh-key-for-use-with-saml-single-sign-on).</Warning>

        #### Step 1: Specify the private repository in your project

        To add a Python package from a private repository to your Astro project, specify the Secure Shell (SSH) URL for the repository in a new `private-requirements.txt` file. Use the following format for the SSH URL:

        ```text wrap theme={null}
        git+ssh://git@github.com/<your-github-organization-name>/<your-private-repository>.git
        ```

        For example, to install `mypackage1` and `mypackage2` from `myorganization`, add the following to your `private-requirements.txt` file:

        ```text wrap theme={null}
        git+ssh://git@github.com/myorganization/mypackage1.git
        git+ssh://git@github.com/myorganization/mypackage2.git
        ```

        This example assumes that the name of each of your Python packages is identical to the name of its corresponding GitHub repository. In other words,`mypackage1` is both the name of the package and the name of the repository.

        #### Step 2: Update Dockerfile

        1. (Optional) Copy and save any existing build steps in your `Dockerfile`.

        2. Add the following to your `packages.txt` file:

           ```bash wrap theme={null}
           openssh-client
           git
           ```

        3. In your Dockerfile, add the following instructions:

           ```docker wrap theme={null}
           USER root
           RUN mkdir -p -m 0700 ~/.ssh && \
               echo "github.com ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOMqqnkVzrm0SdG6UOoqKLsabgH5C9okWi0dh2l9GKJl" >> ~/.ssh/known_hosts

           COPY private-requirements.txt .
           RUN --mount=type=ssh,id=github pip install --no-cache-dir --requirement private-requirements.txt
           USER astro

           ENV PATH="/home/astro/.local/bin:$PATH"
           ```

           In order, these instructions:

           * Switch to `root` user for SSH setup and installation from private repository
           * Add the fingerprint for GitHub to `known_hosts`
           * Copy your `private-requirements.txt` file into the image
           * Install Python-level packages from your private repository as specified in your `private-requirements.txt` file. This securely mounts your SSH key at build time, ensuring that the key itself is not stored in the resulting Docker image filesystem or metadata.
           * Switch back to `astro` user
           * Add the user bin directory to `PATH`

        <Info>
          See GitHub's documentation for all available [SSH key fingerprints](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/githubs-ssh-key-fingerprints).

          If your repository isn't hosted on GitHub, replace the fingerprint with one from where the package is hosted. Use `ssh-keyscan` to generate the fingerprint.
        </Info>

        #### Step 3: Build a custom Docker image

        1. Run the following command to automatically generate a unique image name:

           ```sh wrap theme={null}
           image_name=astro-$(date +%Y%m%d%H%M%S)
           ```

        2. Run the following command to create a new Docker image from your `Dockerfile`. Replace `<ssh-key>` with your SSH private key file name.

           ```sh wrap theme={null}
           DOCKER_BUILDKIT=1 docker build -f Dockerfile --progress=plain --ssh=github="$HOME/.ssh/<ssh-key>" -t $image_name .
           ```

        3. (Optional) Test your Dags locally. See [Restart your local environment](/docs/cli/v1.39/run-airflow-locally#restart-a-local-airflow-environment).

        4. Deploy the image to Astro using the Astro CLI:

           ```sh wrap theme={null}
           astro deploy --image-name $image_name
           ```

        Your Astro project can now use Python packages from your private GitHub repository.
      </Tab>
    </Tabs>
  </Tab>

  <Tab title="Private PyPi Index">
    #### Install Python packages from a private PyPI index

    Installing Python packages on Astro from a private PyPI index is required for organizations that deploy a [private PyPI server (`private-pypi`)](https://pypi.org/project/private-pypi/) as a secure layer between pip and a Python package storage backend, such as GitHub, AWS, or a local file system or managed service.

    To complete this setup, you'll specify your privately hosted Python packages in `requirements.txt`.

    #### Prerequisites

    * An [Astro project](/docs/cli/v1.39/develop-project#create-an-astro-project)
    * A private PyPI index with a corresponding username and password

    #### Add Python packages to your Astro project

    To install a Python package from a private PyPI index, add the package name and version to the `requirements.txt` file of your Astro project.

    If you don't specify a version, Astro installs the latest version. Use the same syntax for installing private packages that you use when you add public packages from [PyPI](https://pypi.org). Your `requirements.txt` file can contain both publicly accessible and private packages.

    At the beginning of the `requirements.txt` file, add `--index-url https://myuser:example.com/api/pypi/pypi/simple` for Airflow 3 based projects and `--extra-index-url https://myuser:example.com/api/pypi/pypi/simple` for Airflow 2 based projects. Replace this example URL with your index's location.

    #### Example `requirements.txt`

    ```requirements wrap theme={null}
    --index-url https://myuser:example.com/api/pypi/pypi/simple
    pandas
    my-custom-package-company
    apache-airflow-providers-snowflake
    ```

    <Warning>
      If your organization configures pip globally with `/etc/pip.conf`, ensure that the following is included:

      ```text wrap theme={null}
      [global]
      constraint = /etc/pip-constraints.txt
      ```

      The `constraint` setting ensures that dependencies are pinned to versions compatible with your Airflow version on Astro. Omitting the `constraint` setting can lead to compatibility issues.
    </Warning>

    Your Astro project can now use Python packages from your private PyPi index.
  </Tab>
</Tabs>