For AI agents: a documentation index is available at the root level at /llms.txt and /llms-full.txt. Append /llms.txt to any URL for a page-level index, or .md for the markdown version of any page.
Astro Private Cloud (APC) offers a suite of best-in-class capabilities to help build, secure, scale, and monitor your self-hosted enterprise data platform built around Apache Airflow.
Run, scale, and optimize Airflow across platforms:
Kubernetes-Native Deployment: Astro Private Cloud runs on your own Kubernetes cluster (EKS, GKE, AKS, Openshift or other). Leverage Kubernetes for service coordination, communication, and fault tolerance.
Run Cross-Cloud and Cross-Region: Manage Airflow Deployments across multiple Kubernetes clusters, regardless of cloud provider or region.
Airflow Deployment Lifecycle Management: Create, update, and delete Airflow Deployments.
In-Place Upgrades: Update to the latest Airflow version without costly downtime or lengthy migration processes. Manage upgrades independently from platform updates.
Airflow Rollbacks: Roll back to earlier Airflow versions, and track update and rollback history.
Multiple Executor Support: Choose between Kubernetes Executor for dynamic task isolation or Celery Executor for distributed task processing. Automatically scale workers based on workload demands.
Resource Management: Dynamically scale resources per Airflow Deployment. Adjust CPU, memory, and worker counts through the UI to match your workload requirements.
Private Docker Registry: Each Airflow Deployment maintains its own Docker image with custom libraries and environment settings. Images are automatically built and pushed to your private registry.
Deployment Isolation: Each Airflow Deployment runs in its own Kubernetes namespace, providing data isolation and protecting against noisy neighbors.
Flexible Dag Deployment Options: Deploy Airflow Dags via the following options
Image-based deploys (Dags baked into container image),
Dag-only deploys (Dag bundles pushed to running Airflow Deployments), git-sync per Pod (no shared volume),
NFS shared volume (enables a single shared clone of your Dag repository per namespace, giving all Airflow components (scheduler, webserver, workers, triggerer) a consistent Dag view, reducing Pod cold-start time, and minimizing network traffic, disk usage, and credential copies)
Environment Deletion Cleanup: Automated infrastructure and database cleanup when Airflow Deployments are deleted.
Extensible Platform: Bring your own Postgres or MySQL database; Bring your own container registry, ingress controller, Elasticsearch; export logs and metrics to tools of your choice.
Comprehensive Monitoring and Observability
Centralized Airflow and Platform Performance Dashboards: Pre-built Grafana dashboards visualize Airflow and platform metrics. Create custom dashboards to meet specific monitoring needs.
Centralized Metrics: Track scheduler performance, task success rates, and resource utilization. Centralized time-series metrics collection in Prometheus for both platform and Deployment-level monitoring.
Alert Manager: Configure email alerts based on platform and infrastructure health metrics. Get notified of issues such as slow schedulers or resource constraints.
Centralized Logging: Elasticsearch provides powerful log search across all Airflow Deployments. Vector automatically collects and indexes Airflow logs.
Security and governance for highly-sensitive workloads
Deploy in Air-Gapped or Restricted Network Environments: Run Astro Private Cloud entirely within your own environment. Maintain full control over data location and network security boundaries.
Tenant Isolation: Run each Airflow Deployment in its own Kubernetes namespace or cluster with:
Resource isolation: CPU, memory, and storage limits per Deployment
Network isolation: Network policies to control traffic between Deployments
RBAC isolation: Service accounts and roles scoped to specific namespaces
Role-Based Access Control (RBAC): Granular access control at Platform, Workspace, and Airflow Deployment levels. Three role types (Admin, Editor, Viewer) map directly to Airflow RBAC permissions.
Enterprise SSO Integration: Integrate with major identity providers including Okta, Auth0, Microsoft Entra ID (Azure AD), Google OAuth, and AWS Cognito. Support for OpenID Connect (OIDC) and custom OAuth flows.
SCIM Provisioning: Automatically provision and deprovision users and teams based on your identity provider. Maintain centralized user management and access control.
Service Accounts: Create Deployment-level or Workspace-level service accounts for CI/CD pipelines and API automation. Generate API keys with specific permission scopes.
Network Security: NGINX ingress controller enforces authentication and manages traffic routing out-of-the-box; option to bring-your-own ingress controller. TLS encryption for all communications between components.
Secrets Management: Securely store identity provider credentials and API secrets as encrypted Kubernetes secrets.
CVE SLAs: All Astro Private Cloud container images are security hardened and come with CVE mitigation and remediation SLAs.
Run without Cluster Permissions: Install the Astro Private Cloud platform and Airflow Deployments with namespace permissions only.
Houston API: Automate all platform operations with a GraphQL API.
Astro CLI: Install, run, and test Airflow from your command line. Launch a local Airflow stack using Docker for development and testing of Dags, hooks, and operators.
Astro Private Cloud UI: Modern web-based interface to create and manage Workspaces and Airflow Deployments. Scale resources up or down per Airflow Deployment, invite users, and monitor Airflow logs from a centralized dashboard.
CI/CD Integration: Seamlessly integrate with popular CI/CD tools including GitHub Actions, GitLab, Jenkins, CircleCI, and AWS CodeBuild. Use service accounts to authenticate and automate Deployments.
Dag-Only Deploys: A push-based service to update Dags in running Airflow Deployments without rebuilding container images or requiring shared volumes, enabling rapid iteration and Deployment of Dags.
Airflow Registry: Discover over 1,500 integrations to accelerate workflow development. See Airflow Registry.
Committer-led Support
24x7x365 Support: Access to the world’s leading Airflow experts and committers.
Education, Enablement, and Certification: Build Airflow expertise across your organization with diverse training and certification options.
