Astro Private Cloud features

Astro Private Cloud (APC) offers a suite of best-in-class capabilities to help build, secure, scale, and monitor your self-hosted enterprise data platform built around Apache Airflow.

Run, scale, and optimize Airflow across platforms:

  • Kubernetes-Native Deployment: Astro Private Cloud runs on your own Kubernetes cluster (EKS, GKE, AKS, Openshift or other). Leverage Kubernetes for service coordination, communication, and fault tolerance.
  • Run Cross-Cloud and Cross-Region: Manage Airflow Deployments across multiple Kubernetes clusters, regardless of cloud provider or region.
  • Airflow Deployment Lifecycle Management: Create, update, and delete Airflow Deployments.
  • In-Place Upgrades: Update to the latest Airflow version without costly downtime or lengthy migration processes. Manage upgrades independently from platform updates.
  • Airflow Rollbacks: Roll back to earlier Airflow versions, and track update and rollback history.
  • Multiple Executor Support: Choose between Kubernetes Executor for dynamic task isolation or Celery Executor for distributed task processing. Automatically scale workers based on workload demands.
  • Resource Management: Dynamically scale resources per Airflow Deployment. Adjust CPU, memory, and worker counts through the UI to match your workload requirements.
  • Private Docker Registry: Each Airflow Deployment maintains its own Docker image with custom libraries and environment settings. Images are automatically built and pushed to your private registry.
  • Deployment Isolation: Each Airflow Deployment runs in its own Kubernetes namespace, providing data isolation and protecting against noisy neighbors.
  • Flexible Dag Deployment Options: Deploy Airflow dags via the following options
    • Image-based deploys (Dags baked into container image),
    • Dag-only deploys (Dag bundles pushed to running Airflow Deployments), git-sync per Pod (no shared volume),
    • Git-sync relay (local repo clones reduce external git load),
    • Object storage (e.g. S3/GCS/Azure Blob),
    • RWX storage classes,
    • NFS shared volume (enables a single shared clone of your Dag repository per namespace, giving all Airflow components (scheduler, webserver, workers, triggerer) a consistent dag view, reducing Pod cold-start time, and minimizing network traffic, disk usage, and credential copies)
  • Environment Deletion Cleanup: Automated infrastructure and database cleanup when Airflow Deployments are deleted.
  • Extensible Platform: Bring your own Postgres or MySQL database; Bring your own container registry, ingress controller, Elasticsearch; export logs and metrics to tools of your choice.

Comprehensive Monitoring and Observability

  • Centralized Airflow and Platform Performance Dashboards: Pre-built Grafana dashboards visualize Airflow and platform metrics. Create custom dashboards to meet specific monitoring needs.
  • Centralized Metrics: Track scheduler performance, task success rates, and resource utilization. Centralized time-series metrics collection in Prometheus for both platform and Deployment-level monitoring.
  • Alert Manager: Configure email alerts based on platform and infrastructure health metrics. Get notified of issues such as slow schedulers or resource constraints.
  • Centralized Logging: Elasticsearch and Kibana provide powerful log search and visualization across all Airflow Deployments. Vector automatically collects and indexes Airflow logs.

Security and governance for highly-sensitive workloads

  • Deploy in Air-Gapped or Restricted Network Environments: Run Astro Private Cloud entirely within your own environment. Maintain full control over data location and network security boundaries.
  • Tenant Isolation: Run each Airflow Deployment in its own Kubernetes namespace or cluster with:
    • Resource isolation: CPU, memory, and storage limits per Deployment
    • Network isolation: Network policies to control traffic between Deployments
    • RBAC isolation: Service accounts and roles scoped to specific namespaces
  • Role-Based Access Control (RBAC): Granular access control at Platform, Workspace, and Airflow Deployment levels. Three role types (Admin, Editor, Viewer) map directly to Airflow RBAC permissions.
  • Enterprise SSO Integration: Integrate with major identity providers including Okta, Auth0, Microsoft Entra ID (Azure AD), Google OAuth, and AWS Cognito. Support for OpenID Connect (OIDC) and custom OAuth flows.
  • SCIM Provisioning: Automatically provision and deprovision users and teams based on your identity provider. Maintain centralized user management and access control.
  • Service Accounts: Create Deployment-level or Workspace-level service accounts for CI/CD pipelines and API automation. Generate API keys with specific permission scopes.
  • Network Security: NGINX ingress controller enforces authentication and manages traffic routing out-of-the-box; option to bring-your-own ingress controller. TLS encryption for all communications between components.
  • Secrets Management: Securely store identity provider credentials and API secrets as encrypted Kubernetes secrets.
  • CVE SLAs: All Astro Private Cloud container images are security hardened and come with CVE mitigation and remediation SLAs.
  • Run without Cluster Permissions: Install the Astro Private Cloud platform and Airflow Deployments with namespace permissions only.

Developer Productivity and Platform Automation:

  • Newest Airflow Features: Astro Private Cloud supports Airflow 2 and Airflow 3.
  • Houston API: Automate all platform operations with a GraphQL API.
  • Astro CLI: Install, run, and test Airflow from your command line. Launch a local Airflow stack using Docker for development and testing of dags, hooks, and operators.
  • Astro Private Cloud UI: Modern web-based interface to create and manage Workspaces and Airflow Deployments. Scale resources up or down per Airflow Deployment, invite users, and monitor Airflow logs from a centralized dashboard.
  • CI/CD Integration: Seamlessly integrate with popular CI/CD tools including GitHub Actions, GitLab, Jenkins, CircleCI, and AWS CodeBuild. Use service accounts to authenticate and automate Deployments.
  • Dag-Only Deploys: A push-based service to update dags in running Airflow Deployments without rebuilding container images or requiring shared volumes, enabling rapid iteration and Deployment of dags.
  • Astronomer Registry: Discover over 1,500 integrations and dag templates to accelerate workflow development. See Astronomer Registry.

Committer-led Support

  • 24x7x365 Support: Access to the world’s leading Airflow experts and committers.
  • Education, Enablement, and Certification: Build Airflow expertise across your organization with diverse training and certification options.