Back-end Engineer | Core Services

Engineering | Full-time, Remote

Astronomer is the commercial developer of Apache Airflow, a community-driven open-source tool that’s leading the market in data orchestration. We’re a globally-distributed and rapidly growing venture-backed team of learners, innovators and collaborators. Our mission is to build an Enterprise-grade product that makes it easy for data teams at Fortune 500’s and startups alike to adopt Apache Airflow. As a member of our team, you will be at the forefront of the industry as we strive to make Apache Airflow the de-facto standard in data orchestration.

A Core Services Engineer at Astronomer builds the services infrastructure and API layers that form the connective tissue between Astronomer Cloud’s control and orchestration planes - you will be responsible for conceiving, designing, and implementing components and interfaces that underpin key functionality within Astronomer’s enterprise grade, managed Airflow service.

What you get to do

  • Design and implement the API layer of AstronomerCloud, including building custom first-party APIs and integrating them with third-party APIs, with a focus on scalability and extensibility (e.g., selecting the right systems to integrate, designing tight control over data representations and abstractions, etc.)
  • Build with security in mind, connecting Astronomer’s managed platform can securely integrate with any customer environment
  • Design and implement system-level features and architectural improvements - e.g., converting the execution environment from synchronous to event-driven asynchronous
  • Build Kubernetes infrastructure (helm charts, operators, …) to effectively and efficiently manage k8s deployments on behalf of customers

What you should know

  • Data engineering fundamentals: Scheduling theory and scheduler implementation, virtualization and containerization
  • Distributed system building: database system implementation and internals, distributed storage structure, query languages, caching layer architecture, cluster computing
  • Distributed computing theory: distributed job execution, concurrency, shared-memory model, message-passing model, complexity measures, election algorithms
  • Networking basics: protocol functionality (TCP/IP, HTTP), RPC-like connectors, message queues

What you've already done..or..what you want to do

  • Built entire or significant portions of large, complex SaaS infrastructure operating at scale (e.g., hundreds of thousands of users, thousands of customers, tens of thousands of jobs executing simultaneously)…
  • Developed greenfield cloud-based systems with service-oriented architectures, and/or added new features or services to a large complex system with performance constraints…
  • Built tools (e.g., internal APIs, debuggers, etc.) for streamlining internal development and external deployment of new services and features and enhancing developer productivity…
  • Designed services for resiliency and high availability (e.g., implemented replicated storage and multi-homing in production deployments)...
  • Led a team of multiple people designing, from scratch, a complex service (e.g., at the complexity level of a Google Maps navigation service, Slack notification, etc.), encompassing both individual contribution and delegation and tracking of other engineers on related tasks...
  • Been a top contributor or senior engineer building large, complex distributed systems, be they proprietary (e.g., Google Spanner, BigTable, CFS, Google search infrastructure, etc.) or open source (e.g., Apache Kafka or Spark, Postgres, etc.)...
  • Worked with domain-specific languages, learned the design principles of parser and compiler construction, and gained proficiency in a diverse set of programming languages...
At Astronomer, we value diversity. We are an equal opportunity employer: we do not discriminate on the basis of race, religion, color, national origin, gender, sexual orientation, age, marital status, veteran status, or disability status.  Astronomer is a remote-first company.
Apply Now