How to build reliable data products with Astro
Hosted By
We’ll cover how we think about data products at Astronomer and highlight best practices learned from the most critical data product applications we’ve seen.
Astronomer Webinars
Join us for upcoming online events!
Implementing reliable ETL & ELT pipelines with Airflow and Snowflake
In this webinar, we’ll cover everything you need to know to orchestrate ETL operations in Snowflake with Airflow.
Watch Recorded Webinars On-Demand
Organizing Your Airflow Project Code with the Astro CLI
One of the benefits of Airflow is having pipelines as Python code, which lets you treat your data pipelines like any other piece of software. In this “Live with Astronomer” session, we’ll dive into how to use the open-source Astro CLI to effectively manage your Airflow project code so you can share code with your team, test DAGs before you deploy them, keep your code organized for easy reviews, and more.
Airflow 101: Essential Tips For Beginners - Dec 2022
What is Airflow? Apache Airflow® is a platform used to programmatically author, schedule, and monitor data pipelines.
Data-Aware Scheduling with the Astro Python SDK
Live with Astronomer dives into implementing data-aware scheduling with the Astro Python SDK. The new Airflow Datasets feature allows you to schedule DAGs based on updates to your data and easily view cross-DAG relationships. This feature is part of the Astro Python SDK, so it requires almost no effort from the DAG author to implement. We'll show you everything you need to do (and don't need to do) to take advantage of Datasets.
Running Airflow Tasks in Isolated Environments
Running tasks in a separate environment can help you avoid common data pipeline issues, like dependency conflicts or out-of-memory errors, and it can save resources. Airflow DAG authors have multiple options for running tasks in isolated environments. In this webinar, we'll cover everything you need to know.
How to Migrate from Oozie to Airflow: A Guided Walkthrough
Migrating between orchestrators can be a difficult process fraught with technical and organizational hurdles. However, the end result of applying Airflow’s orchestration capabilities is worth the effort, and working with the right partner can make this journey much easier. In this webinar, we’ll cover everything you need to know about migrating from Oozie to Airflow.
The New DAG Schedule Parameter
Live with Astronomer will discuss the new consolidated `schedule` parameter introduced in Airflow 2.4. We’ll provide a quick refresher of scheduling concepts and discuss how scheduling DAGs is easier and more powerful in newer versions of Airflow.
Batch Inference with Airflow and SageMaker
We’ll walk through using the new SageMaker Async Operators, as well as the new SageMaker OpenLineage integration, for end-to-end ML OPs for batch inference use cases.
Dynamic Task Mapping on Multiple Parameters
On October 25, Live with Astronomer will dive into updates to the dynamic task mapping feature released in Airflow 2.4. We’ll show a couple of new methods for mapping over multiple parameters, and discuss how to choose the best mapping method for your use case.
Dynamic Tasks in Airflow
With the releases of Airflow 2.3 and 2.4, users can write DAGs that dynamically generate parallel tasks at runtime. In this webinar, we’ll cover everything you need to know to implement dynamic tasks in your DAGs.
