Previous Events
How to easily test your Airflow DAGs with the new dag.test() function
In this “Live with Astronomer” session, we’ll dive into the new `dag.test()` function, which allows you to debug DAGs directly in your IDE in a single, serialized Python process. We’ll show how this function lets you quickly iterate and debug errors during your DAG development, and even test argument-specific DAG runs.
How to Save Money using Airflow’s asynchronous Azure operators
Many Azure users leverage Airflow for best-in-class orchestration of their Azure services. Recent updates to the Azure provider have brought greater functionality and new Airflow features to Azure operators. In this “Live with Astronomer” session, we’ll dive into the newly developed asynchronous Azure operators that offer cost savings and greater scalability. We’ll show how with only small updates to your DAGs, you can take advantage of asynchronous functionality when orchestrating services like Azure Data Factory and Azure Databricks.
The Airflow Templates VS Code Extension
In this “Live with Astronomer” session, we’ll walk through one of those contributions – new Airflow Templates VS Code extension, which includes code completion for all Airflow Provider operators.
Using the new Fivetran provider
With more than 30,000 downloads per month, the Fivetran provider for Airflow is incredibly popular. Using Fivetran and Airflow together gives users the benefits of first-class orchestration, pipelines as code, and automated ELT processes.
Organizing Your Airflow Project Code with the Astro CLI
One of the benefits of Airflow is having pipelines as Python code, which lets you treat your data pipelines like any other piece of software. In this “Live with Astronomer” session, we’ll dive into how to use the open-source Astro CLI to effectively manage your Airflow project code so you can share code with your team, test DAGs before you deploy them, keep your code organized for easy reviews, and more.
Data Driven Scheduling
In this session, Live with Astronomer explores the new datasets feature introduced in Airflow 2.4. We’ll show how DAGs that access the same data now have explicit, visible relationships, and how DAGs can be scheduled based on updates to these datasets.
Data-Aware Scheduling with the Astro Python SDK
Live with Astronomer dives into implementing data-aware scheduling with the Astro Python SDK. The new Airflow Datasets feature allows you to schedule DAGs based on updates to your data and easily view cross-DAG relationships. This feature is part of the Astro Python SDK, so it requires almost no effort from the DAG author to implement. We'll show you everything you need to do (and don't need to do) to take advantage of Datasets.
The Python Task Decorator
Live with Astronomer will dive into the Python task decorator. We’ll show how to easily turn your Python functions into tasks in your DAG using functional programming, and how using the Python task decorator can limit the boilerplate code needed in your DAGs.
Using the Snowflake Deferrable Operator
Live with Astronomer will dive into using the Snowflake Deferrable Operator. We’ll show how with a very small update to your DAGs, you can start saving money when orchestrating your Snowflake queries with Airflow.
Reusable DAG Patterns with TaskGroups
In this session we’ll show how Astronomer’s data and intelligence team uses TaskGroups to reduce the amount of code the team has to write while adhering to DAG authoring best practices.
The SQL Column Check Operator
In this session we’ll show how you can easily use the SQLColumnCheckOperator operator to implement data quality checks in your DAGs, ensuring that errant data never makes it to production.
The SQL Table Check Operator
In this session we’ll dive into the new Common SQL provider package and show how to use the SQLTableCheckOperator. We’ll show how you can easily use this operator to implement data quality checks in your DAGs, ensuring that errant data never makes it to production.
The Astro Python SDK Load File Function
The next Live with Astronomer will dive into the Astro Python SDK load_file function. The Astro Python SDK is an open source Python package that allows for clean and rapid development on ELT workflows. We’ll show how you can use load_file for the ‘Extract’ step of your pipeline to easily get data from your filesystems into your data warehouse, without any operator-specific knowledge.
Data Transformations with the Astro Python SDK
On September 13, Live with Astronomer will dive into implementing data transformations with the Astro Python SDK. The Astro Python SDK is an open source Python package that allows for clean and rapid development on ELT workflows. We’ll show how you can use the transform and dataframe functions to easily transform your data using Python or SQL and seamlessly transition between the two.
Dynamic Task Mapping
On May 24, Live with Astronomer will dive into the Dynamic Task Mapping feature introduced in Airflow 2.3. We’ll show how to easily add dynamic tasks to your DAGs, and discuss ways to make the best use of this feature.
Dynamic Task Mapping on Multiple Parameters
On October 25, Live with Astronomer will dive into updates to the dynamic task mapping feature released in Airflow 2.4. We’ll show a couple of new methods for mapping over multiple parameters, and discuss how to choose the best mapping method for your use case.
The New DAG Schedule Parameter
Live with Astronomer will discuss the new consolidated `schedule` parameter introduced in Airflow 2.4. We’ll provide a quick refresher of scheduling concepts and discuss how scheduling DAGs is easier and more powerful in newer versions of Airflow.