Develop ML Pipelines with the Astro Cloud IDE
Hosted By
In this webinar, we’ll show how the Astro Cloud IDE is the easiest way to develop and test your ML pipelines and schedule them with Airflow.
Astronomer Webinars
Join us for upcoming online events!
Past Webinars
Dynamic Task Mapping
On May 24, Live with Astronomer will dive into the Dynamic Task Mapping feature introduced in Airflow 2.3. We’ll show how to easily add dynamic tasks to your DAGs, and discuss ways to make the best use of this feature.
Astronomer Providers
This webinar will dive into the Astronomer Providers repository, which includes Airflow Providers containing Deferrable Operators and Sensors created by Astronomer. We’ll go beyond the basics to look at key implementation details and best practices.
What’s New in Airflow 2.3
The Airflow project is rapidly evolving, with frequent releases bringing advancements in DAG authoring, observability, and project stability. We’re super excited for the release of Airflow 2.3, which comes with big changes in the flexibility of DAG creation, improvements to the Airflow UI, and much more.
What is OpenLineage
Live with Astronomer will dive into the OpenLineage Airflow integration. We'll cover the basics for getting started with OpenLineage, how lineage helps with everything from recovering from failures to widespread data governance, and additional topics that will be essential as you begin your journey.
Using Airflow as a Data Analyst
Airflow is sometimes thought of as primarily a data engineering tool, but its use cases are really much broader. A data analyst’s workflow typically involves ingesting and transforming data to extract insights, then presenting the insights in a manner that allows business stakeholders to easily interpret trends and take appropriate action. Airflow’s ease of use and extensive provider ecosystem make it an ideal tool for orchestrating such analytics workflows.
OpenLineage and Airflow: A Deeper Dive
Data lineage is the complex set of relationships between your jobs and datasets. Using OpenLineage with Apache Airflow, you can observe and analyze these relationships, allowing you to find and fix issues more quickly. This webinar will provide a deeper dive on OpenLineage, extending beyond the basics into key implementation details and best practices.
Improve Your DAGs with Hidden Airflow Features
Apache Airflow is flexible and powerful. It has a rich ecosystem and an incredibly active community. But are you sure you haven’t missed anything? A new feature or concept that could put your DAGs at another level? It can be challenging to keep up with the latest Airflow features, and sometimes we miss the most useful ones. For this webinar, I'd like to introduce you to a couple of lesser-known features of Apache Airflow that can dramatically improve your data pipelines.
Scaling Out Airflow
Airflow is purpose-built for high-scale workloads and high availability on a distributed platform. Since the advent of Airflow 2.0, there are even more tools and features to ensure that Airflow can be scaled to accommodate high-throughput, data-intensive workloads. In this webinar, Alex Kennedy will discuss the process of scaling out Airflow utilizing the Celery and Kubernetes Executor, including the parameters that need to be tuned when adding nodes to Airflow and the thought process behind deciding when it’s a good idea to scale Airflow, horizontally and vertically. Consistent and aggregated logging is key when scaling Airflow, and we will also briefly discuss best practices for logging on a distributed Airflow platform, as well as the pitfalls that many Airflow users experience when designing and building their distributed Airflow platform.
Data Quality Use Cases with Airflow and Great Expectations
At this webinar, Benji Lampel (Enterprise Platform Architect @ Astronomer) and Tal Gluck (Software Engineer @ Superconductive) will present several Airflow DAGs using Great Expectations that cover more advanced DAG patterns and data quality checking cases.
