Machine Learning
Improve your machine learning platform with the right data orchestration tool! Apache Airflow is a flexible, community-supported, and Python-based solution that allows you to access more data, iterate better, and speed up your work.
Machine Learning and the challenge of complexity
As modern organizations rely more and more on data to make the right business decisions and guarantee their growth, creating successful machine learning processes has never been more crucial. Even though a lot of organizations are machine learning-driven, 90 percent of all machine learning models never make it into production.
Apache Airflow—an end-to-end solution for your machine learning needs
Machine learning engineers point to non-reproducible pipelines and inefficient integration with different databases and tools as the main obstacle to efficient MLOps. Create a robust production environment, using a flexible, reliable, and extensive data orchestrator.
Working with machine learning models in production requires automation and orchestration for repeated model training, testing, evaluation, and likely integration with other services to acquire and prepare data. With Apache Airflow you can easily orchestrate each step of your pipeline, integrate with services that clean your data, and store and publish your results using simple Python scripts for 'configuration as code'.
Productionize
your ML pipelineAirflow allows you to build pieces of a machine learning pipeline easily and systematically. Reuse the same code for different machine learning models and datasets, solving the problem of complexity.
Conduct an end-to-end
ML process from one placeHave full observability over your data and models: from getting the data, cleaning it, and putting your models in production.
Simplify and speed up
ML engineeringApply principles of generalizability, scalability, and reproducibility to machine learning. Airflow is an extensive, Python-based, open-source tool with a wide range of operators, hooks, and modules that can be used and adjusted to your specific needs.
Alexandra Abbas
Machine Learning Engineer at Wise
Airflow has a central place in our machine learning platform because it is responsible for retraining the models in SageMaker. We use Airflow with SageMaker to retrain workflows, spin up the SageMaker training instances, and then train the models regularly.
Gautam Doulani
Data Engineering Lead at CRED
After 6-7 months with Apache Airflow, we’ve built more than ninety DAGs. The tool made the experience so much easier.
Alaeddine Maaoui
Product Owner at Societe Generale
An open source project, such as Apache Airflow, works great in the production environment, even for the sensitive use cases of the banking industry.
Find the Apache Airflow resources you're looking for.
Using Airflow with SageMaker
Amazon SageMaker is a comprehensive AWS machine learning service that is frequently used by data scientists to develop and deploy ML models at scale. Learn how Airflow can work together with the SageMaker and make your machine learning tasks easier.
Airflow at Wise
A talk with Alexandra Abbas—a Machine Learning Engineer at Wise—about how they leverage Apache Airflow in their ML initiatives
Astronomer Registry
A sample data science pipeline demonstrating extraction from BigQuery to modeling that uses an XCom backend in Google Cloud Storage to pass intermediary data between tasks.
Do Airflow the easy way.