Astronomer's the Dataflow Cast

The Future of AI in Data Engineering With Astronomer’s Julian LaNeve and David Xue

The world of data orchestration and machine learning is rapidly evolving, and tools like Apache Airflow are at the forefront of these changes. Understanding how to effectively utilize these tools can significantly enhance data processing and AI model deployment.

This episode features Julian LaNeve, CTO at Astronomer, and David Xue, Machine Learning Engineer at Astronomer. They delve into the intricacies of data orchestration, generative AI and the practical applications of these technologies in modern data workflows.

 

Key Takeaways:

(01:51) The pressure to engage in the generative AI space.

(02:02) Generative AI can elevate data utilization to the next level.

(02:43) The transparency issues with commercial AI models.

(04:27) High-quality data in model performance is crucial.

(06:40) Running new models on smaller devices, like phones.

(12:19) Fine-tuning LLMs to handle millions of task failures.

(16:54) Teaching AI to understand specific logs, not general passages, is a goal.

(21:56) Using Airflow as a general-purpose orchestration tool.

(22:00) Airflow is adaptable for various use cases, including ETL and ML systems.

 

Resources Mentioned:

Thanks for listening to The Data Flowcast: Mastering Airflow for Data Engineering & AI. If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.

#ai #automation #airflow #machinelearning

Be Our Guest

Interested in being a guest on The Data Flowcast? Fill out the form and we will be in touch.

By proceeding you agree to our Privacy Policy,
our Website Terms and to receive emails from Astronomer.

Ready to Get Started?

See how your team can fuel its data workflows with more power and less complexity than ever before.

Start Free Trial →

Which plan works best for your team?

Learn about pricing →

What can Astronomer do for your organization?

Talk to an expert →