<img height="1" width="1" style="display:none" src="https://www.facebook.com/tr?id=293842391037690&amp;ev=PageView&amp;noscript=1">

Astronomer Blog

Culture / Data Science / Dev / Growth

Apache Airflow and the Future of Data Engineering: A Q&A with Maxime Beauchemin


 

*Estimated reading time: 8 minutes and 42 seconds

Every once in a while I read a post about the future of tech that resonates with clarity.

A few weeks ago it was The Rise of the Data Engineer by Maxime Beauchemin, a data engineer at Airbnb and creator of their data pipeline framework, Airflow. At Astronomer, Airflow is at the very core of our tech stack: our integration workflows are defined by data pipelines built in Airflow as directed acyclic graphs (DAGs). A post like that gives validation as to why right now is the best time for a company like Astronomer to exist.

After reading the post, I reached out to Max about doing an interview post, and to my delight he entertained the request with thoughtful answers to our questions about Airflow and the future of data engineering. You’ll find his answers below, but first I’d like to add a little context.

Topics: dev

Top Ten Customer Data Integrations, According to Our Customers

 

Estimated reading time: 4 minutes and 41 seconds

There’s a good chance, if you’re reading this, that a hodgepodge of data sets are currently accumulating in your organization—collecting social media, blogging, email marketing, SEO, CRM, web analytics and more. Maybe you’re already gathering customer data from favorite tools. Maybe every step of a user's journey across your website or app, every "click," is tracked. (If you don't feel like you have enough data flowing, check out this post and kick off something new.) 

But each data set gives only a snapshot of the whole picture, which, anyone with an Instagram account knows, doesn't tell you evverything. Fortunately, once data is routed through one platform (you know, like Astronomer), new integrations can be turned on and off instantaneously, with the flip of a switch. Why not make the most of that opportunity and add something new? Or switch to a new, better tool, while keeping your data inact? Or just read on and see what kinds of data you can connect, if you wanted to? 

Of course, here at Astronomer, we don’t play favorites. Our customers, on the other hand, absolutely do. They have to, if they’re going to use the tools best-suited to their business needs. So here is a list of their ten favorite clickstream integrations, starting with the most popular.

Topics: growth

MeasureCamp is coming to Cincinnati

The internationally-renowned data analytics unconference, MeasureCamp, is coming to Cincinnati on May 13, 2017! 

MeasureCamp is a free event run by the digital analytics community, for the digital analytics community. The “unconference” format has no scheduled presentations. Instead, attendees exchange ideas and ask each other to create “sessions” on the spot, whether it’s leading a technical conversation or a creative brainstorm. Anyone who wants to join the analytics conversation is welcome to attend.

Translating Real-World Randomness To Create Digital Security

Or, how I used a deck of cards to protect my data.

*Estimated reading time: 11 minutes, 20 seconds

Every year, our data becomes more accessible as the world is increasingly interconnected and more services we depend on become available online. This allows for an ease that was unimaginable just twenty years ago. You can bank, apply for a mortgage and credit cards, share personal photos, store important documents and get a real-time view into your home at any time.

Topics: culture

Scaling off AWS: Exploring Go for High Performance Services

Estimated reading time: 14 minutes, 22 seconds

Several months ago, our CTO wrote about our transition from AWS services to open source counterparts. In that post, he discussed the reasons for building on AWS in the first place, as well as why we now had to move off of it. In essence, Astronomer needs the ability to run in any cloud, so breaking our dependence on any one cloud provider is a must. In addition to the SaaS version of Astronomer, we have an enterprise edition that gives enterprises full control of running the Astronomer platform in their own cloud deployments.

Topics: dev

A Sales Experiment: Why We Scrapped the SDR Role

Estimated reading time: 7 minutes, 21 seconds

“In any moment of decision, the best thing you can do is the right thing. The worst thing you can do is nothing.” - Teddy Roosevelt

In business, decisiveness is of critical importance. Yet decisiveness does not operate in a vacuum. The power to make decisions should not be measured by the results it produces, but by the information fueling the process. Why? Because the information driving a decision causes a result, whereas looking at the result alone only suggests a correlation between the information powering the decision and the subsequent outcome. This begs the question: how does one go about getting the best information in order to make a decision, leading to what Ol’ Teddy calls “the right thing?”

Topics: growth