<img height="1" width="1" style="display:none" src="https://www.facebook.com/tr?id=293842391037690&amp;ev=PageView&amp;noscript=1">

Astronomer Blog

Culture / Data Science / Dev / Growth

Why Every Data Scientist Needs A Data Engineer

*Image cred: Eunice Lituanas; estimated reading time: 3 minutes, 32 seconds

The data scientist was deemed the “sexiest job of the 21st century.” The Harvard Business Review reasons that this “hybrid of data hacker, analyst, communicator and trusted adviser” is a rare combination of skills, worth a high paycheck.

Too good to be true? Yes, according to Forbes. Turns out, data scientists spend most of their time (up to 79%!) on the part of their job they hate most.

Topics: big data data science data engineering

Why Are Data Scientists Frustrated?


In this video, Aaron Brongersma (data engineer) asks Viraj Parekh (data analyst/scientist) about the most frustrating part of his job and talks about a how the Astronomer platform has changed the game for one customer's data scientist.  

Topics: big data data science data engineering

Ask RBK: Tell me more about getting the right data pipelines.

Getting data from point A to point B requires data pipelines, whether that's putting Facebook ad data into Redshift or adding a third party data set to a data lake. Considering the fact that popular sources like Salesforce or GitHub house a LOT of data—not all of which is necessary to answer a company's business questions—moving all of it without intentionality can create a lot of "noise." Our answer to getting focused, valuable data is recommending reusable "recipes."

Topics: big data data engineering data pipelines

Data Engineers Talk Data Engineering: A Webinar

Topics: big data data science dev data engineering

The Power of DC/OS, Apache Mesos and Containerization: A Q&A With Mesosphere

*Photo cred: Kai Oberhauser; estimated reading time: 8 minutes and 12 seconds

*When it comes to some of our core needs—speed, ability to manage big data, capacity to scale—Apache Mesos is a no-brainer for us. In fact, last week, Greg Neiheisel wrote a blog post about how we're building our platform with Apache Mesos and DC/OS. And since we're such fans of both, we asked our friends at Mesosphere some questions about their tech and what they're most excited about down the road. Here's what Product Marketing Manager Amr Abdelrazik said: 

Topics: dev

Write With Us

At Astronomer, we love geeking out about things like big data trends, new tech and data engineering. This blog began as a place for every team member to share a slice of his or her expertise, but we’d like to open it up to other contributors, starting with our favorite readers.

Our best posts continue to get regularly and organically shared (even months after being posted) and our overall website traffic growth is averaging 150% per month. So this is a chance to share your original content with a growing community of like-minded folks. Also, feel free to add a 25-word bio that links to a personal or professional website.

Topics: culture experiment

Data Engineering Platform Astronomer Closes $3.5M Financing

The Astronomer platform collects, processes and unifies data, allowing organizations to scale analytics, data science and insights.

Building Next-Generation Data Infrastructure with Apache Mesos and DC/OS

 Photo cred: CarrieLu / Flickr

Estimated reading time: 8 minutes and 52 seconds

Astronomer is a data engineering platform that collects, processes and unifies users' enterprise data, so they can get straight to analytics, data science and—more importantly—insights. We make it easy to capture data from any source and send it to any location, from a custom dashboard for visualization to a database for analysis.

Data is becoming the most valuable resource a company owns, and building pipelines that can accommodate all of a company's use-cases while easily scaling is a challenging problem. We’re lowering the barrier to entry for anyone who wants to put their data to work.

Topics: dev

Normalizing Data for Warehouse Centralization

*Image thanks to William Bout

*Estimated reading time: 8 minutes and 46 seconds

A very common initiative these days is data warehouse centralization across an organization. DWaaS (data warehouse as a service) has become commoditized to the point that organizations of every size can begin setting up a reporting infrastructure starting at only a couple hundred dollars a month. This is really exciting and, when copying data from other structured databases, a relatively simple process.

Topics: big data data science

Innovation Isn't Just for Startups

Image credit: Garrhet Sampson; estimated reading time: 3 minutes and 51 seconds

Among Fortune 500 and startups alike, the proliferation of data silos and use of external data sources is posing a significant challenge. Why? Because organizations know to remain competitive, drive faster growth and reduce costs, they need to be data-driven. And they can’t be driven by just hindsight (traditional reporting) or observation (basic business intelligence)—they need to be driven by foresight (predictive analytics).

Topics: growth data engineering