Join us for Astro Days: Chicago on December 6!
How to Succeed in the Data Revolution

The target for this article is business leaders of medium/large organizations who are waking up to the potential of the data revolution and want to get into the game—and help their company win.

What Is the Data Revolution? 

From phones, credit cards, computers, wearables, and sensor-equipped devices, data is now streaming into the world at an unprecedented velocity. We’ve shifted into a new gear. It’s exciting, scary, and full of opportunities.

We call this the Data Revolution—a seismic shift in the way the world operates, no less significant than the Agricultural, Industrial, and Digital Revolutions.

Organizations of all sizes in every industry are uncovering groundbreaking, data-driven insights and are putting data to extraordinary uses.

A study by Bain and Company shows how important big data analytics can be for companies in terms of competitive differentiation. When 400 large corporations were examined, they found that the companies with the best big data analytics capabilities were doing better than their competitors by a huge difference. They were two times as likely to be in the top quartile of financial performance within their industries and to use data very frequently when making decisions, three times as likely to execute decisions as intended, and five times as likely to make decisions much faster than market peers. — Sethuraman Janardhanan

We’ve seen this play out with many of our customers. Decision-making confidence has risen as they become more data-driven:

  • A major airport focused on improving customer experience has recognized that relying only on yesterday’s data isn’t enough. They need to develop and share predictions with their experience collaborators (airlines, vendors, passengers) to really move the needle.
  • A leading e-commerce company is leveling up their ability to collect, route and analyze their data—with the goal to get the right products to the right customers and streamline customer communications.
  • A fast-growing church (with 20,000+ regular attenders) recognizes that engagement with their congregation online and via mobile apps is the future, and in the process they realize that in order to understand congregational experience in this new medium, they needed to invest in data and analytics, and hired a full-time data scientist to catalyze a data-driven culture.

Self Assessment

  1. Does your company employ a full or part time data scientist?
  2. Have you centralized your important data for analytics/BI?
  3. Is someone inside your organization focused on pulling insights from existing data assets?
  4. Is someone considering how to put external data (customer, partner, 3rd party, open data) to work for you?
  5. Are you measuring D aily ETL T asks (DETLT)? Is that number growing?

If you answered yes 4 or 5 times, congratulations, you’ve likely moved to more advanced concerns. Otherwise, you may find the prescriptions below useful.

The Prescriptions

  1. Hire a full-time data scientist with the mandate to make the organization more data-driven. Great data scientists are in incredibly high demand right now but you have to find one. It will catalyze a movement within the organization. The end goal is to get to predictive models and forecasts using machine and deep learning.
  2. Hire a couple of data analysts fresh out of college whose job is to support the data scientist, taking on the “low hanging fruit” work that is valuable but comparatively easy—basic reporting and simple models (SQL queries, correlations, regression analysis). Having a small team should ensure mutual accountability and team continuity, and will help your nascent movement progress more quickly.
  3. Allow this team to establish a central data warehouse with technology of their choosing. Let them choose their own tools. Don’t require them to use existing tech.
  4. Establish an accountability mechanism where this new team regularly reports to the entire organization. Ask them to list every potential data source/silo, and to report what percentage of the potential value of that data they’ve captured for the organization.
  5. Challenge the data team to seek external data , not only relying on internal data. Winners in the Data Revolution will push themselves to use data in innovative ways and will engage in data collaboration projects with external parties.
  6. Track your DETLT metric (D aily ETL T asks). It’s simple: how many ETL jobs has your organization automated to run each day? ETL jobs create value. Used correctly, they generate information from raw data , which is the first step towards obtaining insights from raw data. Automating this value creation should be a key goal.  Your accountability report could look something like this. Expect data sources to grow.


You’re Not That Far Behind…

Many 1000+ employee organizations do not have a data scientist or any semblance of a modern data team yet. Many don’t have control of their data. Many are still shackled to technology tools that hold them and their data hostage. But many are also waking up. And threats from innovative, data-savvy startups have them concerned. To win, the time is now to establish a data team and begin putting data to work.

… But the Revolution Marches On

Analytics, Business Intelligence, and even Machine Learning are only the beginning. We’re still in the first phase of the Data Revolution. In the near future, analytics and business intelligence will be “table stakes” because computers are ready to do real work for us.

We can envision a future where humans and machines become workplace collaborators and these machines will only be as smart as the data fed to them. This powerful trend— software can learn from data—is really just starting . But machine learning is very data hungry. It wants data from as many sources as you can provide; providing more data is more effective than trying to improve algorithms.

“Just as all real-world workflows became software, all software will become analytical. Computers will have cognition; machines will have intelligence.”
— Zetta Venture Partners

Many changes are coming, and we don’t know exactly what to expect. But we love doing our part in this exciting Revolution.

We outfit organizations with modern data infrastructure, tailored to their unique needs. If you need help, contact us and let’s get started!

Modern,Cloud-Native Data Orchestration powered by Apache Airflow

Start building your next-generation data platform with Astro.

Get Started