We’re entering a new Internet era due to a confluence of trends:
- Computers are everywhere (mobile, wearables, IoT) and as a result everyone is becoming data-competent. This means more of us will become data-ambitious.
- Cloud computing is entering a major deployment phase—this means data will be in the cloud, staged to flow more fluidly.
- SaaS/microservices proliferation spreads organizational data across borders, into many data silos.
- Computer cognition is emerging, and it’s data hungry.
“Data is incredibly valuable. It helps create superior products, it forms a barrier to entry, and it can be directly monetized.”
Computers are Everywhere
Because we all have super computers in our pocket, on our desk, and soon in devices all around us, responsibility for information technology has become dispersed. People are taking responsibility for their own technology; IT departments are releasing power.
In the same way that organizations who can master their own data will thrive, people who choose to wield their organization’s data will thrive too. The population of these “newly technically confident” people is growing.
But if we ask people in our company — do you understand what your customers are doing? Most people don’t have access to the data — and if they do, it’s well after it has happened. This will change.
Cloud computing is in the deployment phase. For example, GE is 25% done moving 1,000 apps to AWS — it’s a matter of when, not if, the rest of the businesses follow.
As more organizations adopt cloud computing, all their data will be in the cloud, and data is now staged to flow with more fluidity.
“Big data in the cloud democratizes the use of big data beyond large companies.”
If everyone has access to the real-time data, the organization becomes more agile, allows you to iterate and measure effects on your own. This has an effect of flattening of the organization, fostering a collaborative culture, and makes decision making faster.
We are approaching a tipping point where more of an organization’s data is generated and stored outside of the company, than within.
With SaaS, this effect is fractal. If you look inside the company that you think has your data, the majority of their data (which is really your data) is in their SaaS tools, and so on down the rabbit hole.
And even within your organization, your developers are likely adopting a microservice architecture, including “the right database for the job”—these are technically sound decisions, but the effect is that even within your organization, there can be several databases per application, and applications often don’t share databases anymore.
As a result of this, your data is becoming more and more spread out. You need to un-silo the data in a way that doesn’t move your organization backwards towards unsustainable centralization.
Finally the most powerful trend that is just starting — software can learn from data. Once we all have self-driving cars, every job that a human does will be scrutinized — can a self-learning computer do this work?
Analytics and business intelligence are becoming table stakes. Computers are ready to do real work for us. But machine learning is very data hungry. Data from many sources.
Machine learning is incredibly data-hungry, it’s common to find scenarios where just providing more data beats any possible algorithm improvement. So there will be a hunger for all data we can gather, as computers develop cognition and intelligence.
“Just as all real-world workflows became software, all software will become analytical. Computers will have cognition; machines will have intelligence.”
— Zetta Venture Partners
Organizations that are masters of their own data will have a competitive advantage. Data enrichment between companies will grow more common.
Will you isolate like some brick and mortar companies did?
You have a choice… be an isolated island, or put your data on the wire. It reminds me of the dawn of the Internet —when there was serious debate by big companies if they really needed to have a website, especially the brick and mortar companies.
We see the same thing now with data. Any company that provides value creates exhaust data that is valuable, and the smart ones will be prepared to share that data.
“Big data in the cloud empowers hundreds of thousands of smaller organizations who don’t have any access to big data analytics, don’t have expertise or budgets to create an on-premise data center.”
— Peter Levine
Most companies aren’t prepared to succeed in this new age of data opportunity. There is a steady increasing value for an organization to gather it’s data into a flexible real-time firehose.
We’re building Astronomer to help companies easily and securely collaborate on data initiatives, across organizational boundaries.
Ready to build your data workflows with Airflow?
Astronomer is the data engineering platform built by developers for developers. Send data anywhere with automated Apache Airflow workflows, built in minutes...