4 Real-Time Data Analytics Predictions for 2021
Data management, analytics, data science, and real-time systems will converge this year enabling new automated and self-learning solutions for real-time business operations.
The global pandemic of 2020 has upended social behaviors and business operations. Working from home is the new normal for many, and technology has accelerated and opened new lines of business. Retail and travel have been hit hard, and tech-savvy companies are reinventing e-commerce and in-store channels to survive and thrive. In biotech, pharma, and healthcare, analytics command centers have become the center of operations, much like network operation centers in transport and logistics during pre-COVID times.
While data management and analytics have been critical to strategy and growth over the last decade, COVID-19 has propelled these functions into the center of business operations. Data science and analytics have become a focal point for business leaders to make critical decisions like how to adapt business in this new order of supply and demand and forecast what lies ahead.
In the next year, I anticipate a convergence of data, analytics, integration, and DevOps to create an environment for rapid development of AI-infused applications to address business challenges and opportunities. We will see a proliferation of API-led microservices developer environments for real-time data integration, and the emergence of data hubs as a bridge between at-rest and in-motion data assets, and event-enabled analytics with deeper collaboration between data scientists, DevOps, and ModelOps developers. From this, an ML engineer persona will emerge.
1) Hyperconverged data management and analytics
Visual analytics, data science, data
management, and AI-enabled business processes are colliding as the quest for business optimization accelerates.
This is being accelerated by the global pandemic, forcing many enterprises into
a swift digital transformation. AI engines and software suggestion systems are
enabling this convergence – bringing automated and low code data prep and
machine learning into BI, analytics, data science, business process, and
app-dev working environments. Automating routine tasks frees up time for
innovation and business optimization, allowing humans and AI-automation to work
convergence of data management, analytics, data science, and DevOps will
accelerate in 2021 and beyond, incorporating elements from cloud vendor services along
with open-source and homegrown tooling. This will make it easier to implement end-to-end
workflows and develop event-enabled business solutions in hybrid cloud
environments. This hyperconvergence will also drive a need for broader data
literacy, which is becoming critical across all business functions. The most
efficient businesses in 2021 will be those that work to improve data literacy
across functional areas and organization levels while embracing the convergence
of data management, analytics, DevOps, and business process technologies in
event-enabled environments. The result will be rapid and ongoing development of
AI-infused apps, along with deployment and management of data science
technologies in business operations.
2) Creative combinations of in-motion and at-rest data
All data begins as business events that arrive at many velocities.
Event-enabled companies integrate data on event streams, monitor trends, make
decisions, and take actions on real-time data-in-motion. Modern businesses also
accumulate data from event streams into at-rest data systems, where they train
predictive models and create visual analytics applications for business
strategy, tactics, and operations. Users consume these analyses at many
Going forward, we will see a combination of data-in-motion and
data-at-rest feeding closed-loop, self-learning systems in operations.
Specifically, I see a downtick in solutions that involve moving data and an
uptick in solutions where data are analyzed directly on the event stream,
within delegated cheap storage systems, and in fluid combinations of the two.
According to IDC analyst Dan Vesset, “exciting use cases are in analyzing data from moving and
stationary ‘things.’ We’re still in the early days of developing intelligence
about the physical world.”
3) The emergence of data hubs as a bridge between hybrid data assets
multi-cloud, AI-infused apps continue to drive value on the front lines,
rolling updates from multiple disparate sources are fueling the underlying
data. As a result, I see advances in data hub technologies for bridging cloud
and on-prem data assets, transforming individual sources of data via cloud
integration services like AWS Lambda and TIBCO Cloud Integration, and injecting
real-time data to drive situational awareness. Master data management and data virtualization are key
technologies for managing and bridging data assets with secure access across
hybrid cloud environments for at-rest and in-motion data sources.
Ulta Beauty is providing real-time
personalized experience, accelerated customer service, loyalty, and order
updates with immediate data updates across all channels. Use cases like BOPIS
(buy online pick up in-store) and curbside pickup are enabled by a responsive
app mesh architecture where inner APIs abstract legacy data. Outer APIs surface
shipping, customer, and inventory APIs for real-time customer experience.
Early on in the pandemic, Norton
Healthcare mobilized an analytics command center for their medical records
system Epic and operational systems for hospitalizations (including PUIs),
testing, capacity, utilization, ventilators, and all healthcare services and
operations. Norton pulls data in real time to refresh the apps, giving them a
wealth of up-to-date information to monitor hospital operations and the
pandemic effects. Additionally, Norton opened new telehealth lines of business
while recasting equipment supply chains to protect health workers on the front
As the business economy reopens and
social behavior moves to new states of normal, we will see new ad-hoc data
combinations come to fruition. For example, we will see real-time analytics on
health status, building access and occupancy, crowd density, and network
evolution to meet new business operations processes.
In all of these applications, master data
management and advanced data virtualization will be key underlying
technologies, combining at-rest and real-time data sources to continuously
update critical analyses with governed access and security. I predict a
confluence of data virtualization, data catalogs, and master data management
systems to keep data and AI applications in order and to scale the merge of
real-time and at-rest systems. In combination with the API-led microservice
mesh, this will accelerate the rise of new high-value business applications in
the COVID and post-COVID era.
4) The convergence of data science with DevOps, ModelOps
from home has accelerated collisions and collaborations between teams of data
scientists, DevOps, and ModelOps developers to manage data science apps in
This convergence will accelerate as
ModelOps continues to grow as a discipline. As a direct result, a new machine
learning engineer persona is starting to emerge. In their role, they configure
deployment scenarios in hybrid cloud environments, working with data scientists,
data engineers, business users, DevOps, ModelOps, application development, and
design teams to manage deployments, monitor and update data science workflows
in production environments.
Deploying and managing models in
production environments is a non-trivial exercise. To quote British
statistician George Box, “all models are wrong, some are useful.” Models bring
risk, and that risk needs to be managed. Unintended consequences can be
significant. For example, if a model calls credit risk incorrectly, real money
and reputation risk is at stake. Model robustness and deployments that track
bias, provenance, and usage, along with rules-based systems that provide guard
rails around interpretation and action, is critical. Model explainability is
also important, as is adherence to the ever-accelerating set of regulatory
guidelines from CCAR, CECL to CCPA, and emerging EU regulations on trustworthy
ModelOps includes many components,
including data preparation, data versions, quality checks, feature engineering,
labeling, to name a few. We will see considerable evolution in this area as the
data science to DevOps handshake adaptively radiates across business functions
and applications. From regulated environments in financial services and
healthcare to the more fluid worlds of tech and martech.
The future of real-time analytics
data management and analytics will be key to surviving and thriving in 2021. We will see a convergence
of data management, analytics, data science, and real-time systems enabling new
automated and self-learning solutions for real-time business operations. We’ll
see data hubs emerge as a bridge between in-motion and at-rest data. We’ll see
data scientists, app developers, DevOps, and ModelOps worlds collide and
converge. Though the near-term future state of business and travel remains
uncertain, we can confidently predict the ongoing central importance of data
management and analytics as input to real-time connected experience systems
while continuing to uplift
in data literacy across all business functions and levels.
Read More …