DataOps: The Answer to Paying Down Organizational Data Debt


In this particular visitor characteristic, Petr Travkin, Solution Architect within the knowledge and analytics follow at EPAM Systems, introduces the idea of Data Debt which might be measured as the price related to mismanaging knowledge and the sum of money required to repair the information drawback. EPAM Systems is a number one international product growth, digital platform engineering, and prime digital and product design company. Petr has multi-year cross-industry area experience in serving to firms develop and implement enterprise knowledge methods and architectures. He is a DataOps method evangelist and adept at knowledge governance, at all times keen to join and alternate concepts about constructing data-driven processes and organizations.

Many knowledge engineering and analytics groups are too busy to cease and take into consideration how and why they work the way in which they do. Organizations are thrilled to apply superior analytics to enterprise areas however are very hardly ever as enthusiastic in utilizing it to enhance workflows. As a consequence, there’s waste in knowledge engineering, data science and analytics efforts, leading to what’s generally known as “Data Debt.”

Data Debt is the price related to mismanaging knowledge, and the sum of money required to repair the information drawback. Companies can have a look at all their present knowledge challenges and supply a tough estimate of the price it might take to repair them. Data Debt can be utilized as a powerful argument when discussing the significance of revamping outdated processes and insurance policies. Until the debt is paid, a company will at all times pay extra to keep its knowledge panorama than progressively investing in lowering Data Debt. Implementing knowledge governance and DataOps ideas is an efficient method to pay down Data Debt and keep away from accumulating it sooner or later. 

Since Data Debt can take varied varieties, let’s break down among the most typical pitfalls that organizations fall into and the way DataOps might help tackle the challenges firms is likely to be dealing with:

Excessive, Wasteful Processes

Duplicating knowledge, storing knowledge in a number of areas throughout the group, failing to reproduce work due to an absence of configuration administration or utilizing a fancy algorithm as an alternative of an easier possibility all lead to wasted effort and time. Companies could function with inefficient, out of date workflows with out realizing the long-term penalties. Avoiding knowledge silos, usually reviewing processes to adapt to change, and orchestrating knowledge, schemas and instruments can tackle these challenges.

Wait, Wait, Wait Some More

Waiting for entry to methods or knowledge, or delays in folks being assigned initiatives accordingly can all lead to delays and waste. A foundational facet of analytics perception effectivity is to keep away from the repetition of earlier work. Analytics pipelines needs to be constructed with the aptitude to robotically detect abnormalities and points in code, configuration and knowledge, with steady suggestions to knowledge groups to keep away from errors sooner or later.

What is the Actual Problem?

Establishing an accurate drawback definition is surprisingly laborious in analytics, particularly in data science. Solving the improper drawback, normally due to poor communication and misaligned expectations, is flawed work. Data and code can have bugs, main to wasted effort find and fixing issues. Writing good code on prime of unhealthy knowledge is merely a case of rubbish in, rubbish out. Software engineering, lean and DataOps practices might help by uncovering all defects within the knowledge as quickly as doable. Implement mistake-proofing assessments so poor high quality knowledge doesn’t enter knowledge pipelines. Stop and repair the issue, after which add a brand new take a look at so the error can’t trigger the issue once more.

Work that Never Makes it to Production

Work that’s solely partially performed hinders a company’s means to make choices and ship an efficient buyer expertise. By failing to think about interpretability or explaining options clearly to stakeholders, implementation is delayed or cancelled unnecessarily. The greatest waste is figure that by no means makes it to manufacturing. To mitigate this, collect suggestions as ceaselessly as you’ll be able to. Share data, simplify communication and supply suggestions at each stage of the information analytics lifecycle. Whenever a neater answer presents itself, it’s seemingly a superior one.

Multitasking & Loss of Knowledge due to Handoffs

All data-related disciplines are refined and require deep focus to remedy issues. Multitasking usually doesn’t work – it imposes excessive switching prices, wastes time and hinders the early completion of labor. Additionally, handing off work between staff can lead to data that’s left behind. To tackle these challenges, chorus from assigning knowledge crew members to a number of, ongoing initiatives until they’re leads or architects who emphasize coordination. Try to keep the identical crew composition till the top of supply.

Wasting Expensive Data Talent

People are sometimes the costliest and useful useful resource within the data-related course of, so utilizing their expertise successfully needs to be a precedence. Good knowledge analysts, data scientists and knowledge engineers are laborious to discover and costly to rent, and their abilities are sometimes wasted. To comprehensively leverage their abilities, preserve key knowledge crew members knowledgeable of the whole lot within the enterprise IT panorama, then they’ll anticipate dangers and expertise adjustments associated to new knowledge flows. Implement knowledge administration practices to enhance high quality and availability, so time isn’t wasted finding and cleaning knowledge.

Missed Opportunities

Often, companies concentrate on what actions will occur, however little emphasis is positioned on what actions are unable to occur due to an absence of correct knowledge processes and insurance policies. Recap current points with knowledge, reporting, poor content material administration, compliance points or the excessive value of possession by way of factual losses, in addition to missed future alternatives.

Poor Data Management

Companies lose some huge cash due to low high quality knowledge; this normally serves as the first manifestation and metric of a purposeful knowledge governance program. Exceptionally delicate knowledge can create threat for companies due to misuse, poor high quality or compliance points, which may lead to lofty fines. Storage prices for manufacturing environments, in addition to growth, testing and person sandboxes, can quickly enhance when knowledge just isn’t archived or deleted shortly. Businesses want to measure present prices and dangers related to poor knowledge high quality to guarantee these prices aren’t contributing to their Data Debt.


To measure your Data Debt, think about the price of advantages acquired or misplaced. Here are some areas that can show you how to begin to assess the quantity of collected Data Debt:

  • Processes: enhance cycle time, decrease value or enhance high quality
  • Competitive Advantage: acquire aggressive intelligence and create differentiators
  • Product Development: determine a brand new product or characteristic
  • Intellectual Capital: embed data into services and products
  • Human Resources: allow staff to do higher work
  • Risk Management: cut back varied varieties of threat (monetary, knowledge or authorized compliance)

If a enterprise makes choices with out contemplating the influence on their knowledge, future prices will happen when coping with inconsistency, errors and redundancy. As organizations turn out to be extra data-driven, it’s essential to measure the worth of knowledge by introducing the idea of Data Debt and implementing DataOps ideas to assist repay it. Estimating your Data Debt can reveal the prices related to ineffective processes and spotlight the worth of an information governance program.

Sign up for the free insideBIGDATA publication.


Source hyperlink

Write a comment