When good data analyses fail to deliver the results you expect
By Brittany Davis, Data at Narrator.ai.
Any good data analyst will spend their days asking, “What action (X) impacts my bottom line (Y)?” We discover the data and rigorously take a look at every speculation. When we lastly uncover that X influences Y, we run to our crew and persuade them to concentrate on X.
“Get more customers to join our loyalty program to improve LTV by 30%!” we are saying.
Finding this perception and convincing the crew to act upon it’s a feat in itself — so we go residence feeling pleased with a tough day’s work. Maybe we even arrange a pleasant dashboard to assist them monitor their progress in opposition to this new goal. Each week they’ll monitor the % of consumers in the loyalty program and devise new ways to enhance that quantity. So data-driven!
This is the dream. We discovered an perception, and we made a significant impression… didn’t we? LTV stays comparatively unchanged. How is that potential? What occurred?
The evaluation was strong. Our KPI was a good metric to monitor, however that is the place issues fell aside. We tracked our progress in opposition to the KPI however failed to monitor all the assumptions that led us to concentrate on that KPI in the first place. The downside was that we didn’t regulate the assumptions. There are a couple of methods to resolve this downside, which we’ll discuss in a while.
Time to First Response: A Cautionary Tale
In my final function, I spent plenty of time analyzing help tickets. Our help crew was laser-focused on “time to first response.” It was so ingrained in the operational mannequin that we even had leaderboards to have a good time the reps with the quickest response instances.
I’m not kidding — some folks would take their laptops into the rest room so they may reply immediately if a request got here in.
This purpose was born out of nice intentions. We checked out the data and noticed that tickets with shorter response instances had greater satisfaction scores.
So… “Answer tickets faster to improve customer satisfaction rates,” we stated! And they did.
But satisfaction charges didn’t enhance — they really declined.
“Time to first response” used to be a good indicator for buyer satisfaction, however as soon as it grew to become a goal, the relationship between “time to respond” and “customer satisfaction” fell aside.
This occurs all the time! The common guide Tyranny of Metrics is filled with tales identical to mine.
We do an evaluation to perceive how a buyer behaves. A metric turns into a KPI, it goes right into a dashboard, and we fully neglect how we bought there. We placed on our blinders, and we concentrate on optimizing that metric at any value, forgetting all the assumptions that bought us there.
When we alter our conduct, we additionally change our buyer’s conduct, so the data powering the whole evaluation begins shifting. The assumptions that had been as soon as true at the moment are flawed. And consequently, we don’t see the outcomes we wish.
There’s really a time period for this phenomenon: Goodhart’s Law:
“When a measure becomes a target, it ceases to be a good measure.”
Meaning that when you begin to optimize for a metric, you run the threat of basically altering the worth it represents.
Sadly, which means most groups will proceed their days blissfully unaware that their KPI (which was at one level significant) has now change into a conceit metric.
Don’t fall into this entice.
What can we do about it?
Track the context that led to every determination!
This means monitoring the KPI and the context of how you got here to that conclusion.
With every evaluation, there’s a set of standards that wants to be met earlier than you can declare that the findings are dependable:
- Does X relate to a change in Y? Yes …possibly you do a speculation take a look at to verify for statistical significance.
- Has that relationship been constant over time? Yes.
Ok nice! It’s affordable to assume that if we alter X, then it’ll lead to a change in Y.
But if any of the solutions to these questions change, then you want to replace your suggestion.
How can we do that?
Create stay analyses (not dashboards) to seize the context
As data people, our go-to possibility for monitoring data over time is a dashboard. Create the related plots and put them in a dashboard. Set it and neglect it.
To seize context, we want to create greater than only a assortment of dashboards. No one is aware of what to do with that.
What occurs when 6 months from now, you’re centered on a very totally different venture, the underlying assumptions have modified, however your crew has forgotten how every plot influences the suggestion?
We love dashboards as a result of they’re stay.
But we want analyses as a result of the plots are already interpreted, and the takeaways are actionable.
So what if we may have the advantages of each? We would have stay analyses.
Live analyses protect the context of our decision-making standards whereas holding the insights recent and well timed, so our crew can reliably and confidently attain the similar conclusions as if we had been there deciphering the plots for them.
Time to First Response (as a stay evaluation)
A stay evaluation would have stored us from falling into the entice of optimizing for a conceit metric like Time to First Response. As quickly as the underlying assumptions modified, we might have the context alongside every plot to know precisely what occurred and why we shouldn’t concentrate on time to first response anymore.
The Customer Support evaluation would inform us:
- How did we arrive at “Time to First Response” as our KPI?
- What assumptions did we verify?
- Are these assumptions nonetheless true?
And when these assumptions failed, the suggestion would have up to date to “Don’t focus on decreasing first response times, as it is not impactful because XYZ is no longer true.”
We’d have the ability to rapidly see this modification and reply rapidly! No extra “secret” self-importance metrics.
At Narrator, we use “Narratives” to share stay analyses in a story-like format.
At Narrator, each evaluation we create is a stay evaluation. Each one captures our suggestions and the steps we took to get there. And we embrace context alongside every plot of its interpretation and why it issues. We’ve even found out a method to situation the interpretations primarily based on the data from the plot utilizing variables and if-statements below the hood.
So if the data adjustments, the suggestions will replace, making it not possible for a metric to change into a conceit metric with out us realizing it.
Since we adopted this strategy of making live-analyses in a story-like format, we’ve by no means been blindsided by secret shifts in data. We’ve created analyses that deliver worth yr after yr and can inform us a brand new story as our clients’ behaviors change.
Teams which can be severe about making impactful choices with data want stay insights to guarantee they’re all the time optimizing for the proper factor. Don’t repeat the similar errors I did.
Monitor the data, present the context, and validate every assumption over time to guarantee your crew is all the time centered on one thing impactful.
Original. Reposted with permission.