The State of Experimentation Traps - 2023
We’ve been running Experiment Pitfall Assessments in 2023 with organizations all around the world and would like to share some of our findings with you.
The goal here is to better understand where companies are struggling when it comes to repeatedly testing new product, service and business ideas.
Let’s dig in.
Outsourced Testing is Going Out of Style
Several bespoke agencies popped up in the early 2010’s focused on:
MVP Development
Clickable Prototypes
Customer Discovery
Fast forward 10 years later, most of those agencies in the U.S. have either been acquired by big corporations or simply shut down.
The trend continues to shift to more of an in-house experimentation and training capability.
The assessments we conducted in 2023 tend to reinforce that notion.
Source: 2023 Experiment Pitfall Assessment - Powered by Yerbo
As you can see in the illustration above, the majority of the employers we assessed did not outsource their early stage idea testing.
This also illustrates the continued shift in consulting to a more “facilitative” approach, where experts are brought in to teach an idea testing process, not provide all of the test results.
Very Few Companies Are “Really Good” or “Really Bad” at Experimentation
Out of the 100+ companies we assessed, only 17% - 19% scored very low or very high on the Experiment Pitfalls scale.
Most companies are somewhere in the middle.
Source: 2023 Experiment Pitfall Assessment - Powered by Yerbo
While this doesn’t surprise us, it does amplify the need for better in-house training and coaching.
Standard frameworks only work so well, and we’re witnessing companies taking what they’ve gathered from Testing Business Ideas and customizing it to meet their unique needs.
For example, a B2B Hardware Corporation isn’t going to have the same testing process as a B2C Software Startup.
Books and YouTube videos will only get you so far in that journey without a trusted partner.
Our Need to Be Right is Driving Dysfunction
When we look across all of our results and see how different traps relate, we begin to see some patterns, specifically with confirmation bias.
We phrase confirmation bias in our assessment as favoring results that confirm our existing beliefs or hypotheses.
Source: 2023 Experiment Pitfall Assessment - Powered by Yerbo
Employers who scored poorly on confirmation bias also struggled with:
A failure to learn and adapt
Incomparable data/evidence
Weak or light data/evidence
From our experience, this isn’t just a correlation.
There are many ways to shape up and refine your assumptions into well formed hypotheses before testing to mitigate the data/evidence issue.
In Testing Business Ideas, we use the method of refining the top right of the Assumptions Map (most important, least amount of evidence) using the model of testable, precise and discrete.
We also recommend choosing an experiment from the library which matches the stage of your testing journey with the theme of risk (desirability, viability or feasibility).
In the experiment design itself, we use a Test Card to define the hypothesis, test, measurements and acceptance criteria.
Now there are certainly scenarios where you may have unclear or new insights from the experiment, which usually lead to a re-testing of your riskiest hypotheses or a pivot in a new direction.
However, none of this matters if you simply refuse to believe the test results.
This illustrates the continued need for developing a testing mindset within your organization.
Looking at 2024
While we expect these 2023 testing trends to continue through early 2024, there is a continued optimism that we’re moving in the right direction.
We predict that more and more companies will move from fine tuning their existing testing processes to building a larger culture of experimentation across the organization.
It’s a multi-year journey that doesn’t happen overnight, and you aren’t in this alone.
