As a team of experimentation and optimization experts at Cro Metrics, we have run more than 18,000 tests across multiple types of industries including e-commerce, fintech, SaaS/subscription, non-profits, technology and travel.
As a result of the knowledge and data we have, clients often ask us the multi-million dollar question: What really drives an effective conversion rate optimization (CRO) program? The solution is velocity first.
Cro Metrics & “Velocity First”: Our Findings Thus Far
One of the strategic approaches that jived with me when I joined Cro Metrics from another CRO agency is the idea of velocity first. From my 8+ years of CRO experience working at enterprise companies and also consulting Fortune 500 companies, it’s something I knew in the back of my head but didn’t have the readily available data to back up.
A major reason that attracted me to join the talented team here at Cro Metrics was that they had done the analysis. Here are the incredible findings from analyzing a random sample of over 5,000 tests.
1. There’s No Correlation Between Time Spent vs. Win Rate or Impact
These first two points are pretty straightforward. Looking at the level of effort (LOE) perspective, there is no correlation that more time spent developing an experiment contributes to better performance.
Too often, organizations put forth more effort than necessary to launch experiments. There is an imaginary expectation that more stakeholders, ideas and refinements when developing an experiment, means a higher quality experiment and therefore, higher performance.
Our data tells us there is no such correlation between time spent and performance. From an ROI perspective, it’s much more beneficial to prioritize the lower LOE experiments than the higher LOE experiments.
2. The Top 5% of Experiments Contribute to over 90% of All Positive Impact
At Cro Metrics, we’ve found that the top 5% of the most impactful experiments account for more than 90% of all positive impact. This is called a fat-tailed distribution, meaning certain outlier experiments are so big, they skew the entire distribution.
These are the jackpots of experimentation. And to find them, the best approach is to test as much as possible.
The Math Behind the “Velocity First” Mindset
Let’s take a look at this mindset from another perspective. First, consider this formula:
Experiment Win Rate x Number of Experiments Launched x Average Revenue Impact per Experiment = Annual Impact of Your Experimentation Program
Now, consider these two scenarios:
Baseline: 1 Experiment per month at 30% win rate
30% x 12 experiments x 500k = $1.8 million
Scenario #1: Increase in win rate by 10%
40% x 12 experiments x 500k = $2.4 million (+$600k)
Scenario #2: Increase in velocity by 4 experiments per month
30% x 60 experiments x 500k = $9 million (+$7.2 million)
As you can see, just increasing your monthly velocity by four experiments as opposed to spending it all on getting a hypothetical (and not likely) 10% increase in win rate is worth way more. And in the above scenario, it’s actually worth $6.6 million more!
Velocity has no limits. As organizations become larger, they can increase their testing velocity by 100 or even 1000 times. Given these factors, think about the impact of your CRO program with an increase in your experiment velocity as opposed to focusing your energy on strategy.
Working with CRO experts like us is a great way to increase your return. Because our experts are already equipped with the knowledge and skills needed, we can hit the ground running and make an instant velocity impact on your ROI with our seasoned team of growth strategists, program managers, designers, analysts, and engineers.
Next Up: 4 Ways to Apply the “Velocity First” Mindset to Make an Instant Impact to Your CRO Program
In the next post, we will dive into ways you can apply this “velocity first” mindset to your CRO program. We’ll also touch on why we recommend applying the 80/20 rule to just about every aspect possible to get the most value out of your experiments.
Questions? Need testing ideas? Cro Metrics has you covered. Reach out to our team today!