Hey! This is the second blog post in our series on the “velocity first” mindset. Check out the previous post, Why the “Velocity First” Mindset Will Drive the Most Value to Your CRO Program now!
Many conversion rate optimization (CRO) programs struggle to attain high experimental velocity out of the gate. Two issues come to our mind that mainly affect velocity: politics and process.
Sometimes it’s a matter of organizational structure where the experimentation team does not have much power to make quick decisions when compared with stakeholders. Sometimes, it’s a matter of organizational processes that simply hinder speed.
In this post, we’ll show you four simple approaches you can take to increase your experimental velocity with a core emphasis on embracing the 80/20 rule.
Core Principle: Embrace the 80/20 Rule
Also known as Pareto’s Principle, it states that approximately 80% of outcomes come from 20% of the effort. In the context of experimentation, this means test planning, strategies, concepts, and development don’t have to be perfect.
Too often, organizations struggle with “doing it the right way” for things that don’t have a right or wrong answer. The examples below illustrate a few of those areas as well as our suggestions on how you can apply the “velocity first” mindset to boost your program impact.
#1: Too Many Cooks in the Kitchen
As the old saying goes, when there are too many people working together, the resulting product is inferior to what could have been a better product with fewer stakeholders.
A common mistake we’ve seen with many CRO programs is that each test variant must go through multiple stakeholder approvals before it goes out to the public. It must go through all the teams that would be affected such as product management, design, marketing, and even executive-level stakeholders before launch.
Like too many cooks in the kitchen, with each flavor the stakeholders inject into your experiment, the more value you’ll lose. This could be in the form of adding extra variables to the experiment or increasing the time to launch.
On top of those reasons, feedback often ends up subjective and not data-backed. When you begin to jump from letting data tell you what works to allowing blocks and changes based on opinion, you are getting further away from the culture of experimentation.
#2: Don’t Seek Imaginary Perfection
Are you spending hour after hour trying to “perfect” the strategy or concept? There is an imaginary expectation that the more effort you put into developing an experiment, the higher the impact. In fact, our data proves that there is no correlation between effort and performance.
Unless the experiment itself is of bad quality (broken functions, bad user experience, changes that are not obvious to the customer, etc), effort will marginally make a difference.
Going through multiple rounds of revisions trying to find that killer headline or being afraid that the variation isn’t to someone’s taste isn’t cost-effective. When you feel it’s good enough, it’s time to move on and test to see if it’s better than the control. No one knows for sure what will work. That’s why you test.
#3: Lean Into Lean Experiments
One of the four agile values that tie in well with experimentation is the idea of a working software over comprehensive documentation. In other words, minimum viable product (MVP).
Under the agile philosophy, it should be one of the highest priorities to get your software (in the case of web experimentation, a UI, or an experience) in the hands of a customer quickly to collect feedback and iterate.
Think of experimentation as a series of MVPs to build that total customer experience from beginning to end. Focus on small and frequent deliveries like testing one or a few elements at a time over expensive big bangs like page redesigns.
#4: Painted Door Tests to Mitigate High Costs
If an actual MVP is not possible, consider a painted door test. If a feature is too expensive to be developed for experimentation purposes, a painted door is a low-cost method to gauge the interest of said feature with customers without actually developing it.
Take for example an augmented reality (AR) widget for an online furniture store. To gauge whether customers would be interested in such a feature, you build a Call-to-Action (CTA) along the lines of “See How This Item Fits Into Your Room” with the CTA anchoring customers to the product dimensions section.
The result may not live up to the customers’ expectations. The important part here is you were able to capture the necessary engagement data and customer insights to see if investing in AR would be worth the price versus spending your limited resources on something that may not work out.
What’s the Right Amount of Velocity?
There’s no particular standard for velocity. As an experimenter, the key question to answer is: How much faster can I launch my experiments? Over 90% of the time, organizations don’t lack in test ideas, they lack in speed to get experiments out the door.
Our suggestion is to think about each untested page as an opportunity cost. Each day that a page is not tested is an opportunity cost for your CRO program. To minimize opportunity costs, the experimentation team should focus their efforts on testing as quickly as possible to find the jackpots of experiments.
When Is a High Level of Effort Appropriate?
The analysis we mentioned in our first post only reviewed our experiments from a revenue impact perspective and suggests that low level of effort (LOE) experiments should be valued much higher than high LOE experiments. That’s our mantra when it comes to delivering the most impact for our clients.
However, in experimentation, it’s also important not to get too hung up on just the revenue impact. Experimentation is about learning what works and what doesn’t. Aside from experiment winners, the experiment losers can also tell the story of how they saved your organization millions of dollars from implementing a new feature.
You shouldn’t be afraid to test high LOE experiments once in a while because they can still be highly valuable to your business given the right circumstances. Think for example that new chatbot you want to incorporate or the product recommendation widget for your e-commerce site.
Organizations need to leverage experimentation to be innovative and it’s imperative to test new features before going fully live to the public.
Adopt a “Velocity First” Mindset With Cro Metrics
With experimentation, you need to find what works and what doesn’t with your customers quickly and efficiently. The key to getting highly impactful insights through experimentation is to test as many changes as possible as quickly as possible.
Each day a page or feature isn’t tested is an opportunity cost for your CRO program. You can minimize that opportunity cost by adopting the velocity first mindset. To learn more about this mindset, reach out to our team!