Do “Quick View” previews of products within listings help or hurt ecommerce sales? We ran an experiment to find out and the answer may surprise you.
Watch this episode of The Cro Show, a game show for conversion rate optimization and marketing experimentation fans, and see if you can guess which test variation performed better during a recent Cro Metrics client experiment.
Subscribe to the Cro Metrics newsletter to get future episodes sent to your inbox.
Full Transcript:
Tom Sharkey: All right.
So we have a client who is a very popular consumer packaged goods, desserts company, e-commerce.
And we had a test where we were trying to answer the question, what if we removed quick view from our collections pages and instead drove folks to PDP’s?
So this was a bit of a bigger swing of a test, as you can imagine – taking the Quick View experience out of collections pages.
Our hypothesis was we would see more folks flowing in through the PDP’s and having more folks stop at PDP’s in their shopping journey may give them more information that will make them more likely to convert.
So this is what we were testing.
Got a break out here of our control.
Everyone here should be pretty familiar with a quick view shopping experience, but in case you’re not on the collections page, when you hover over a product, tile in the control is a quick view which expands kind of a side cart that gives some product information and the ability to add to cart right from that screen.
Our variant tested removing that quick view experience and if you clicked on a product tile, you then instead were taken straight to the PDP.
So any questions here on what we were testing?
And I would also love to get folks thoughts on which of these two they think performed best.
Our primary metric here was add to carts, but we were also obviously very interested in tracking the lower funnel metrics like conversion rate, purchases, and revenue per visitor.
James Buo: In the V1, how does one get back to the previous view?
Just going back in the browser versus like the little ‘x’ box?
Tom Sharkey: Yup. Okay.
Great question.
Matt Vincent: Tom, are you able to share the average number of items in a checkout or something along those lines?
Even if it’s just by gut, not an actual true average number?
Tom Sharkey: It’s just over one.
Matt Vincent: Yeah.
Tom Sharkey: Yeah. So folks are their customers are typically buying one product, although interestingly, they do have a subset of their audience that buys multiple products usually for gifting purposes.
But yeah, this test was this was for everybody.
So it was on mobile and desktop and it was on all their collections pages.
Tsering Alleyne: Sorry…What was the primary metric again, Tom?
Tom Sharkey: Add to carts.
Tsering Alleyne: Okay.
Katie Green: Yeah so in terms of where I’m leaning, I’m going to align with Tsering, thinking that you’re variation 1 mostly because the photo is larger and the headline is larger.
The button seems to be about the same size, but the photo is larger.
Leila Meidam: I want to say the control won, because if I’m here, I’m trying to compare.
Katie Green: Okay.
Leila Meidam: It seems a lot more difficult to compare, jumping back and forth to the PDP.
Tom Sharkey: Great points all around, really.
Cara Binsfield: I actually think that V1 won for add to carts, but then lost at total checkouts.
And so it was like primary metric won, but down funnel results were negative because I think more people would be enticed to click through, but they might still be like conferring or like thinking about what they want and it could end up being a distraction.
Plus not being able to close out could end up like you’re kind of messing, I don’t know, with user’s intent.
I think.
Tom Sharkey: Very interesting.
Bonnie Buchanan: Tom, what was the, was the experience the same for both of these variations after user added to cart?
Tom Sharkey: Yes, another great question.
Kevin Hough: I know somebody asked if it are there more buttons on the variation?
More like menu buttons.
Tom Sharkey: This this is a PDP, so.
Kevin Hough: Oh, right. Yeah.
Tom Sharkey: When in the control when you click click view this expands, you’re still on the collections page and it’s just kind of that side cart.
On mobile it takes up the whole screen.
Tsering Alleyne: Yeah.
Tom Sharkey: But you know to get to the PDP there’s this learn more button here and then our variation again just pretend that there’s not a button there you know you would just click the product and it would take you to the PDP of the product that.
Tsering Alleyne: I was just going to ask about mobile and it sounds like you said mobile it basically looks the same except you have an additional click to get to the product page.
Tom Sharkey: Yes. Yep.
This basically this side part is the mobile experience in the control.
Kevin Hough: I love how many questions there are about this.
Tom Sharkey: Yeah.
I mean, yeah, it’s, it’s quite a big difference in the user experience.
So it was interesting to see the results.
I think we should probably take a vote now unless there are any other pressing questions?
Kevin Hough: We should ask more questions like, come on, let’s keep going.
Okay, so everybody, let’s take a vote, please.
Thumbs up if you think variation won, thumbs down if you think the control won, and flat
if you think it was flat.
A lot – it looks mostly it’s not – okay.
All right. Pretty even, actually.
Pretty even.
I think what’s even more rare than the interesting results of this test is that Cara’s intuition was actually off and it was the reverse effect.
So.
Drew Seman: I almost said I think it’s the opposite of what Cara said for what it’s worth.
Tom Sharkey: But you didn’t – sure you did, Drew.
Cara Binsfield: Yeah. Thanks, Drew.
Tom Sharkey: But, yeah that was the case.
It was really interesting results here where we saw add to carts actually take a big hit in our variation, but the lower funnel metrics weren’t really impacted, which is super interesting.
It does make sense intuitively if you think about, okay, you know, in the control where we have the ability to – Layla to your point, have the browsing still be part of that experience and add the car is super easy right there.
It makes more sense why we saw more add to carts in the control.
And then for the variation, we’re adding another step for folks and making it harder or there’s some friction before they can add to cart.
There, they’re being taken to a whole new page.
But the fact that the lower funnel metrics were flat tells us those folks who did go to the PDP in the variation, we’re actually converting at higher rates than the folks in the control, which is super interesting.
So yes, there was some drop off in terms of we added friction for adding to cart.
But for the folks who did make it to the PDPs and did add to cart, they were continuing through the rest of the checkout process at higher rates, less leaky check out there.
And this led us to an iteration test which we are hopefully going to be launching soon.
And the hypothesis here is, okay, we don’t want a 6% hit to our add to cart.
So let’s then think about testing, pulling in some of that additional information from the PDP onto the quick view side cart screen.
The hypothesis here being those folks were leaking less from the funnel in the variant because they did have that stop point at the PDP where they were getting more information, we were giving them assurance about scheduling delivery 30 days in advance etc.
So we’re going to test adding that information to the side part and we’ll keep you all posted on the results that we see here.
Unknown: Thanks for watching and be sure to subscribe to have more test ideas that right to your inbox.