Steve Meyer: All right.
So this test is for a client that is an online retailer that sells apparel.
And this test was pretty simple overall.
So the hypothesis and the problem we’re trying to solve with this hypothesis is that customers do not know how to log in.
Or maybe they just really don’t have that little nudge to actually log in.
So you’ll see in the control, the only indication that you can log in is this little teeny little icon of a person.
And this test, we simply change that to “log in”.
So.
Katie Green: I assume this was cross device?
Steve Meyer: Yep. All devices.
And, you know, to support the hypothesis, we looked in – and looked at the data and we saw that only 2% of users actually log in before checkout.
So we thought, hey, this is a great opportunity to increase that number.
And then also rationale for the test that, you know, logging in can help you rebuy products you – you need to restock on.
It helps streamline checkout because, you know, it fills in all your address and name and all that stuff.
It also can help you continue like a previous shopping journey from like mobile to desktop from a different device.
So a lot of great reasons why this, why it’s important to get people to log in.
And that’s about it.
It’s a really simple concept.
Any other questions?
Carl Light: Steve, what was the primary metric for this one?
Steve Meyer: Orders.
So overall orders.
And just to also add, the audience we were targeting here is “logged out” users too.
Carl Light: If a user logged in or exposed to the test and logged in on one device and then continued their journey on another device, but we were able to attribute it because of the because of the log in.
Was that carried through in the test as well?
Steve Meyer: No.
We were unable to do cross device tracking for this experiment.
Matt Vincent: Hey Steve, I have a learning question for you…So I – around the primary metric being orders, was there a particular reason why you all chose that as opposed to just actual successful logins as the primary metric?
Steve Meyer: Yeah.
I mean, pretty much the entire team and myself thought that, you know, we’re adding a clear CTA, we’re obviously going to see an increase.
You know, so that’s a given.
So we wanted to look further down funnel and we knew we probably had the traffic to do that.
Matt Vincent: Ok, cool.
Steve Meyer: All right.
So is it time we vote?
It is time we vote.
Katie Green: Thank you, Steve, for stopping so we make sure to do that. OK.
If you think variation one was the winner, do thumbs up.
And if you think the control remains on top, bam, thumbs down.
A couple flats, throwin’ in a curve ball there.
Okay. A lot of thumbs up, though, Steve.
Steve Meyer: All right.
And TADA!
This was a big win.
We actually saw a 4.1% overall lift on completed orders with full significance.
RPV was lifted 5.3%.
We saw account logins go up by over 50%.
And the main takeaway was that it really helped streamline the checkout process.
So we saw checkout steps go up by 3% to almost 5% through each step.
And also, the impact was much stronger on desktop.
This was actually the biggest win we’ve had with this client in two years if you can believe that, by changing one single icon to a word.
Katie Green: Thanks for watching and be sure to subscribe to have more test ideas sent directly to your inbox.