Natalie Brown: So for this test, it was for a healthy dog food subscription and delivery service.
For some background and context on this business, they have a decently long signup subscription flow where it asks users to answer a bunch of questions about their dog and then ultimately recommends a personalized meal plan for them to check out with.
But the problem is that there’s too many steps in the flow.
It’s preventing users from continuing on and ultimately making it to the checkout.
So our hypothesis for this test was to remove a step that was unnecessary early on in the flow and in doing so, would hope that users are able to make it through the flow more expeditedly and checkout.
So for the variations, the stuff in question that we identified as being unnecessary, it was dubbed the Emily Step.
And in control, when users enter into the flow anywhere through the site, the first page they’re taken to is this Emily Step.
Now, Emily’s job is just to greet users, let them know she’s a pup specialist, and then she’s going to guide you through the rest of the questionnaire – and also provides a few value props.
Once users click on Let’s Get Started in control, they’re taken to the first question in the questionnaire.
And then in V1, we banished Emily from the flow.
So when users enter in anywhere on the site, they’re automatically directed right to this first question.
Our primary metric was order completions.
And then additionally, we tracked page view metrics throughout the flow to see how far users were making it.
Katie Green: Sorry, and did you say if this is on mobile and desktop, and if so, what the split is in traffic?
Natalie Brown: Yes, this is both on mobile and desktop.
I’m not quite certain about the split, but I do know that the majority of the users are desktop users.
Katie Green: Got it.
Philippa Boyes: Does Emily come back around again like through email as like the brand ambassador person?
Natalie Brown: I don’t believe so.
I don’t believe they have any like email remarketing.
Philippa Boyes: She’s just used this once?
Natalie Brown: Yes.
Except for you can see kind of in these screenshots throughout the flow, there’s like little icons with Emily pictured and then they’ll be like a little helpful blurb kind of explaining what the input is asking for.
So she does appear in V1 throughout the flow just not greeting users into the flow.
Zeinoun: And is Emily used in any of like their advertising or commercials or anything – like is Emily a persona of sorts?
Natalie Brown: Not that I know of.
To the best of my knowledge, Emily is just a stock photo model.
Jamie Brown: So I’ll go ahead and give my guess – so I personally really hate these stock photo Emily’s, and I think they’re really fake and disingenuous.
So I’m saying goodbye to Emily.
Katie Green: Is there another place where those value props underneath the start button appear?
Jamie Brown: Good question.
Natalie Brown: Yes, good question.
These value props are actually at the checkout screen.
Jamie Brown: Lit.
Katie Green: So then in that case, bye Emily, we’re reducing a click barrier.
You end up getting those value props when you’re – when you’re better validating yourself has a stronger lead.
So, yeah, V1 is my vote.
Cara Binsfield: I’m going to go like counter culture, I guess, because I think that people like the feel of talking to a specific person.
And I don’t think like everyone, if you don’t work in Web services, realizes how fake these are.
So I’m going to go ahead and say, like, I don’t think Emily lost.
I also might have tried like a couple different types of Emily’s and said like, okay, does Emily work better when she’s a Frank and he’s, you know, smiling?
Does Emily work better when she’s like, you know, a different kind of person or a different age group based on our user profile?
That kind of thing, so I might have done some like Emily tests.
Pro-Emily over here.
Katie Green: If Emily wins, pro-Emily iterations.
Emily Martin: I was going to say something similar.
It’s like having it over to the side where it was more of a conversation.
James Buo: And I’m going to go against what Cara said.
I think that when I see this, I feel like I’m going to be set up to have a very lengthy conversation because I’m being introduced to someone who’s guiding me through this process and I might abandon before I even get to the next step.
So I think cutting that out is going to help.
Katie Green: Okay, let’s move to voting because we have 3 minutes left.
So great content, everybody.
Thumbs up if you think V1 won, thumbs down, if you’re pro Emily.
Natalie Brown: Okay.
Without further ado.
For this test overall, it was generally pretty flat with low statistical significance, a slight increase in orders, but also slight decrease in users getting throughout the flow.
However, when we segmented between desktop and mobile, we saw that this test is actually a pretty big winner for desktop users.
So 16% increase in orders with 91% stat sig.
And additionally, we saw for desktop users they were making it further into the flow.
So the takeaway from this test is that desktop users enjoy a more expedited experience.
They don’t really care for Emily, but mobile users prefer having that greeting into the flow.
So Emily still lives on for mobile users.
Amazing.
Katie Green: Thank you for watching and be sure to subscribe to have more ideas sent directly to your inbox.