What’s better for driving new sign-ups: a 14-day free trial or 30-day free trial? The answer might surprise you.
Watch this episode of The Cro Show, a game show for conversion rate optimization and marketing experimentation fans, and see if you can guess which test variation performed better during a recent Cro Metrics client experiment.
Subscribe to the Cro Metrics newsletter to get future episodes sent to your inbox.
Full Transcript:
Emily Schreiber: Awesome, so this client, they – we essentially work with a food delivery service company, and this test specifically focuses around increasing sign ups for restaurant owners so that they will list their business on this food delivery service app.
So our test here was trying to understand, you know, can we increase urgency by limiting the duration of a free trial?
So for a period of 30 days, which is their standard free trial, there are zero commissions you have to pay. And then after that 30 days, you pay the standard commission rate.
We wanted to understand the impact of having a 14 day trial instead. There’s a little bit of value in that, you know, they – this company can start collecting commission a little bit earlier.
But we also just wanted to understand does this drive urgency or does it completely turn people away from it as a ‘too short’ of a trial?
So we tested it throughout their signup flow.
So from the landing page all the way through each step in terms of building this test, it was a little bit complex because we had to kind of hunt
down everywhere that it said 30 days and make sure that that changed appropriately.
So if you’re thinking about doing this with a client, make sure that you get the full scope of this test because it seems very simple, just changing a couple of numbers, but it can get tricky when you have to make that change in many different places on a site.
So any questions about this test?
Cristi Alvarez: Emily, this one ran on both devices or just desktop?
Emily Schreiber: Yes, I believe it ran on both devices.
Dave Albert: So your hypothesis was by shortening the duration, it would create a sense of urgency.
Emily Schreiber: Yep.
Dave Albert: But ultimately, it’s not as good of a deal for the merchant, right? Because they don’t get the extended period before they actually start paying.
Emily Schreiber: Yeah. Correct.
Dave Albert: And the only change was that was that the duration of days, like no messaging or anything. OK.
Emily Schreiber: Yep. So and it just kind of went down from a month to two weeks, which two weeks is still kind of a decent amount of time for, you know, a company to to try something out.
So we wanted to see. Does bringing it down to two weeks, is that still enough of an incentive to try this out?
Cristi Alvarez: Did anybody have any other questions or have any insights to which variation they believe won and why?
Bonnie Buchanan: Well, Emily, I have a question. Did your results analysis include looking at the cost savings for the company, knowing that they’d be saving on 14 days, approximately not paying out commissions?
Emily Schreiber: Yeah. So that was kind of part of it as follows. Even if this test fell flat, it would still be a win in the sense that there are a cost savings on the client side, for sure. So that was kind of the added benefit here.
Cara Binsfield: So was that sort of the break even point was like flat to a lift or like, were you willing to take a hit as well?
Emily Schreiber: I think if we were, if we were willing to take a hit, it would have to be small enough to where the tradeoff wasn’t too drastic. I will say we did get pretty conclusive results, so we didn’t actually end up having to run through that particular calculation step but if we were in that situation, we would have certainly performed that analysis.
Cristi Alvarez: I’m gonna go with variation one as the winner here. Just because it’s it has, I guess, a little bit more leniency as far as like what is considered a win in this situation. And I think 14 days is still pretty compelling. So versus 30, so I think that would be my option there.
Katie Green: I would say 30 simply because anecdotally, I feel like my brain functions more in months than a does in weeks.
Like, I can’t really understand what 14 days of free anything is to me, but I can definitely think about like, what is 14 days free rent? Right? Definitely know what a month’s free rent is, you know what I mean? Not totally anecdotal, but…
Emily Schreiber: That’s actually a really great point. We’ve even had conversations about, do we say two weeks or do we say? But in order to keep playing as much parity between the two regions, we just stuck with the two days.
Nhi Shirley: Emily, one question for you in terms of your audience, you might have mentioned this. I might have missed us, but is it a US based audience? Is it international? What does it look like?
Emily Schreiber: Great question. It was – It was everyone so international and US.
Cristi Alvarez: All right, let’s vote. So if you think the control, the 30 day option won, you’ll put a thumbs down. If you think the variation won, you’ll put a thumbs up. Go ahead and cast your vote. Oh, and if you think it was flat – You just put something in the middle.
Emily Schreiber: So overall, this test actually lost when we looked at it in Optimizely. It was like a five percent kind of drop. When we started to look at it based off of country type, we saw something completely different, told a totally different story.
And this is why it’s important when you look through your results not to just make a decision based on something overall, because there could be a little more nuance in here that actually has a lot of opportunity.
So we realize that the biggest drop was occurring in international spaces in Australia and Canada, both with Leeds and even more importantly, by the time when it came down to actually closing the deal, Significant drop.
Part of the reason our hypothesis is is that this company has a lot of brand awareness in the U.S., but internationally they’re still working to kind of grow into these markets. And so there’s a little more hesitation in international spaces.
And that 30 day trial just is a little more comfortable for that group in the U.S., however, totally opposite story. As soon as we dropped it down to 14 days, we saw a big uptick and users jumping on that two week deal.
Part of that could be that they are used to seeing 30 days and they’re coming back. And now seeing at two weeks is now, Oh, shoot like I. I need to jump on this now.
Or it could be that just 14 days does drive more urgency to start and finish the signup process, because then you want to maximize all of that time that you’re working through.
So we’re actually going to continue kind of retesting this. This company rebranded some of their site. Things are a little bit different now, and they have pricing in packages. So we’re going to retest this concept in in conjunction with some of those new changes that they’ve rolled out.
But overall, this had a pretty substantial impact on their revenue just on a monthly basis.
Cristi Alvarez: And this, I think, shows the importance of not just looking at the overall picture.
Sometimes it’s good to get granular and segment your data to really find out which audiences your test is working for.
So that’s a great test to showcase that.
Thanks for watching. Be sure to subscribe to have more test ideas sent to your inbox.