How much does a subject line matter when filling out advocacy contact forms? One small change couldn’t lead to a ~4x improvement, could it? The answer might surprise you.
Watch this episode of The Cro Show, a game show for conversion rate optimization and marketing experimentation fans, and see if you can guess which test variation performed better during a recent Cro Metrics client experiment.
Subscribe to the Cro Metrics newsletter to get future episodes sent to your inbox.
Full Transcript:
MJ Castells: Alright, so this is a environmental nonprofit client that we have.
We work with a couple of different teams with them, but this one is specifically on their advocacy team, which is basically in charge of increasing actions and actions taken for specific issues that are going on around the country.
The actions can be regional based or they can be national. So it depends on where the person lives and what’s going on in that particular state.
The test that we were running here is specifically on these action forms that users fill out, so usually the action form has some kind of write up on the left hand side here and then the actual form on the right hand side – it’s a two step form. The first step they fill out their name
and they put their zip code to make sure it matches the action that they’re taking, in terms of if it’s regional.
And then the second step is this, and this is where we’re focusing our test on.
So the issue that we realized that was going on, or rather the team really realized was going on, was that this subject line right here that you see is actually editable.
But folks don’t really know that it’s editable, right? Like it doesn’t look very editable. It doesn’t look like I can do anything.
So what was happening was these actions get sent over to lawmakers and all the lawmakers were receiving the same subject line.
So it kind of looks like. There was really nothing new that was being said. Folks did have access to add a personal message down here, so that was a way for them to kind of put their their personal take on it.
But in terms of what was being received by the lawmakers, they were not really like it wasn’t anything new, right? It was the same like homogeneous kind of subject line.
So in our test we decided to see like what would be the best way to show that this is editable and encourage folks to make that editable.
So we ran… we ran a V1, where we had these radio buttons right here.
So what we had was a default subject line, it defaulted to that as soon as the user hit this section of the form and then the user has the option to customize your subject line and then this would become editable and they could they could change the subject line or if they decided to change it and then they wanted to go back to actually click on default subject line again and it would it would populate the same subject line again.
And then at that point they could change it whatever they wanted to do, add their personal message, add their street address, all that stuff, and then hit the submit button down at the bottom, down here, and then it goes off to the lawmakers.
But yeah, what do folks do folks have any opinions or any thoughts?
Cristi Alvarez: Do you happen to see any – have you guys ever done any like heat map data to see how much people interact with that or is there any data around like how much people interact with the subject line within the message?
MJ Castells: We didn’t run any kind of heat mapping kind of data on this and I don’t know if we have on any advocacy forms. I’m not sure if this particular client has Hotjar synced up to them, but I think the big giveaway in terms of like what was the interaction of the subject line was essentially coming from from the top of the organization where they just realized that the subject line was literally the same. Folks were just not touching the subject line.
Cristi Alvarez: Right.
MJ Castells: That was kind of like a big giveaway.
Cara Binsfield: What is the value tradeoff here? Like are we just looking for more submissions, or are we looking for higher quality submissions and therefore we want more people engaging with the subject line over the submissions themselves?
MJ Castells: Yeah, our primary metric was the submissions themselves, but also – but some of our secondary metrics included clicking on the subject line, right? Like interaction with the subject line. So yeah, but our primary metric was basically are folks – does it hurt to do this, and if not, does it help to do it, right? So primary metric was to submit the actual action form.
Cristi Alvarez: I will say I think – just for a guess here – I’m going to go with Control winning just because I think adding an element to the form might distract people from potentially moving to the next steps. That’s just my hypothesis there.
MJ Castells: Is it – what do we do, Cristi? Thumbs up, thumbs down?
Cristi Alvarez: Yeah, so for VO, if you think the V0, you’ll go thumbs down if you think it was V1 you’ll do thumbs up, and think it was flat, go sideways.
MJ Castells: Alright, so this test actually was a huge winner and it reached statistical significance, which is actually pretty rare for advocacy forms just because it doesn’t get a lot of traffic. But we increased the CTA click, which is that last button on the form by almost 3% at full stat SIG. And then a whopping almost 400% on enters subject line clicks at 40% stat SIG. So as you can see here, there are folks that were in V0 still entering the subject line, but 62 to 299, which was a huge, huge win. What we did notice which was interesting was that there was a decrease in the number of folks signing the petition with a personal message. So that was the personal message area down here. So what we were noticing is that essentially folks are utilizing the subject line to kind of give their message to lawmakers right away, which is probably a lot more impactful than putting it in the body of the message. So it was a huge win all around. We’re soft coding this and the client will hardcode it eventually, but pretty – pretty successful in this one.