Tests nobody's customers asked for
We took over a testing programme built on a playbook. So we started asking the customers instead, and ran it at nearly three times the pace.
Research-led testing, live within a weekA year of playbook testing
When we started working together there had been a year of testing already. Thirty-or-so experiments, but nearly all of them template-style ideas, full of widgets and things you'd find on a lot of websites. Countdown timers, “only a few left”, loss-aversion copy, badges telling you how many people had just bought a ticket.
The programme ran on a playbook of what usually makes people buy, and nobody had been asked to go and find out what these particular customers actually wanted.
The boring stuff won
And like a lot of playbook stuff, it didn't work. The pushy stuff lost or did nothing. Customers were already a bit wary of prize draws, largely because of how some operators in that space behave, and a ticking clock just suggested this might be another one of them.
The tests that did win were fairly mundane by comparison. Making it clearer what you were actually getting for your money. Telling people their entry was safe when they were about to pay.
So we listened
The team came to us because they wanted to do it differently. So at the same time as testing, we were listening. We put AWIP Listen, our own privacy-first survey tool, straight after checkout, so people who'd just bought could tell us how it went without being tracked for it. We ran a perception study, and we went through reviews and customer service conversations.
It told us things no playbook would. People weren't always sure how the subscription worked or what they'd pay after the first month. The offer they'd clicked on in an ad wasn't always the first thing they found on the site. And a lot of people thought it sounded too good to be true, and wanted to check up on it somewhere else before paying.
As those findings came in, they became the tests, and a countdown timer fixes none of them.
And we're fast
The first tests went live within a week of the programme starting, with the research running alongside rather than holding everything up.
The previous agency averaged about three tests a month, and by the end were down to one. We're running at nearly three times that, with eight tests live right now. Doing the research properly doesn't slow you down. Guessing does, because you have to keep guessing again.
We took down the pop-up
Every new visitor was met with an email sign-up box before they'd seen a single prize. It's a standard move, and it's there for the business, not the customer. We tested taking it away.
Revenue per visitor went up, clearly, and people on mobile scrolled further and tapped through to more prizes. We were straight about what we couldn't see: the sign-up tool wasn't connected to the testing platform, so we couldn't measure the sign-ups it cost. The team weighed that up and switched the pop-up off for good.
Inherited a testing programme that isn't going anywhere?
Start with a Teardown. We'll tell you what's worth keeping, what to bin, and what your customers have been trying to tell you.
Get in touch