← All articles

Experimentation operating models: centralised, hub and spoke or everyone at once?

Who owns A/B testing in your business changes how much of it happens and how much you can trust it. The three common operating models, what each is good for, and the signs you've outgrown yours.

Jon Crowder7 minute read
Four rows of Victorian motion-study photographs: a baboon on all fours, a horse walking, a racehorse galloping and, in orange, a cockatoo in flight
Credit: Eadweard Muybridge, Animal Locomotion, plates 747, 574, 626 and 758 (1887), Boston Public Library

For a while AWIP.io has ranked first on Google for "what is the best operating model for experimentation?" which is very flattering but also awkward because I hadn't actually written that article... Until now! Google was presumably making do with something that I had written that was sorta adjacent. But here's the actual answer!

Why a model matters

Most businesses don't really explicitly choose an operating model for experimentation and it is just something that happens to them. Usually marketing either contracts an agency or contractor, buys a testing tool, runs some tests, gets some result and that's kinda where it stops. Sometimes they add to their teams and experiment in more places, but even then those teams act independently of each other without shared governance or oversight.

Your model will decide multiple things that matter, which is how many experiments can you run, how far can you trust the results, and do we learn anything from the tests we run? If we get it wrong in one specific direction (governance) then testing becomes a bottleneck. If we get it wrong in the other direction (throughput) then it becomes meaningless noise.

The three most common models

Centralised

One team owns, and is responsible for, experimentation. They are the disciplined and trained team who people must go to for experiments. Everyone else submits their ideas and hunches and the central team researches, prioritises, builds, runs and analyses the experiments.

A row of small squares queueing along a dotted line towards one large circle
Centralised: every idea queues for one team.
  • Good for: getting started, keeping quality high, businesses with one main website and one main funnel.
  • Bad for: volume. This team is rarely staffed as highly as production teams. Experiments are queued and the central team becomes a bottleneck that the rest of the business frequently learns to route around. If you're in this model, you've probably experienced "Oh this team has launched that new feature... No they never even asked about testing it"

Centralised is pretty common, because businesses don't particularly want to restructure the way a lot of their processes work, train a load of people and bring a load of new skills into a team. So they like the idea that a central team is "plug and play" and also that it can be unplugged if they don't get the results or the speed they were expecting from it.

I've seen this go wrong in a very particular way. When tests are scarce, every test becomes a precious resource and so the business starts "filtering" its ideas by guesswork and hunches to find the one that will "definitely win" and is therefore the best use of the precious resource. This is really the opposite of what testing is for, because the selectiveness has once again steered away from what customers need or want and towards what the team likes or wants. At the same time those changes that can't get through the queue just end up going out live and untested. That's exactly what AWIP found at a global leisure and tourism business that had managed 12 experiments last year when we took over from another experimentation function.

Hub and spoke (a centre of excellence)

A central team owns the platform, the methods, the training and the quality bar. Product, marketing or regional teams run their own experiments within those rules, often with someone from the hub embedded or on call.

One large circle joined by spokes to eight smaller circles, each with a dot inside
Hub and spoke: the centre sets the standard, with a tester embedded in each team.
  • Good for: businesses with several teams or products who all need to test, and want results they can compare.
  • Bad for: businesses without the senior people or resource to staff the hub. A centre of excellence without excellence driving it just ends up becoming another gatekeeper or slow approval step.

At enterprise level this is the much more common model. You almost have an embedded testing function within each team, but those people are largely responsible to a central team that acts as the centre of excellence. Intuit worked that way. Sky, I believe, works that way now and there are plenty of other businesses of that size that do the same.

Decentralised

Every product team runs its own experiments, with its own analysts, on a shared platform.

Seven separate circles, each with its own dot, resting on one long bar
Decentralised: every team tests for itself, on one shared platform.
  • Good for: product knowledge. The people who know the product best are the ones designing the tests, which really helps with the nuances.
  • Bad for: any business where nobody owns, or is responsible for, the method. Without strong shared standards you will get every team defining success completely differently plus peeking at results, p-hacking, cherry picking, declaring wins that weren't real, failing to track guardrails because the team needs a win etc etc.

In an ideal world you wouldn't really have experimentation teams so much as you would have product teams with experimentation as a standardised part of their workflow. If you're a product team shipping website changes, or software, or features, you should build each one in a way that is not just "management wants us to build this" but rather "here is a problem we're trying to solve, here's our hypothesis on how we can change behaviour, and here's the experiment we're going to run to see whether the treatment we've come up with holds water". You should have architecture that supports it, so you can build that test, run it, measure it and make a decision based on the outcome. That's pretty cool and it's a really good, strong way to run the programme.

I have rarely seen it done in practice. Somebody almost always needs to own the methodology and be rigorous about it, because I guarantee you other parts of the business, the ones who want to ship faster or claim a success without the data to show it is one, will try to go round the process. Also, a lot of senior leadership in those businesses are hired on the expectation they'd be the brave captain steering the ship and to a lot of people that means hands-on direction, telling people what to build and where to go, which is incompatible with this model. So for the decentralised model to really work, standards still need to be really, really high.

How the big experimenters got there

There's decent research on how companies grow into this.

In 2017 researchers working with Microsoft described four stages they called crawl, walk, run and fly.

At "crawl", product teams lean on a standalone data science team because they don't have the expertise themselves.

Four frames of a baboon moving on all fours
Crawl. Muybridge, Animal Locomotion, plate 747.

By "walk", product managers own experiments while data scientists, sometimes embedded in the teams, still run the analysis alongside a central platform team.

Four frames of a heavy horse walking
Walk. Plate 574.

By "run", product teams employ their own data scientists, trained by the central team, and are running about a hundred experiments a year.

Four frames of a racehorse and jockey at full gallop
Run. Plate 626.

At "fly", controlled experiments are the norm for every change to every product, minor bug fixes included. Thousands of them are live at once, so the platform has to spot tests that clash with each other and shut down the harmful ones automatically. Very few businesses will ever fly.

Seven frames of a cockatoo in flight, printed in orange
Fly. Plate 758.

The pattern to take from this is that nobody starts decentralised. You centralise to build skills and standards and then hand them out as teams prove they can act independently whilst maintaining the high standards for experimentation.

Which one should you have?

A rough guide:

  1. Fewer than a few experiments a month, one main site: centralised. Usually a small team or one person and an outside partner.
  2. Several teams who all want to test, and someone senior to set the rules: hub and spoke.
  3. Hundreds of experiments a year, analysts in most teams and a platform you trust: decentralised, with a central team responsible for owning the methods and the platform.

And whichever one you pick there are three rules you should follow across all of them. Before anything launches you set what you're measuring and for how long. Keep a single shared log of each test and what it taught you (and give teams time to own and maintain this!). Agree as a business what your red lines are, which in our case includes anything that "works" by misleading people, but for businesses may include anything that erodes your core offering or weakens your position in the market.

Signs you've outgrown your experimentation model

  • Ideas wait for months to be tested and often just end up going live on vibes because the test would take too long. This means that you need more capacity or more teams that are able to run tests.
  • Teams are testing around the programme rather than through it.
  • Two teams have declared wins that contradict each other. Team A says that a sticky banner made them more sales. Team B says a sticky banner harmed their sales. (This may still be true, but it's a great signal for somebody skilled to investigate)
  • Nobody can really tell you what we have learned as a business or what other departments should do with the learnings.
  • Results frequently get presented in meetings, but can be easily challenged by anyone who doesn't want to believe them because there isn't agreement or alignment on the core vitals of the programme (metrics, confidence intervals, guardrails).

If you're trying to move from one model to another or need support with your programme, AWIP can offer all parts of a CRO service, including support on developing infrastructure, as well as coaching existing teams.

Jon Crowder runs Another Web Is Possible.

Sources

  1. The evolution of continuous experimentation in software product development, The Morning Paper, summarising Fabijan, Dmitriev, Olsson and Bosch, ICSE 2017

Ready to get started?

Book a Call