Beyond A/B testing: how ANWB experiments when traditional testing isn't an option

From measuring behaviour to changing it

Martijn Ras works as a Web Analyst and Conversion Specialist at ANWB. His experimentation journey started with a fascination for understanding why people behave the way they do online.

Early in his career, he mainly worked with online marketing, content, and web analytics. But over time, he realised that measuring behaviour only tells part of the story. Data can show what people are doing, but experimentation gives you the opportunity to actively improve their experience and validate the assumptions behind your decisions.

That gradually changed his role. Instead of primarily reporting on performance, Martijn became increasingly involved in helping teams use evidence to make decisions.

Today, he works in a customer acquisition team that supports several business lines within ANWB. A typical week combines analysing the performance of acquisition channels, studying user behaviour, identifying optimisation opportunities, and developing experiments.

But there is also an educational side to his role. Martijn helps colleagues understand data and experimentation, interpret results, and set up experiments themselves. For him, building experimentation capability is therefore just as important as running individual tests.

What if you can’t run an A/B test?

That broader view on experimentation ultimately resulted in the case that won ANWB the Beyond Web Award in 2025: Causal Impact analysis with Prophet.

The experiment started with a question around branded search campaigns for Wegenwacht. ANWB wanted to understand the incremental effect of these campaigns, but a conventional A/B test was not a realistic option.

Instead of randomly splitting users into a test and control group, the team took a different approach. They stopped branded search campaigns for selected keywords and used historical data and unaffected control variables to model what would most likely have happened if the campaigns had continued.

By comparing this counterfactual scenario with the actual results after the intervention, the team could estimate the incremental impact – including the uncertainty surrounding that estimate. For Martijn, the most important outcome went beyond the specific result.

“It proved that it is definitely possible to experiment and get learnings from situations where traditional A/B testing is not an option.”

That is precisely what made the case such a strong fit for the Beyond Web category. Experimentation does not have to start and end with a website or app, nor does every question need to fit neatly into a conventional A/B testing framework.

Celebrate insights, not just wins

That philosophy also shapes Martijn’s advice for organisations trying to build a stronger experimentation culture: start small and build trust.

Large success stories can help create enthusiasm, but they are not what ultimately makes an experimentation programme sustainable. Consistent learning does.

Within his own team, Martijn therefore tries to work with a continuous optimisation cycle that begins with discovery. Involving the whole team at this stage helps identify problems and opportunities before jumping straight into solutions or experiments.

Just as importantly, he believes teams should celebrate insights rather than only successful variants.

“Most experiments are technically ‘losers’ because the variant performs worse. However, they disproved assumptions and prevented costly mistakes.”

Looking back, Martijn would have invested even more time in discovery and research when he first joined the team. Building a strong backlog of opportunities and hypotheses early on would have made it easier to maintain that foundation later, when day-to-day priorities started competing for time.

A strong case doesn’t need to reveal everything

Submitting the Causal Impact case for Experimentation Heroes was not originally part of the plan. A colleague who had previously presented at the event suggested that Martijn enter the experiment.

Before submitting, several people were involved, including product owners, the CRO lead, and the specialists who had worked on the experiment. As with many organisations, ANWB is careful about publicly sharing customer and commercially sensitive information.

That did not turn out to be a major obstacle. Results could be communicated in percentages without revealing sensitive business information, allowing the submission to focus primarily on the methodology, learnings, and business value.

Involving stakeholders early also helped. Rather than creating resistance, the case became an opportunity to demonstrate experimentation internally. Some stakeholders even offered to help get the story out.

Martijn received plenty of advice on how to present the case as a representative of ANWB. What surprised him most during the judging process, however, was what the jury wanted to hear about.

“As practitioners, we’re often focused on results and uplift percentages, but the judges seemed equally interested in the reasoning and the choices that were made in the process.”

For Martijn, that reinforced an important principle: experimentation is ultimately about learning, not just winning.

From one experiment to three new teams

Winning the Beyond Web Award also gave experimentation greater visibility within ANWB.

External recognition provided additional credibility, particularly among colleagues who had previously had little involvement with experimentation. More stakeholders became interested in how experimentation worked and how it could help their own teams make decisions. More importantly, the methodology behind the winning case started spreading.

Three other teams were inspired to explore the same approach for their own experiments, while Martijn was invited to give presentations to several teams about Causal Impact analysis.

Not everything they hoped to achieve materialised. There were plans to work with a data science team to automate parts of the methodology and make it easier for other teams to replicate, but that initiative did not ultimately get off the ground.

Still, something more fundamental had changed. Colleagues increasingly realised that the inability to run an A/B test does not mean you cannot experiment. That mindset may ultimately be more valuable than replicating one specific methodology. Instead of asking “Can we A/B test this?”, teams can start by asking “What are we trying to learn, and what method can help us answer that question?”

Don’t optimise experimentation for the number of experiments

That distinction is also central to something Martijn believes the experimentation field sometimes gets wrong.

Many organisations put considerable effort into increasing testing velocity: more experiments, faster cycles, higher volumes. Martijn does not dispute that velocity can be valuable, but argues that it only matters if those experiments result in better decisions.

“Organizations spend too much time trying to increase the number of experiments and not enough time improving the quality of their learning.”

That can become particularly problematic when teams avoid larger experiments because they require more time or resources, or because legacy systems make them difficult to execute.

Meanwhile, customers are changing quickly – and so are their expectations, motivations, and behaviour.

For Martijn, the objective should therefore never be experimentation for experimentation’s sake. Running more tests is not the end goal. Building a deeper understanding of customers and making better decisions is. It also explains the advice he would give his younger experimentation self: “Spend less time trying to get everything perfect from the start and more time trying to learn something new.”

Experimentation beyond the browser

Looking ahead, Martijn expects the boundaries of experimentation to expand even further. He sees experimentation becoming increasingly intertwined with AI, product discovery, and personalisation. AI can help teams identify opportunities, generate hypotheses, analyse qualitative feedback, and accelerate experiment design.

But he also expects the environment in which experimentation takes place to change fundamentally. If developments continue at their current pace, traditional websites could become less central as customers increasingly spend their time interacting with AI platforms rather than browsing websites directly.

That would create new questions for experimentation professionals. How do you understand customer behaviour when journeys no longer take place primarily on properties you control? How do you measure impact across different environments? And which methods do you use when a traditional controlled experiment is impossible?

In that sense, ANWB’s winning case points to a much broader lesson for the field: don’t let the methodology determine which questions you are willing to ask.

Sometimes the most valuable experimentation starts precisely where the traditional A/B test ends.

An experimentation hero from an unexpected field

Asked who he considers an Experimentation Hero himself, Martijn looks beyond CRO altogether. As a mixed martial arts practitioner, he points to Dutch MMA pioneer Bas Rutten. The early days of MMA offered few established routes to success, forcing its pioneers to continuously experiment with training methods, techniques, and even equipment.

For Martijn, Rutten’s open, positive, and enthusiastic approach embodies much of what makes a good experimentation mindset: try new things, learn from experience, keep improving, and remain positive when things do not work as expected.

That experimental approach ultimately helped Rutten become the first Dutch UFC Heavyweight Champion and later earn a place in the UFC Hall of Fame.

It is perhaps an unconventional example for an experimentation professional. But it illustrates the same principle that runs through Martijn’s work at ANWB: when there is no established path forward, experimentation becomes even more valuable.

Will you take experimentation Beyond Web in 2026?

On 5 November 2026, experimentation professionals will once again come together at DDMA Experimentation Heroes at Hotel CASA Amsterdam. The nominees will share their cases, the latest developments in the field will take centre stage, and the new Experimentation Heroes will be announced.

Do you have a case that, like ANWB’s, shows that experimentation can go beyond websites, apps, and traditional A/B tests? There is still one final opportunity to compete for an award. Submit your full case for Experimentation Heroes 2026 by 18 September 2026 before 17:00 PM.

Maybe your team will be taking the Experimentation Heroes stage on 5 November. More information >>

Content DDMA 

Kennisbank voor marketing en data