Automisation in Experimentation: 8 ways to automate your CXO Program

Mark de Winter is the Director of Product at Clickvalue. Edwin de Brouwer is Lead Conversion Specialist at Clickvalue. Clickvalue one of the sponsors of DDMA Experimentation Heroes 2023 on October 31st. Curious about the very best experiments of 2023? Secure your ticket now at: experimentationheroes.com.

The benefits of CXO automation

The benefits of automation within experimentation programs can be divided over 4 categories:

• Efficiency: automating certain parts of your CXO Program will help lower costs and enables opportunities to reallocate resources and increase experimentation and change velocity.
• Scale: When scaling towards double or triple digits in monthly experiments, you can’t do without automation. Otherwise you’ll quickly lose grip over your program.
• Process: Many aspects of the experimentation process can be highly standardized. From detecting winners to implementation requests, MDE calculations, Priority calculation, etc.
• More fun: Having automation in place, will help minimizing boring, cumbersome tasks, increasing the opportunities for team members to focus on the stuff they like most.

When considering the potential applications of these automations, we look at the four phases of the CXO process, each of which presents numerous opportunities for automation – 8 in total.

• Phase 1: Problem Discovery: Exploratory research to identify opportunities
• Phase 2: Problem Validation: Prioritizing opportunities by stacking evidence
• Phase 3: Solution Discovery: Creation of solutions
• Phase 4: Solution Validation: Validating solutions (through experimentation)

Phase 1: Problem Discovery (Research)

In the Problem Discovery phase, where the goal is to uncover as many problems and optimization prospects as possible, several tasks can be automated. If don’t have access to a continuous flow of opportunity research to fill your opportunity backlog, you can automate things to still continuously discover opportunities to accelerate experimentation and increase the pace of change.

Some examples:

• Automated anomaly detection: Anomalies are large deviations or changes in your data. Many UX tools offer some sort of anomaly detection such as Contentsquare, Fullstory, Decibel, etc. But anomalies can also be set-up based on predetermined thresholds in your own analytics stack. These anomaly reports can be automated into your workstream by making sure these automatically end up in your insights backlog to be processed by a researcher.
• AI in research: AI is finding its way into opportunity research quickly. Finding rationale behind user behaviour or for quickly processing user data. Parts of this process can be automated as well. An example: One can set up a simple language processing AI on your customer insights, opportunities and/or experiment database to generate optimization opportunities based upon your own data. Of course these opportunities need to be reviewed by an experienced researcher, but it can quickly become very powerful when increasing the size of your database and the learning capabilities of GPT’s.

To increase the number of discovered problems and opportunities it becomes crucial to move away from ad-hoc research towards always-on research where opportunities find you instead of (re)starting expensive user research resources. Use those resources to interpret data, improve algorithms and stack evidence for opportunity prioritization.

Phase 2: Problem Validation

During the Problem Validation phase of the CXO Process, insights and problems found in the problem discovery phase are validated. Any discovered user problems are looked at closely to determine if they actually exist and if they need to be fixed or acted upon. This is done by stacking evidence from multiple research sources and prioritising problems and opportunities. Several aspects of this validation can be automated.

Some examples:

• Prediction models: To predict what opportunity will have the largest impact or has the highest chance of success, you can use prediction models. You can use AI for this, but you can also use predefined search queries upon your insight-, opportunity-, or experimentation database. For example, if you’d like to understand if opportunities with urgency on a homepage lead to a higher impact compared to opportunities with social proof on a homepage, you can set up this in your database and use predefined queries to provide you with the opportunities with the highest impact based on experiment or research data. You can do this by setting up a customer knowledge bank with opportunity and experiment data, which can be used to determine the highest amount of impact or success.
• Automated opportunity mapping: Such as Customer Journey Maps, Opportunity Solution Trees, Goal trees, etc. Opportunities are not one dimensional and not linear. They are dynamic and have clear relationships with each other. Mapping these opportunities in a structured manner, for example in an Opportunity Solution tree can drastically improve the grip on your research process. A way to do this is to ensure that relations between opportunities, experiments, solutions, etc. are identified in your customer knowledge bank. Once these relations are established, a visual representation can be made to present opportunities and opportunity areas to your team ensuring a better understanding of what research is doing.

Phase 3: Solution Discovery

During the Solution Discovery phase solutions are gathered to fix the issues found during problem discovery and validation. During this phase UX Designers, copywriters and psychologists think about ways to best solve issues and create the best performing user experience.

Some examples:

• Prediction models: Similar to prediction models in the problem validation phase, prediction models can be used in the solution discovery phase to determine what solution will likely have the highest impact or chance of success. If you’ve setup your customer knowledge database in a way that solution designs and proposed changes can be quickly processed, you can leverage that information into information to prioritize your solutions and UX Designs based on for example impact or successfulness.
• AI in solution design: Many design tools such as Figma embrace the power of AI to quickly generate multiple designs based on certain elements or one design. Also AI can be used to quickly generate solution ideas for a specific problem or prompt, saving large amounts of precious UX design resources.

Phase 4: Solution Validation

The Solution Validation phase, in which solutions are validated through – for example – experiments, is a phase where automation is omnipresent. Many parts of this process are highly standardized and can be easily automated.

Some examples:

• Automated experiment reporting: Because of the standardized nature of specifically experiments, reporting can be automated easily. For example, Google sheets can be connected to predefined datasources to include a hypothesis, KPI impact/dashboard, revenue impact, imagery, conclusions and recommendations. This saves up a lot of work creating decks. Also, many workflow tools offer API’s to automatically generate large parts of reports, leaving more room and time for optimizers to add interpretation and context.
• Automated monitoring & decision support: Monitoring your own first party experiment-data from analytics can become quite a hassle. Especially when running a high velocity program. Due to the standardized nature of how experiments are setup with predetermined primary goals for example, this task can be easily automated.

Automations in solution validation can bring a lot. From automatically fetching all active experiments from testing tools, to creating the needed segments on the fly and extracting all data on a daily basis. In short you can benefit from it in three ways:

• Reduces manual work of monitoring tests with 80%
• Increases quality as we have a richer view of the data and lower chance of human errors
• You can automate things in coherence with any AB testing tool. 

To conclude

Automation in CXO is an important step to the future and it will continue to grow as capabilities and innovations are increasing exponentially with AI. Automation creates room for optimizers to be creative and to work on expanding an experiment driven culture within the organisation. The fun stuff 😊.

Mark de Winter

Managing Consultant CRO/CXO & Conversion Optimization Specialist | ClickValue (voorzitter)

Edwin de Brouwer

Lead Conversion Specialist | Clickvalue