Navigating the Evolving Landscape of Experimentation: Challenges and Insights

Maarten Plokker is the Managing Director of SiteSpect Europe. SiteSpect is one of the sponsors for the 2023 edition of DDMA Experimentation Heroes, scheduled for October 23rd. On this day, the most outstanding experimentation heroes from the Dutch Market will gather to showcase their finest experiments on stage. If you’re keen on participating or attending, you can find additional details at experimentationheroes.com.

Q: Gathering reliable data nowadays is easier said than done. What leads to this difficulty?

A: Privacy laws have and will continue to tighten. With a rising focus on user privacy, browsers now have built-in privacy features activated by default. We appreciate this as users, however, for those curating user experiences, these changes have impacted data previously used for enhancing and A/B testing digital interactions. Browsers have all but eliminated the use of third-party cookies. This has been an issue for the ad-serving industry, but a non-issue for experimentation since testing platforms deal with first-party cookies. However, many browsers now also delete first-party client-side cookies after 7 days, some in as little as 24 hours. This means a returning user might be repeatedly treated as a new user if they leave the site and don’t return for a period of time. This not only dramatically skews data on things like loyalty and new user acquisition, it also creates a whole segment for which you can’t provide a personalized, sticky user experience. And without any evident errors, everything appears to be working fine in terms of personalization, data collection, and data accuracy, but it’s not – far from it. Skewed data, bad user experience, the inability to personalize and experiment on a segment of users…it’s very counterproductive to experimentation and CRO. With client-side tools, you also have issues like JavaScript errors, flicker, and latency, and these all can negate the “improvements” you’re trying to test. While SiteSpect natively addresses all these issues, most tools don’t, unless you move to a more server-side approach.

Q: You used the terms client-side and server-side a bit just now. Could you educate us a bit on the distinctions between the two and why some organizations or agencies might choose one or the other?

A: Sure, let’s start with some quick definitions. Client-side pertains to changes and tests done at the browser, while server-side involves changes and tests delivered from the servers. SiteSpect is capable of doing both, because it uniquely alters things in the middle of that flow, but more on that later In traditional client-side experimentation, a browser receives a web page, and the page gets rendered in the browser. Then, code is used to reach out to a third party and modify the experience after the page is loaded and collect data on it. A tool like the now-retired Google Optimize is a prime example of a client-side tool. Client-side tools are typically user-friendly and easy to get going but have limitations. While they can assess elements like copy, images, and promotions, they can’t optimize more complex elements like checkout flow, shipping thresholds, search and recommendation algorithms, or test mobile. They also suffer from the data and performance issues I outlined a few minutes ago.

Server-side experimentation addresses those performance and data issues and allows for more advanced use cases. However, server-side is far more resource-intensive, requiring coding and developers, and typically constrains experimentation to release cycles. Often a transition to server-side moves control from Marketers to Developers, with the hope that the Product team can and will balance marketing, feature, and UX priorities equitably. Still, whether it’s client-side, server-side, or both, we’re often talking about siloed tools that confine teams, because the tools themselves are almost always offered with different implementations, logins, and interfaces, and they are designed based on different technical skill sets. SiteSpect is the only tool that straddles these two worlds by making changes between the client and server, and so it combines the benefits of both client and server-side and doesn’t drag with it the issues of either, and it does so in a single user interface which is something few tools do.

Q: You mentioned Google Optimize and its recent sunset, and I recently heard Oracle is sunsetting its Maxymiser product as well. What do you make of that? Is it a coincidence?

A: I don’t think one necessarily has to do with the other, but that doesn’t mean it’s only a coincidence. Some of these larger companies like Google, Oracle, and I could even throw Adobe in there as well, they do many things, but experimentation has never been at their core. It has seemed more like an afterthought. To their credit, these companies keep a keen eye on their longer-term strategies and their core focus and are willing to bail on things that aren’t part of that. There are companies out there that are 100% focused on experimentation and have really honed their offerings. I think that plays into a larger company’s appetite to continue to develop, market, sell, and support what has become a set of sub-par experimentation tools. So that’s what I think happened there. We’ve already converted a ton of former Google Optimize customers. I’m almost sad that the sunset deadline has passed because it’s been good for business [smiles]. I’ll be watching the movement with Oracle, as their announcement is fairly recent. Separately there’s a host of companies like Optimizely that were once heavily focused on experimentation, but they’ve gone through acquisitions and mergers, etc. and the market feedback shows they are diversifying their R&D and sales focus significantly towards the other offerings that are now part of their portfolio. It’s all pretty interesting.

Q: It certainly sounds like there is a lot of movement out there. So what guidance can you give organizations who are either embarking on experimentation for the first time, who have been displaced by their tool sunsetting, or who are looking to expand from pure client-side to server-side

A: There are things you want to consider about your organization, your UX priorities, your culture, and your technology disposition. We actually have a short and fairly solid eBook we put out to help organizations walk through that process. Anyone can contact me for a copy of that. Most organizations have a build-or-buy question to answer, and the answer may differ based on the technology and what it’s providing to their competitive edge and overall strategy. We sometimes see super large enterprises with a ton of developers build their own experimentation platforms, but it’s rare, and when we see it, it’s almost always of the server-side variety. However, those companies are still often functionally behind a fairly mature experimentation space. So while we may see that with a Netflix or Booking.com, for the most part, we still see most organizations, midsize, large, and even very large enterprises buying solutions like ours, and that’s because they want something that already brings Marketers, Developers, Product Managers, and CRO Specialists under one platform umbrella. They want client-side, server-side, personalization, API transformation, and product recommendations in one interface, and they want to include agency-level optimization consulting as extensions of their own teams.

To conclude

In the dynamic landscape of experimentation, where influences, regulations, and technological advancements shape the path forward, the challenges and insights discussed in this interview provide a profound understanding of the intricacies involved. We hope that Maarten Plokker’s insights shed light on the evolving challenges of data privacy regulations, the transition from client-side to server-side experimentation, and the changing dynamics of major players in the field. As organizations embrace experimentation to enhance user experiences and drive growth, the complexities of choosing the right approach, tools, and strategies become paramount. With the sunsetting of tools like Google Optimize and Oracle Maxymiser, a pivotal shift is observed, reflecting the market’s demand for dedicated, specialized solutions. As the experimentation journey continues, organizations must carefully consider their unique needs, technological landscapes, and collaborative aspirations, aiming to strike a harmonious balance between innovation and stability. In this era of transformation, this combination illuminates the path forward.

Bob Younge

Contentspecialist