Against the force of fierce modern competition for the next customer and market share, companies need an edge. However, that edge will not be apparent, nor will it present itself plainly in a Harvard case study. To find your edge, a process of experimentation will be key to deciphering what works in your favor and what does not. Each company’s path will be different, shaped by industry, line of business, size, and geography. Yet, in general, any experimentation program starts with an experimentation hero and ends with a culture of experimentation. Our key takeaway from Opticon17 was that the platforms exist to empower experimentation by businesses, ranging from those just starting out—the experimentation heroes—to those enterprises seeking a companywide culture of experimentation.
How do we know? Through the success stories from a diverse set of users of experimentation platforms. Rocksbox, a direct-to-consumer curated jewelry service, exemplifies an experimentation hero. Their teams run experiments around pathing and content on a site built upon a single page architecture. Beyond an experimentation hero, we have a company such as the BBC—the publicly owned British broadcaster—where experimentation has been institutionalized into the workflow of their mobile and media product teams. Better yet, the New York Times Company relies on optimization technology as a catalyst for their firm-wide innovation efforts, which is the spirit of a culture of experimentation. While these companies differ, all three credit an optimization solution provider with unlocking business value by facilitating experimentation.
Beyond building the business case for how optimization can help you achieve results, there have been a number of product enhancements announced recently that we, as advisors on advanced optimization solutions, see as adding high-value. When you combine a powerful experimentation platform with our unique Full Stack Optimization testing services, you can begin to utilize both server-side and client-side testing to provide the greatest returns possible from your experimentation efforts.
While running experiments has changed a fair amount over the years, the pre- and post-experiment activities haven’t evolved as much. In a lot of cases, experimentation, ideation, test planning, analysis, and reporting can be a nightmare hodgepodge of sticky notes, spreadsheets, dashboards, and decks. Recently, the leading optimization solution providers have helped bring clarity to this space. Now, experiment ideas can be customized, cataloged, commented on, and graded within a single interface while different types of users can have access to this central repository, or only see those experiments that pertain to their domain. The newest optimization platforms can also bring in statistics from active and completed experiments so that one can check the health of all their experiments in one place, or roll-up performance in an executive level dashboard view.
In order to understand experiment performance, an organization needs to monitor the right metrics. This often means going beyond standard conversion, revenue, or order value metrics. Through a visual editor interface, an experiment designer can pick from a more detailed list of standard metrics, plus build complex, custom, designs by combining different metric building blocks in different combinations.
Accelerate Your Experiments
Metrics are key to quantitatively analyzing an experiment’s impact on business performance. However, that analysis has always come after a considerable wait time for an experiment to generate statistically significant results against the target population. This is a limitation of frequentist statistics. For example, if you are the New York Times and are testing different headlines for the same article; your experimentation manager needs to see results in hours or days, not weeks. To mitigate this problem, the latest optimization platforms feature machine learning algorithms that can deliver valid and actionable experiment conclusions at a much lower latency than traditional approaches.
If faster experimentation leads to more bandwidth for a greater velocity of experiments, the next two enhancements will fill that bandwidth by enabling experimentation beyond simple static webpages. Optimization solution providers have now rolled out native capability for single page architecture. Instead of hacking together an experiment for a dynamic page, a redesigned tag will allow an experiment designer to listen for page events well past initial page load and then respond with customized content, layout, and flow, as determined by the experiment specifications. This moves more of the setup of Single Page App experiments back into the visual editor from the code editor.
Another area of experimentation that won enhanced support is native apps. App developers, designers, and managers have now been empowered to understand their users in more specific ways. When rolling out new features the normal process is to design several options and review with internal stakeholders, then take a few of the final designs to a focus group. Of course, this designing, editing, and pruning process can still result in negative user outcomes with, consequently, negative business outcomes. Instead, why not experiment with unique designs of the same feature and rollout that feature in a systemic way to a greater and greater slice of your user group to see if the reaction is positive? This is exactly what the new Feature Management tools allow a business to do. An app owner can easily deploy, rollback, and experiment with new app features to get exactly the right customer experience.
Final Thoughts from Vegas
Whether your organization is new to experimentation, have already hired one or two experimentation heroes, have taken it to the next level with a full experimentation program, or have taken it to the max with a culture of experimentation, new enterprise platform technology can be the stimulus needed to increase your program velocity and drive improved business performance.