Experimentation Key Steps

A/B testing Process

 

Within webpages, nearly every element can be changed for a split test. Marketers and web developers may try testing:

  • Visual elements: pictures, videos, and colors
  • Text: headlines, calls to action, and descriptions
  • Layout: arrangement and size of buttons, menus, and forms
  • Visitor flow: how a website user gets from point A to B

Some split testing best practices include:

  • Elimination: fewer page elements create less distractions from the conversion goal
  • Focus on the call to action: text resonates differently depending on the audience
  • Aim for the global maximum: test with the overarching goal of the website in mind, not the goals of individual pages
  • Provide symmetric and consistent experiences: make testing changes consistent throughout the visitor flow to improve conversions at every step of the process

The following is an A/B testing framework you can use to start running tests:

  • Collect Data: Your analytics will often provide insight into where you can begin optimizing. It helps to begin with high traffic areas of your site or app, as that will allow you to gather data faster. Look for pages with low conversion rates or high drop-off rates that can be improved.
  • Identify Goals: Your conversion goals are the metrics that you are using to determine whether or not the variation is more successful than the original version. Goals can be anything from clicking a button or link to product purchases and e-mail signups.

KPI: Success metric, counter metric ,quality metric

  • Generate Hypothesis: Once you’ve identified a goal you can begin generating A/B testing ideas and hypotheses for why you think they will be better than the current version. Once you have a list of ideas, prioritize them in terms of expected impact and difficulty of implementation.
  • Create Variations: Make the desired changes to an element of your website or mobile app experience. This might be changing the color of a button, swapping the order of elements on the page, hiding navigation elements, or something entirely custom. Many leading A/B testing tools have a visual editor that will make these changes easy. Make sure to QA your experiment to make sure it works as expected.
  • Power Analysis: calculate sample size and test duration
  • Run Experiment: Kick off your experiment and wait for visitors to participate! At this point, visitors to your site or app will be randomly assigned to either the control or variation of your experience. Their interaction with each experience is measured, counted, and compared to determine how each performs.
  • Sanity Check on randomization between different test/control groups and check sample ratio (is it the same or significantly different from the deigned ratio split)
  • Check assumptions for ANOVA

Each group sample is drawn from a normally distributed population.

All populations have a common variance.

All samples are drawn independently of each other.

Within each sample, the observations are sampled randomly and independently of each other.

Factor effects are additive

After fitting an ANOVA model it is important to always check the relevant model assumptions. This includes making QQ-plots and residual plots.

  • Analyze Results: Once your experiment is complete, it’s time to analyze the results. Your A/B testing software will present the data from the experiment and show you the difference between how the two versions of your page performed, and whether there is a statistically significant difference.

If your variation is a winner, congratulations! See if you can apply learnings from the experiment on other pages of your site and continue iterating on the experiment to improve your results. If your experiment generates a negative result or no result, don’t fret. Use the experiment as a learning experience and generate new hypothesis that you can test.

Google permits and encourages A/B testing and has stated that performing an A/B or multivariate test poses no inherent risk to your website’s search rank. However, it is possible to jeopardize your search rank by abusing an A/B testing tool for purposes such as cloaking. Google has articulated some best practices to ensure that this doesn’t happen:

  • No Cloaking – Cloaking is the practice of showing search engines different content than a typical visitor would see. Cloaking can result in your site being demoted or even removed from the search results. To prevent cloaking, do not abuse visitor segmentation to display different content to Googlebot based on user-agent or IP address.
  • Use rel=”canonical” – If you run a split test with multiple URLs, you should use the rel=”canonical” attribute to point the variations back to the original version of the page. Doing so will help prevent Googlebot from getting confused by multiple versions of the same page.
  • Use 302 Redirects Instead Of 301s – If you run a test that redirect the original URL to a variation URL, use a 302 (temporary) redirect vs a 301 (permanent) redirect. This tells search engines such as Google that the redirect is temporary, and that they should keep the original URL indexed rather than the test URL.
  • Run Experiments Only As Long As Necessary – Running tests for longer than necessary, especially if you are serving one variation of your page to a large percentage of users, can be seen as an attempt to deceive search engines. Google recommends updating your site and removing all test variations your site as soon as a test concludes and avoid running tests unnecessarily long.

An e-commerce company might want to increase the number of completed checkouts, the average order value, or increase holiday sales. To accomplish this, they may A/B test:

  • Homepage promotions
  • Navigation elements
  • Checkout funnel components

A technology company might want to increase the number of high-quality leads for their sales team, increase the number of free trial users, or attract a specific type of buyer. They might test:

  • Lead form components
  • Free trial signup flow
  • Homepage messaging and call-to-action

Comments & Responses

Leave a Reply

Your email address will not be published. Required fields are marked *