Your site's goal is to persuade visitors to take some action -- purchase a product, donate to a cause, take a survey. It's easy to track webpage traffic and conversions once they happen, but it can be tough - and time consuming - to anticipate what sort of page content will motivate your potential customers to take that desired action. Website optimization helps you to design a page that maximizes the effectiveness of your site.
This overview provides technical details on how Google Website Optimizer works. It is a companion document to the Website Optimizer Overview, which offers a high-level view of this tool.
Contents
Audience
The intended audience for this overview includes webmasters, web content designers, marketing managers or anyone else who wants to evaluate this optimization tool for use with their websites. This overview assumes that readers have a basic knowledge of website management and e-commerce.
| How Optimization Works | Back to Top |
A webpage is made up of a number of disparate elements, all of which work together to influence your viewers in ways that are not always intuitive. As a result, the only way to truly test page effectiveness is to actually change a page and see what happens. This is where Website Optimizer can help: by providing a way to quickly test many page designs and track user preferences.
Experimentation and Website Optimizer
Website Optimizer offers a robust laboratory environment for testing multiple versions of a webpage simultaneously, with extensive reporting tools to analyse and compare the effectiveness of each version.
Website Optimizer can be used with any webpage that attempts to persuade a visitor to take a specific action. In Website Optimizer, page effectiveness is measured by conversion rate; that is, the percentage of visitors who accept your call to action.
Using the Website Optimizer interface, you will set up and run an experiment to present one of many webpage designs to each visitor. As the experiment progresses, the Optimizer gathers traffic and conversion data, analyses the data and presents the analysis along with suggestions on next steps. You can then make strategic decisions based on hard data.
Website Optimizer uses Google Analytics for tracking and analysing experiment data. As a result, Website Optimizer has the same high quality performance and reliability as Google Analytics. As with Google Analytics, Website Optimizer can support experiments on webpages with millions of page views per day.
Multivariate Testing
Website Optimizer uses a multivariate testing model rather than the more common A/B testing. Multivariate testing (also called "factorial" testing) can offer much more fine-grained analysis. Instead of comparing the effectiveness of two completely different webpages, as in A/B testing, multivariate testing lets you change specific sections of your page and analyse how changes to each section affect the overall page's effectiveness. In addition, multivariate testing explores how combinations of page section content work together to influence your visitors.
For example, suppose you want to test two distinct versions of a webpage and, unbeknownst to you, visitors are most attracted to the headline on page A and the image on page B. With A/B testing, you will never discover this fact. Multivariate testing is specifically designed to detect exactly this circumstance.
In this approach, you identify one or more sections of your webpage, such as a headline, an image, a block of descriptive text or some script and then design some content variations for each section. During the experiment, visitors will see a combination of page section variations. The Website Optimizer collects data on the effectiveness of each combination and performs analysis on the data.
In a typical scenario, you might choose two page sections to be tested, such as a headline and an image and create one new variation for each section. In this scenario, four combinations would be tested:
- Original headline / original image
- Original headline / new image
- New headline / original image
- New headline / new image
Multivariate testing can identify useful information, such as
- which variation in each section performed best (that is, generated the most conversions)
- which combination performed best
- which sections seemed to have the greatest effect on the visitor
Interestingly, the best combination is not always a combination of the highest performing variations. A primary advantage of multivariate testing is that you are not simply testing the effectiveness of each individual variation, you are testing how well a combination works together (or not) to improve performance. For example, visitors to your mountain bike product page might really like the picture of a biker careening off a cliff, except when it is coupled with the also popular headline "SMASHING!!". The best combination turns out to be the careening biker photo with the headline "Push the Limit!".
It is also very useful to identify which elements on your page most influence your visitors. Multivariate testing looks at which sections result in the most extreme reactions between variations. For example, you might discover that your page gets the same conversion rate no matter which headline version is shown, but that visitors' reaction to the image varies drastically. This information helps you concentrate on the most important aspects of your page.
A note about "fractional factorial" testing:
A similar type of experimentation is the "fractional factorial" model. Taguchi, orthogonal arrays and other similar types of experiments are special cases of fractional factorial experiments. Fractional experiments are designed to limit (often severely) the number of combinations tested. In real world industries, such as industrial or agricultural, each combination tested carries a significant resource cost. However, the benefit of reducing combinations comes at the cost of limiting the conclusions that can be drawn from the experiment.
In website experiments, however, there is no cost to adding additional combinations, which means that there is no downside to using a full factorial design. With the same number of impressions, a full factorial design will reach the same conclusions as a fractional design, and—as the number of impressions increases—can yield deeper conclusions. In particular, with full factorial design, you can learn about interactions among factors, such as whether a specific text block influences the performance of a specific image.
| Experiments in Optimizer | Back to Top |
This section provides technical information relevant to the decisions that an experiment designer must make when preparing and implementing an optimization experiment. Use this information to determine the scope of an experiment and to recognise when you have enough data. If you are looking for a step-by-step guide to setting up and running an experiment, go to the Quick Start Guide.
Note: Using Website Optimizer, you can set up and run multiple experiments on your site.
Preparation and Setup
Website Optimizer provides a step-by-step user interface for setting up and running your experiment. For the first step, you will need to identify the content that you want to test and define the scope of your experiment.
Identifying Experiment Pages
The first step is to identify the page that you want to optimize. The page must meet two requirements:
- It should have some "call-to-action" mechanism that represents the goal of the page, such as a "Buy Now" link. This page is called the "test page".
- Accepting the call to action on the test page must eventually lead to a second page that can only be viewed by someone who has taken the desired action. This is called the "conversion page".
Both test and conversion pages are required to track webpage effectiveness.
For example, your test page might describe a product for sale with a "Buy Now" button. Clicking the button steps the visitor through additional, non-experiment pages for inputting billing and shipping information, followed by a "Submit Order" button. Once the visitor clicks the Submit button, they are directed to a confirmation page. A visitor only sees the confirmation page if they have completed the purchase process. As a result, a hit on this page is a true indicator that a conversion has occurred.
Selecting Test Page Sections and Variations
The next step is to select the parts of the test page content that you want to experiment with. As discussed in "Multivariate Testing", you need to identify one or more page sections.
Some obvious examples of a section are a page headline or a prominent image. You can designate just about anything as a section, including blocks of text, script and graphics, up to 64k of HTML (64,000 characters). Page content can be static or dynamically generated. (If you choose to experiment with dynamically generated content, you will need to do some extra scripting.)
For each page section that you select, you will need to develop one or more variations for it. The experiment essentially tests how much more or less effective your new variations are over the original content.
Note: A section must contain a single contiguous block of HTML. The HTML can consist of as much or as little as you want to test. For example, you may want to include two or more related elements, so long as they are physically located next to each other, such as an image and its caption.
Website Optimizer accommodates test scenarios with a huge number of section/variation combinations, more than sufficient for most needs. For those rare cases, however, here are some limits to keep in mind:
- Maximum page sections: 8
- Maximum variations per section: 127 in addition to the original content
- Maximum combinations = 10,000 (to calculate combinations, multiply the number of variations in each section, 2 sections with 3 variations each is 3 x 3 = 9)
Please note that the maximum number of combinations limits the actual number of page sections and variations you can use (127 to the 8th power far exceeds 1000), while still allowing for lots of flexibility. For example, you could designate three page sections, one with 127 variations and two with 2 variations (127 x 2 x 2 = 508). Alternatively, you could create an experiment with 8 sections, one with 7 variations and the rest with 2 variations (7 x 2 x 2 x 2 x 2 x 2 x 2 x 2 = 896).
Determining Experiment Scope
When selecting page sections and variations to test, you need to understand how these choices affect the scope of the experiment. The more page sections you select and the more variations you add for each section, the more combinations will be tested. The number of combinations, in conjunction with other factors including page traffic volume, is a major factor in how long the experiment will take to return meaningful results.
Here is a handy calculator for estimating the duration of your experiment.
The key factors determining duration include:
- Number of test combinations: This value is determined by the number of sections and variations that you have designated. To calculate, multiply the number of variations for each section together. For example, if there are three variations for the first section and two variations for the second, the number of combinations would be 3 x 2 = 6. Keep in mind that this number can add up quickly. If you designate six sections and want to test three variations for each, the number of combinations would be 3 x 3 x 3 x 3 x 3 x 3 = 729!
- Impressions: This value reflects the number of visitors that your test page gets. Pages with more impressions per day will have shorter test durations.
- Percent participation: This value indicates the percentage of visitors tracked during the experiment; you specify this value during the experiment setup. A value of "50" would include half of all visitors.
- Conversion rate: This value should be set to the conversion rate that the webpage has experienced in the past with the original content. Experiments on pages with higher conversion rates will likely return meaningful results more quickly.
- Expected improvement: This value indicates how much better you expect at least one new combination to perform over the original content. This is a best guess value; if your new variations are radically different from your original, then you are more likely to see a large difference in performance between the new and original variations. The faster at least one new combination outperforms the rest, the faster the experiment returns definitive results.
Expected improvement is expressed as a percentage; if the current conversion rate is 30%, then an expected improvement of 100% means that you are expecting to double your conversion rate to 60%. By default, Website Optimizer assumes an expected improvement of 12%.
These factors are described in more detail in "All About Statistics".
Keep in mind that the calculated experiment duration is only an estimate. Not only will actual traffic and conversion rates vary in a real experiment, the actual improvement experienced with some combinations will drastically affect the duration. If, for example, one combination exhibits a significantly higher conversion rate than any other, the experiment will conclude more quickly.
Setting up the Experiment
Once you have figured out the pages that you want to experiment with and the new content that you want to try out, you will use the Website Optimizer UI to set up your experiment and start running it. The UI will walk you through each step, providing help along the way. Later, you will also use this UI to check the progress of the experiment and see how the different combinations are performing.
One step of the experiment setup is to install scripts, which we provide, into your test and conversion pages. You will need to coordinate with whoever manages the website code for this step. These scripts accomplish three main tasks: (1) they identify the page as the subject of a specific experiment, (2) they determine what content is displayed each time someone views the test page and (3) they trigger the server to collect traffic data for the experiment.
During a later step, you will need to enter all the content variations into the Website Optimizer UI, which will store them on the Google servers.
Once the all steps have been completed, you will use the UI to launch the experiment. You will also be able to pause and stop an experiment as necessary.
Experiment Progress
Here is what is happening behind the scenes to make the experiment run.
Determining Content
When a user's browser requests a test page, the following decisions are made:
- Will this page view be included in the experiment? If you have indicated during setup that, for example, only ten percent of your visitors should be included in the experiment, only one in ten visitors will see experimental content and have their visits tracked. All other visitors will see the original content.
- What combination of experiment content will be displayed to the visitor? All combinations get equal and random play during the experiment.
The script installed on the test page triggers additional Google-side script. This script determines which content is to be displayed to visitors and whether to track test and conversion page hits. The experiment content itself, as well as all data collected on page traffic, is hosted by Google.
Delivering Cookies
Web surfing is not a linear activity; people do not move from one site to another in a single direction. Instead, a web surfer—especially one looking to make a purchase or some other commitment—will likely jump back and forth between sites, making comparisons, researching topics.
Why does this matter? Because a surfer might hit your test page multiple times before clicking that call-to-action button. If we counted every individual page hit, your stats would end up skewed—with lots of test page hits and only one conversion page hit—showing an artificially low conversion rate. Equally important for your experiment, a visitor should always be presented with the same test page content. If our mythical web surfer refreshes the page and gets different content each time and at one point clicks the call-to-action button, how can we determine which set of content prompted them to respond?
To solve these problems, Website Optimizer issues a “cookie” to each new test page visitor. Cookies provide a way to recognise return visitors to the page within a specified window of time. As a result, if a user visits your test page multiple times, Website Optimizer records only one test page hit. The cookie also stores the combination being displayed to the visitor so that, for the life of the cookie, the visitor will continue to see the same content. Website Optimizer uses persistent cookies with a life of 2 years.
Collecting Data
Experiment data is collected at two points: (1) when the test page is displayed (if the view is to be included in the experiment) and (2) when the conversion page is displayed.
Data collected when the test page is viewed include
- The experiment identifier key
- A test page hit
- The combination displayed
Data collected when the conversion page is viewed include
- The experiment identifier
- A conversion hit
A conversion is recorded when the visitor hits the conversion page after hitting the test page.
Experiment Completion
Experiments run as long as the experiment designer chooses to keep them running. Website Optimizer will not arbitrarily end an experiment, but will indicate when results can be considered definitive.
While an experiment is running, a definitive result is a moving target. The progress of an experiment depends entirely on the results of the experiment itself. In the end, though:
Experiment results can be considered definitive when one combination emerges as clearly superior to the others.
On the Website Optimizer reports page, the performance of all combinations is illustrated as a bar graph, shown in Figure 1. The horizontal position of each bar reflects the performance of that combination relative to all other combinations. Red, green and grey colours indicate performance relative to the original combination.
Figure 1. Conversion Rate Range Bar Graph
Note: The conversion rate range predicts the likely conversion rate if a combination were to be adopted for all users. This is discussed more fully in Results Details.
A clear winner is defined as a combination whose conversion rate range is higher than all other ranges and does not overlap with any other range. In the graphic above, Combination 2 appears to be a clear winner.
As an experiment progresses, there are several potential outcomes:
- The original combination emerges as the clear winner
- No new combinations emerge as clear improvements over the original
- One new combination emerges as the clear winner
- Two or more new combinations emerge as improvements over the original
If only one combination emerges, even if it is the original, this is an obvious signal that the results are definitive and the experiment can be ended. If several combinations show improvement, it can take more time for one of those combinations to break out from the pack. This scenario is most likely to happen when two or more combinations are very similar to each other. In this case, a clear winner may never emerge.
As the experiment progresses, Website Optimizer will offer suggestions for actions. For example, you may want to refine your experiment design based on progress to date, such as dropping variations that are clearly underperforming. Keep in mind, though, that while the experiment continues, there is always a chance that an underperformer will suddenly shoot into the lead. When conversion rate range bars are significantly overlapping, it is virtually impossible to identify which combinations will prove most effective.
Reports Analysis and Take-Aways
Website Optimizer reports provide rich detail about how your experimental content performed. There are, however, a few core conclusions that you can take away from your finished experiment, and these are summarised here. If you want to delve into the more subtle meanings of these reports, "All About Statistics" provides detailed information on how experiment results are derived and their exact meaning.
Combination Report Take-Aways
If one combination emerges as a clear winner, in the future it will:
- Exceed the performance of any of the other combinations with a 95% confidence or better.
- Deliver a conversion rate that falls within the estimated range with an 80% confidence.
Page Section Report Take-Aways
- If one variation emerges as a clear winner in its section, in the future it will exceed the performance of any other variation in that section (all other section content remaining the same) with a 95% confidence or better. It may, however underperform other variations if other section content is changed.
- A page section with a high relevance rating is likely to have greater impact on the page audience and changes in content will have a more volatile reaction on conversion rates.
| All About Statistics | Back to Top |
This section provides detailed explanations of the statistics used by Website Optimizer. This information is not necessary to design and conduct optimization experiments; it is provided for people interested in the statistics behind the Website Optimizer. It assumes that you have a basic understanding of mathematics and statistics.
Definitions
This section defines some common terms and explains how they are derived.
Conversion Rate
The measure of website success is conversion rate. The simplest definition is:
% conversion = No. of people who respond to your call to action/No. of people who saw your call to action x 100
For example, if 100 people see your website and 15 of those people click on the call-to-action button, you obtain a conversion rate of 15%.
The Website Optimizer measures conversion rate by page hits on the test and conversion pages. So,
% conversion = No. conversion page hits/No. test page hits X 100
Conversion rate is an exact value. It is the conversion rate that is actually delivered during the experiment.
Estimated Conversion Rate Range
While conversion rate indicates performance during the actual experiment, it is more meaningful to know the potential conversion rate outside the experiment environment, when the content in question would be presented to all visitors. The Website Optimizer predicts this future conversion rate and, since all predictions introduce a level of uncertainty, the future conversion rate is expressed as a range.
So if the experiment conversion rate is 35%, then the expected conversion rate range can be expressed as "35% plus or minus 2%" or "33 to 37%".
Percentage Improvement
Percentage improvement describes how well one combination performs over the original content. It is expressed as percentage improvement in conversion rate.
For example, if the original content has a conversion rate of 10% and Combination 5 has a conversion rate of 12%, then Combination 5 shows an improvement of 20%. The formula is:
% Improvement = [(Combo X conversion rate / Combo O conversion rate) -1] *100
Percentage improvement can be expressed as a positive or negative number.
Original Content
This is assumed to be the content displayed on the test page before the experiment began. Website Optimizer assumes that an experiment is testing potential new content against current content.
During the experiment set up, original content is supposed to be retained on the test page, while new content is entered into the Website Optimizer UI as variations. So technically, the "original" content is the content living on the test page. We do not recommend significantly changing the original content prior the experiment.
Relevance Rating
Relevance rating attempts to capture the impact of a page section. Some sections will carry greater weight with the audience than others; for example, it is a pretty safe bet that the quality of a product photo will have more impact than the colour of a headline.
Relevance rating is expressed as a scale of 0 to 5, with zero being no impact at all. Relevance is measured by the varied effectiveness of each variation in a section. For example, if a section has three variations and their conversion rates differ widely, the relevance rating for that section is likely to be high. In contrast, if the variations in a section all have similar conversion rates, the relevance rating will tend to be low.
High relevance ratings tend to be more significant than low relevance ratings. However, it is possible that the variations offered for a section are too similar to evoke radically different reactions. If you are specifically looking for section relevance, it is a good idea to use very different variations.
Duration
The following factors determine the actual length of time an experiment will probably need to run to get definitive results:
- Website traffic: The more traffic a site gets during the experiment, the faster the necessary number of test page hits occurs. (higher traffic => shorter duration)
- Percentage of user participation: Determines what percentage of your total traffic is included in the experiment. If you designate only 50% of your users to be shown experiment content, you effectively halve the website traffic. (lower participation=> longer duration)
- Current conversion rate: The more conversions that occur during the experiment, the faster enough data for meaningful results is collected. (higher conversion => shorter duration)
- Number of content combinations: The higher the number of possible combinations, the fewer visitors will be shown each combination in a given period of time. This means that more page hits will be needed to gain statistically significant data for each combination. In addition, with more combinations contending with each other, it will take longer for one combination to distinguish itself. (more combinations => longer duration)
- Actual improvement: If any new content results in a significant change in conversion rate (positive or negative) over the original, one combination will likely emerge as the clear winner in a shorter period of time. (greater improvement => shorter duration)
This estimate is primarily useful for determining the minimum amount of time needed for the experiment to run its course, that is, the time needed to collect a statistically significant volume of data, in both test page hits and conversions. Actual duration can depart drastically from the estimated duration.
A real experiment need only last until one combination emerges as a clear winner, which is defined as a combination with a conversion rate range that is higher than all others and does not overlap any other. Unexpected dynamics can lengthen an experiment. For example, if two or more combinations exhibit similar conversion rate ranges, their ranges may never stop overlapping.
Experiments including combinations with very high improvement are most likely to finish quickly. If you select variations that are very similar to the originals (or to each other), the estimated improvement will almost certainly be minimal; in contrast, choosing variations that differ radically from each other offer a better chance at achieving large differences in conversion rate.
Reports Details
This section expands on the information in the general Overview, containing detailed information on how each statistic is derived.
The Reports page offers a snapshot of the current status of the experiment during its progress as well as when it is finished. There are two types of reports, each providing a different perspective on the experiment results. The Page Sections report requires less data and will be available before the Combination report. We highly recommend, however, that you wait for the Combination report before drawing final conclusions.
- The Combination Report details conversions and impressions and estimates future conversion rate ranges for each combination displayed to visitors. With this report, shown in Figure 2, you can identify the combination of content that is most likely to result in the greatest conversion rate improvement.
One of the main assets of this report is that it captures the combined impact of new content. While the Page Sections report identifies the individual variations that are most effective, the Combinations report takes into account how combinations can work together (or not) to form a persuasive presentation.
Figure 2. Combination Report
- The Page Sections Report displays the same type of data as the Combination Report, but breaks it down to individual variations. With this report, shown in Figure 3, you can identify the variation for each section that performed best. As discussed, the best combination is not always the same as a collection of the best performing variations.
This report also identifies the sections that have had the greatest impact on the overall results of your experiment. This is called "relevance rating". It can tell you which sections are most effective in reaching your audience. Keep in mind, however, that a relevance rating is only valid for the variations included in the experiment.
Figure 3. Page Sections Report
Both reports contain a mix of actual and estimated data. The first three columns contain estimated data, while the last two present actual data collected during the experiment. See the FAQ for a description of each column.
Estimated Conversion Rate Ranges
Note: Although this section describes conversion rate ranges for combinations, the information is equally relevant for individual variations, as shown in the Page Sections report.
The most significant column in each report is the Estimated Conversion Rate Range. A conversion rate range predicts the conversion rate that a combination will likely deliver if it is adopted and displayed to all visitors (if no other changes are made). Because a prediction adds an element of uncertainty into the equation, estimated conversion rate is expressed as conversion rate with a confidence interval, such as 35% plus or minus 2% or a range of 33-37%. The mean value at the centre of the range is the actual conversion rate experienced during the experiment.
With the ranges displayed in bar graphs, the focus of this column is to compare the expected performance of all combinations. This emphasis reflects the focus of the experiment, which is not to pinpoint an accurate conversion rate (or improvement) for a combination, but to identify which of several possible combinations will bring better results than the others. The position and colour of each range bar provide a visual indicator of how a combination is performing relative to the original combination. The grey portion of a bar indicates where the range overlaps with that of the original content. Red and green portions indicate where the ranges fall higher or lower than the original.
With that in mind, the most significant factor to look for on the Results page is conversion rate ranges that exceed and do not overlap any other ranges. For combinations with overlapping ranges, there is always the chance that either combination might perform better if the experiment were run again. For combinations with ranges that exceed and do not overlap, it can be said with a 95% confidence level that the combination will perform better than the other combinations.
The two "Chance to Beat" columns indicate the likelihood that the combination will eventually exceed the conversion rate ranges of the original or all combinations. As shown in Figure 2, the "Chance to Beat" figures are reflected in the positioning of the ranges on the bar graph. For example, Combination 11, which does not overlap Original at all, shows a 99.0% chance to beat the original. It slightly overlaps two other combinations, which reflects that it has an 89.4% chance to beat all other combinations.
During the progress of the experiment, the conversion rate ranges will start out very wide and overlapping. Over time, as more test data is collected, the ranges will narrow and begin to move left or right relative to the original.
Calculating Estimated Conversion Rate
This section describes how an estimated conversion rate, with a plus/minus confidence interval, is arrived at.
The estimated conversion rate is derived from the collected data using Gaussian distribution. This is familiarly represented as a bell curve, as shown in Figure 3. Applied to an optimization experiment, Gaussian distribution says that the actual conversion rate of a combination will lie somewhere in the area below the bell curve. Depending on the flatness of the bell curve, there is a greater or lesser probability that the actual conversion rate will lie close to the mean value. Because a bell curve is symmetrical, the distance between each outer edge and the mean will always be equal.
Figure 4. Estimated Conversion Rate Bell Curve (click to enlarge)
As shown in Figure 4, the bell curve shape changes over time, it becomes less flat as more data is collected. At the beginning of the experiment, the curve is flat and there is an essentially equal probability that the conversion rate could lie anywhere between 0 and 100. As more data is collected, the curve steepens and outer values become less likely while values closer to the mean become more likely. Figure 4 illustrates the change in bell curve and bar graph over the course of ten days:
- Day 1: Little data collected, large confidence interval in estimate, very wide bar graph.
- Day 5: More data collected, smaller confidence interval in estimate, somewhat smaller bar graph.
- Day 10: Lots of data collected, conversion rate can now be estimated as a very small range.
True bell curves always approach zero (or infinity) but never reach it at the outer limits. This means that there is always the possibility, however remote, that the real conversion rate for a combination could be anywhere from 0 to 100. This view of the data is singularly useless, however. It is common practice to select a cut-off point to determine where the real conversion rate is likely to fall and this cut-off point is expressed as a confidence level. The Website Optimizer uses an 80% confidence level to estimate conversion rate range; that is, the real conversion rate delivered by a combination will fall within the estimated range 80% of the time.
The 80% confidence level is what determines the width of the range. It is what allows us to say, with 95% confidence, that a range that exceeds and does not overlap another range will outperform the other range. Statistically, this is a reasonable outcome. Requiring a higher confidence level for the range width lengthens experiment duration and at this point the trade off becomes very costly.
Viewing Conversion Rate Ranges Over Time
When viewing the reports as the experiment progresses, you will notice that the conversion rate ranges for each combination start out very wide and then narrow over time. As they narrow, each range becomes more defined and there is less overlap between combinations. This phenomenon is illustrated in Figure 5.
Figure 5. Conversion Rate Ranges Over Time
The key factor to look for is ranges with high conversion rates and no overlaps, such as the Combination 2 in Figure 4. By Day 10, Combination 2 has broken away from the pack and emerged a clear winner.
It would be easy to jump to conclusions when viewing the bar graph and assume that any range with a higher mean value than the others would return the best conversion rate. This is not necessarily the case. When viewing these ranges, it is wise to assume that the real conversion rate has a good chance of falling anywhere in the range. So, for example, when viewing the bar graph as shown in the graphic on Day 5, it would be a common mistake to assume that Combination 2 will be the winner; in fact, the large amount of overlap suggests a significant possibility that, if the experiment is run once more, the means of Combinations 0 and 2 would reverse themselves. In fact, this appears to be the case with the means of Combinations 0 and 1, which switch places as the experiment progresses.
Once a combination range no longer overlaps another range, it can be stated, with 95% accuracy, that the combination will perform better than the next lowest range.
