In this guide, we'll outline the information you'll need in order to design your first experiment. Then we'll walk through the different steps you'll take to set up an experiment using the Website Optimizer tool. After reading this guide, you should have a good idea of the basics of testing with Website Optimizer and be ready to begin your first experiment.
You might find it helpful to print out this guide as a reference when completing steps within the tool. Alternatively, if you prefer a video walkthrough of the experiment set-up, you may wish to view our Website Optimizer User Guide (Flash - English only), which covers much of the same process.
Before you use the Website Optimizer tool, you'll need to spend some time designing the scope of your experiment. These are the critical decisions you and your team will need to make:
Choose your test page. This is the page that you'll be optimizing by making changes with Website Optimizer. Your test page needs to offer an action that your visitor can take, like a purchase, download, or sign-up. This action can often be in the form of a link to another page on your site.
For your first experiment, we suggest you choose a test page that receives a high volume of traffic, such as your landing page, so you can see meaningful results quickly. You might wish to pick a page that you drive traffic to using AdWords, or a page that has a low conversion rate which you'd like to increase.
Choose your conversion page. This is the page that represents business results for you - whether it's the page where a user makes a purchase, fills out an interest form, or downloads a white paper. Learn more.
If you chose a test page with a link to another page on your site, choosing your conversion page is easy: it's the page a visitor sees when they click on the test page link. If the test page has more than one link, pick the one likely to get the most traffic. As with the test page, you will need to enter the URL for this page later.
For this first experiment, it's not quite so important what the link does -- whether it takes the visitor to a sign-up page, a product specification, runs some script, or some other action. In future experiments, though, the conversion you choose to track should define how you measure the success of your test page.
Choose what type of test is right for you. Website Optimizer offers two types of tests: A/B and multivariate. A/B tests compare the performance of two entirely different pages, so you can try out different layouts, move around sections of the page, or change the overall look and feel of a page. These tests are simpler and take less time to obtain results. Multivariate tests, on the other hand, are more flexible and robust. A multivariate test allows you to test content variations in different sections of your page simultaneously. For example, you could try out two different headlines, three different images, and two different product descriptions.
Select which content you want to test. Throughout this guide, we'll be working with a multivariate example experiment using a test page headline and an image as our page sections. Pinpoint those sections on your test page; you'll need to identify them specifically later in the experiment set-up process. You can find some suggestions on what to test in our FAQ or Best Practices. If you're doing an A/B test, you won't need to choose specific content.
| Create the content variations you want to test. If you're doing a multivariate test, once you've identified the headline and image you'll be testing, you need to come up with at least one variation for each. During the experiment, visitors will see either the original content or the new variation. This is so the experiment can determine which variation of the content leads to more people taking the desired action and reaching your conversion page. |  |
Try to come up with variations that are significantly different from the original content -- experiments where the variation and original are very similar tend to get inconclusive results. For example:
Original headline: Welcome!
Variation headline: You are entering...the Red Zone
Original graphic: <a picture of a smiling child>
Variation graphic: <an abstract design>
Notice that the originals are unassuming, generic. The variations are more intriguing, even provocative. Visitors will likely have very different reactions to each. The goal of the experiment is to find out which reaction is most advantageous to you.
If you're performing an A/B test, you'll need to create the alternate versions of your test page. Because you create the alternate versions, you have complete flexibility when it comes to what you'd like to try out. During the experiment, visitors will see either your original test page or one of the alternate variations you've created. Website Optimizer will measure the performance of each version of the page to determine which variation leads to more people taking the desired action and reaching your conversion page.
Decide how much of your traffic you want to include in your experiment. When the experiment goes live, visitors to your website will see either the original content or the new variations. You can choose to limit the percentage of visitors who will see the new variations. Keep in mind, however, that by limiting the percentage of visitors participating in the experiment, you are also lengthening the amount of time it will take for the experiment to give you meaningful results. So if your test page and conversion page get limited traffic, you will likely want to include 100% of your visitors.
Now that you've made the crucial decisions for your experiment, you can access the Website Optimizer tool and start creating it.
Starting your first A/B experiment
This section of the guide covers designing and implementing an A/B experiment. If you're creating a multivariate experiment, review the multivariate section of the guide.
- Step 1: Identify experiment pages
- Step 2: Add tags to experiment pages
- Step 3: Preview and start experiment
- View your reports
- Stop the experiment
Before you can do anything, you'll need to access Website Optimizer. You can do so by following the steps below.
- If you're using Website Optimizer within AdWords, sign in to your AdWords account and visit the Campaign Management tab, then choose the Website Optimizer subtab. If you're using Website Optimizer through the standalone site, just sign in there.
- If this is your first time accessing the tool, you'll see an opening page with links to some useful resources. Click the Get started button to continue.
- You'll reach the Experiment List page, which displays a summary of all your experiments. If this is your first experiment, your list will be empty. Click Create new experiment to proceed with setting up your experiment. Since we'll be creating an A/B experiment in this section of the guide, click the Create link next to A/B experiment.
The Website Optimizer tool presents a four-step work flow for setting up a new experiment. As you finish each step, this page will track your progress and display basic information about the new experiment. To begin, click on the Step 1 "Continue" button.
Step 1: Identify experiment pages
You need to supply three pieces of information here:
Experiment name: For your first experiment, names aren't quite so important, so just enter something simple like "My first experiment". For future experiments, you'll want to choose something more meaningful that will help you easily distinguish between all your experiments on the Experiment List page.
Test pages: Enter the URL of the original test page you chose during Step 0. As shown in the example on the Step 1 page, the URL you enter should not contain any information after the page's file name (such as index.htm or productpage.html). If you include query parameters, they will be ignored. Please also enter the URLs for each alternate page you have created. Remember, each alternate page you create, needs to be saved at a unique URL in order to be used in an A/B test.
Conversion page: Enter the URL of the conversion page you chose during Step 0. Again, the URL should not have any extra information after the page's file name.
When you're finished entering this information, you're ready to move on. When you click "Continue", Website Optimizer will try to validate the URLs you entered. If you leave either the test page or conversion page field blank, or if Website Optimizer got an error trying to access either page, it will generate an error message. In most cases, public web pages will not cause any validation problems; if it is, however, you will be given the option of continuing the experiment setup without URL validation.
Once Website Optimizer successfully validates your page URLs, it'll send you back to the experiment work flow page. As you will see, Step 1 has been successfully checked off and your test and conversion pages are listed. This work flow page is an important tool to help you track your progress through the setup and keep track of scope of your experiment. Before you launch the experiment, you will be able to revise any of the setup information from the work flow page.
Step 2: Add tags to experiment pages
Who will install the tags?
| The first decision you need to make in this step is whether you'll install the HTML script tags or whether you'll hand that task off to someone else. This task needs to be done by someone familiar with HTML and with the structure of your webpages. If you choose to hand the task off, you'll need to communicate some specific information to that person. |  |
Regardless of who installs the tags, Website Optimizer provides specific script to be installed, as well as detailed instructions available in our Installation Guide. Be aware that this script sets a cookie, so you may wish to ensure that your site's privacy policy covers the setting of cookies.
Install the tags
If someone else will install the tags, Website Optimizer will give you a link to that you'll need to send to the person who'll be installing the tags. Stay in touch with the colleague who's installing the tags. They'll need you to provide the following information:
- Exactly which elements on the test page will be varied. They need to mark these elements with "page section" tags.
- A page section name for each element. Later, these names will help you identify the element in question. For this experiment, the names "Headline" and "Image" should be sufficient.
Make sure your colleague notifies you once they've installed the tags.
If you will install the tags, follow the Installation Guide, which provides complete instructions. That guide goes into more detail, but we've given you a brief overview of the task below.
There are two types of script that needs to installed on your pages. Although it looks complicated, each set of script serves a purpose:
- First, there's the control script. Among other things, the control script makes sure that the experiment variations are switched randomly and that all variations are displayed an equal number of times. For this experiment (and in most cases), place the control script immediately after the <head> tag. You'll need to install the control script just on your original test page. For this experiment (and in most cases), place the tracking script immediately before the closing </body> tag in each page.
- The second set of script is the tracking script. It ensures that visits to both the test page and the conversion page are tracked by Website Optimizer for the experiment. For this experiment (and in most cases), place the tracking script immediately before the closing </body> tag in each page. You'll need to add the tracking script to your original test page, each of the alternate variation pages you've created, and your conversion page.
Validate tag installation.
Once you've installed all the tags, validate the tags within Website Optimizer. Our validation tool examines your pages and verifies that the tags have been installed properly. If the validation tool detects any problems with the installation, you'll need to fix these before continuing. Website Optimizer won't let you go on to the next step without validation.
The validation tool will check for the following potential errors, which you will need to fix before continuing:
- Was any javascript found on the test page?
- Were any page sections defined on the test page? The tool looks for beginning and ending tags.
- Was tracking code found on the test page?
- Was tracking code found on the conversion page?
There are two methods of validating your pages. The first is to provide the URLs for your test and conversion pages. If your pages are externally visible, Website Optimizer will access them and provide any errors as described above.
Alternatively, if Website Optimizer can't access the pages on your live site, you can also upload HTML source files for your test and conversion pages. This is helpful if your pages are part of a purchase process, behind a login, or inaccessible for some other reason. Simply save the HTML source of your pages, upload the files, and Website Optimizer will validate them.
If you're unable to validate the tags using either method -- for example, if you're using a dynamic webpage that Website Optimizer cannot view -- you can opt out of the validation step. However, please be sure that the tags have been installed correctly on your own if this is the case.
Once your pages pass validation, select the Tags have been added to the pages option, and click Continue to finish this step.
Step 3: Preview and start experiment
At this point, the hard work is done. You've tagged your pages and created new variations. Now it's time to set the experiment off and running. But before you do, make one final check of your experiment settings. Once you start the experiment, you won't be able to change the parameters of the experiment, so be certain they are as you intend.
If you do find you need to update your settings, though, click the "Back" button to return to the experiment work flow page, and click the "view or edit" link on Step 1. Keep in mind that if you change the page URLs now, you will need to go back through Step 2, reinstall the code on the new pages, and re-validate.
Percentage traffic. Remember earlier in the experiment preparation you were supposed to decide how much of your traffic to include in the experiment? Here's where you enter that value. The more traffic is included, the faster the experiment will run.
Variations. Here you get one last chance to preview the alternate page variations that will be displayed to visitors during the experiment. If you need to change any variation, return to the experiment work flow page (click the "Back" button) and click the "view or edit" link.
Ready, set, start your experiment!
Once you click "Start", you'll return to the experiment work flow page. Step 3 is checked off, and now there's an additional section describing the progress of this experiment, including estimated duration and the number of impressions and conversions tracked during the experiment. Though your test page will start showing different variations immediately, there's a delay of about an hour before your reports will begin displaying data. The progress of the experiment, and the estimated duration, depends entirely on the amount of traffic seen on your test and conversion pages.
Once you start seeing some impressions and conversions recorded, the reports will start to have some preliminary results. Click "View report" to see the experiment results.
Once the experiment is launched, impressions and conversions will be recorded almost immediately. You can view this data on the reports page as soon as it's recorded. Plan to check that impressions and conversions are being recorded soon after starting; there are some tagging errors that are not revealed until an experiment is actually running. If you're not getting any impressions or conversions, check the troubleshooting guide for some suggestions on what might be causing this error.
Until a minimum amount of data has been collected, you'll likely get a message like: "We have not gathered enough data yet to produce a complete report with analysis. Please check back in a day or so."
This message may be accompanied by a table/graph combination showing the performance for the individual performance of each variation, showing which variations are most effective. For more information on how to read this report, see What do my reports mean? If you love numbers and want to learn specifics about how we derive results, see the Technical Overview.
As the experiment progresses, the Website Optimizer will identify a "winning" variation. How long this takes depends mainly on how much traffic the experiment pages receive, as well as how similar the variations are. For example, if your experiment has run beyond the estimated duration, it's getting lots of impressions and conversions, but no variation is emerging as a clear winner, it's likely that visitors are not experiencing strong reactions to any of the alternate variations you created. It's probably time to stop the experiment and try some bolder variations.
Starting your first multivariate experiment
This section of the guide covers designing and implementing a multivariate experiment. If you're creating an A/B experiment, review the A/B section of the guide.
- Step 1: Identify experiment pages
- Step 1a: Test page section planning
- Step 2: Add tags to experiment pages
- Step 3: Create variations
- Step 4: Review experiment settings and launch
- View your reports
- Stop the experiment
Before you can do anything, you'll need to access Website Optimizer. You can do so by following the steps below.
- If you're using Website Optimizer within AdWords, sign in to your AdWords account and visit the Campaign Management tab, then choose the Website Optimizer subtab. If you're using Website Optimizer through the standalone site, just sign in there.
- If this is your first time accessing the tool, you'll see an opening page with links to some useful resources. Click the Get started button to continue.
- You'll reach the Experiment List page, which displays a summary of all your experiments. If this is your first experiment, your list will be empty. Click Create new experiment. On the resulting page, since we'll be creating a multivariate experiment in this section of the guide, click the Create link next to multivariate experiment to proceed with setting up your experiment.
The Website Optimizer tool presents a four-step work flow for setting up a new experiment. As you finish each step, this page will track your progress and display basic information about the new experiment. To begin, click on the Step 1 "Go" button.
Step 1: Identify experiment pages
You need to supply three pieces of information here:
Experiment name: For your first experiment, names aren't quite so important, so just enter something simple like "My first experiment". For future experiments, you'll want to choose something more meaningful that will help you easily distinguish between all your experiments on the Experiment List page.
Test page: Enter the URL of the test page you chose during Step 0. As shown in the example on the Step 1 page, the URL you enter should not contain any information after the page's file name (such as index.htm or productpage.html). If you include query parameters, they will be ignored.
Conversion page: Enter the URL of the conversion page you chose during Step 0. Again, the URL should not have any extra information after the page's file name.
When you're finished entering this information, you're ready to move on. When you click "Continue", Website Optimizer will try to validate the URLs you entered. If you leave either the test page or conversion page field blank, or if Website Optimizer got an error trying to access either page, it will generate an error message. In most cases, public web pages will not cause any validation problems; if it is, however, you will be given the option of continuing the experiment setup without URL validation.
Once Website Optimizer successfully validates your page URLs, it'll send you back to the experiment work flow page. As you will see, Step 1 has been successfully checked off and your test and conversion pages are listed. This work flow page is an important tool to help you track your progress through the setup and keep track of scope of your experiment. Before you launch the experiment, you will be able to revise any of the setup information from the work flow page.
This step is a reminder for your technical and marketing teams to collaborate and determine what to content to experiment with on your test page. If you've been following this guide, you should be well prepared to move on to Step 2. If not, take some time to plan your test page sections, as this page prompts.
When you're ready to move on to Step 2, click Go.
Step 2: Add tags to experiment pages
Who will install the tags?
| The first decision you need to make in this step is whether you'll install the HTML script tags or whether you'll hand that task off to someone else. This task needs to be done by someone familiar with HTML and with the structure of your webpages. If you choose to hand the task off, you'll need to communicate some specific information to that person. | |
Regardless of who installs the tags, Website Optimizer provides specific script to be installed, as well as detailed instructions available in our Installation Guide. Be aware that this script sets a cookie, so you may wish to ensure that your site's privacy policy covers the setting of cookies.
Install the tags
If someone else will install the tags, Website Optimizer will give you a link to that you'll need to send to the person who'll be installing the tags. Stay in touch with the colleague who's installing the tags. They'll need you to provide the following information:
- Exactly which elements on the test page will be varied. They need to mark these elements with "page section" tags.
- A page section name for each element. Later, these names will help you identify the element in question. For this experiment, the names "Headline" and "Image" should be sufficient.
Make sure your colleague notifies you once they've installed the tags.
If you will install the tags, follow the Installation Guide, which provides complete instructions. That guide goes into more detail, but we've given you a brief overview of the task below.
There are three types of script that needs to installed on your test page, and one script to be installed on your conversion page. Although it looks complicated, each set of script serves a purpose:
- The first script, to be installed in the head of your test page, is called the control script. Among other things, this script makes sure that the experiment content is switched randomly and that all variations are displayed an equal number of times. For this experiment (and in most cases), place the control script immediately after the tag.
- The second script is used to define page sections for the elements that will be changed during the experiment. Essentially, you need to use the script provided by Website Optimizer to tag the beginning and end of each page section; you also need to name each page section. As an example, let's say you've defined two elements: the headline and the image. Your HTML code for the "Headline" page section should look like this (using example content):
Before:
<h1>Welcome!</h1>
After:
<h1>
<script>utmx_section("Headline")</script>
Welcome!
</noscript>
</h1>
The original content, the "Welcome!" headline, remains in the HTML code. Later you will be entering your new variation content in the Website Optimizer tool. (Notice that we left the heading 1 tags (<h1>) outside the script. For this experiment, we are not varying the HTML tags, only the words of the headline; so there's no reason to include the heading tags.)
Your HTML code for the "Image" page section should look like this (using example content):
Before: <img src="/images/smiling_child.jpg">
After: <script>utmx_section("Image")</script>
<img src="/images/smiling_child.jpg">
</noscript> - The third script is tracking code; it ensures that visits to both the test page and the conversion page are tracked by Website Optimizer for the experiment. For this experiment (and in most cases), place the tracking script immediately before the closing </body> tag in each page.
Validate tag installation.
Once you've installed all the tags, validate the tags within Website Optimizer. Our validation tool examines your pages and verifies that the tags have been installed properly. If the validation tool detects any problems with the installation, you'll need to fix these before continuing. Website Optimizer won't let you go on to the next step without validation.
The validation tool will check for the following potential errors, which you will need to fix before continuing:
- Was any javascript found on the test page?
- Were any page sections defined on the test page? The tool looks for beginning and ending tags.
- Was tracking code found on the test page?
- Was tracking code found on the conversion page?
There are two methods of validating your pages. The first is to provide the URLs for your test and conversion pages. If your pages are externally visible, Website Optimizer will access them and provide any errors as described above.
Alternatively, if Website Optimizer can't access the pages on your live site, you can also upload HTML source files for your test and conversion pages. This is helpful if your pages are part of a purchase process, behind a login, or inaccessible for some other reason. Simply save the HTML source of your pages, upload the files, and Website Optimizer will validate them.
If you're unable to validate the tags using either method -- for example, if you're using a dynamic webpage that Website Optimizer cannot view -- you can opt out of the validation step. However, please be sure that the tags have been installed correctly on your own if this is the case.
Once your pages pass validation, select the Tags have been added to the pages option, and click Continue to finish this step.
You've prepared your webpages and tagged your page sections. Now it's time to tell Website Optimizer what new content variations you want to try out in your experiment.
In this example, you've tagged two page sections, named "Headline" and "Image". For each section, click "Original" to view the original content (in a non-editable format). For section 1, the original content for the headline should be displayed. For section 2, the original content for the image should be displayed.
Add new variations
Now you're ready to add new variations.
- After the Section 1 label, click the link "New variation".
- In the pop-up window, enter the variation name "Headline". This name should match up to the name in the page section tags.
- In the variation content window, the original content is displayed (now in an editable format). Edit the content as designed in Step 0 to create a new Headline variation.
- Click Save to save the new Section 1 variation.
- After the Section 2 label, click the link "New variation".
- Repeat steps 2 through 4 to create and save a new "Image" variation.
Notice that as you add new variations, the number of "Total Combinations" increases. The combination is an important concept in Website Optimizer, and is described extensively in the Overview. Essentially, when your test page is displayed during an experiment, Website Optimizer is testing the performance of not only individual variations, but also the combined effect of all page sections on the page. So in this experiment with two pages sections and 1 variation for each section, the following four combinations will be tested:
- original headline + original image
- original headline + new image
- new headline + original image
- new headline + new image
Preview your combinations
Website Optimizer offers a handy tool for previewing the variations you just entered. Using this tool you can each of the potential combinations that your visitors might see.
- Click the Preview button. A new window opens, displaying your test page with all the original content. In a frame above your page is the preview control bar.
- Use the Prev/Next links in the control bar to scroll through all the combinations (in this experiment, there should be 4) and ensure that the variation content is being properly displayed. Alternatively, view a specific combination by selecting a variation for each page section in the drop-down menus.
- When you're finished previewing, click the Close button to close the Preview window.
Once you've created the two new variations (one for Headline and one for Image) and previewed the resulting combinations, it's time to move on. Click "Save and Continue".
Returning to the experiment work flow page, Step 3 is now checked off. Notice that the step displays a list of all your page sections, the number of variations defined for each section, and the total number of combinations.
Step 4: Review experiment settings and launch
At this point, the hard work is done. You've tagged your pages and created new variations. Now it's time to set the experiment off and running. But before you do, make one final check of your experiment settings. Once you start the experiment, you won't be able to change the parameters of the experiment, so be certain they are as you intend.
If you do find you need to update your settings, though, click the "Back" button to return to the experiment work flow page, and click the "view or edit" link on Step 1. Keep in mind that if you change the page URLs now, you will need to go back through Step 2, reinstall the code on the new pages, and re-validate.
Percentage traffic. Remember in Step 1a you were supposed to decide how much of your traffic to include in the experiment? Here's where you enter that value. The more traffic is included, the faster the experiment will run.
Variations and combinations. Here you get one last chance to preview the combinations that will be displayed to visitors during the experiment. If you need to change any variation, return to the experiment work flow page (click the "Back" button) and click the "view or edit" link on Step 3.
Ready, set, start your experiment!
Once you click "Start", you'll return to the experiment work flow page. Step 4 is checked off, and now there's an additional section describing the progress of this experiment, including estimated duration and the number of impressions and conversions tracked during the experiment. Though your test page will start showing different combinations immediately, there's a delay of about an hour before your reports will begin displaying data. The progress of the experiment, and the estimated duration, depends entirely on the amount of traffic seen on your test and conversion pages.
Once you start seeing some impressions and conversions recorded, the reports will start to have some preliminary results. Click "View report" to see the experiment results.
Once the experiment is launched, impressions and conversions will be recorded almost immediately. You can view this data on the reports page as soon as it's recorded. Plan to check that impressions and conversions are being recorded soon after starting; there are some tagging errors that are not revealed until an experiment is actually running. If you're not getting any impressions or conversions, check the troubleshooting guide for some suggestions on what might be causing this error.
Until a minimum amount of data has been collected, you'll likely get a message like: "We have not gathered enough data yet to produce a complete report with analysis. Please check back in a day or so."
This message may be accompanied by a table/graph combination showing the performance for the variations in each page section. This is called the Page Section report. It reports on the individual performance of each variation, showing which variations are most effective. For more information on how to read the Page Section report, see What do my reports mean? If you love numbers and want to learn specifics about how we derive results, see the Technical Overview.
The page section report is generally the first report available, since it requires less data to draw definitive conclusions. It's worth waiting for the Combinations reports, which takes a little longer but is more meaningful, because it assesses how individual variations work together to create an overall effect.
As the experiment progresses, the Website Optimizer will identify a "winning" combination. How long this takes depends mainly on how much traffic the experiment pages receive, as well as how similar the variations are. For example, if your experiment has run beyond the estimated duration, it's getting lots of impressions and conversions, but no combination is emerging as a clear winner, it's likely that visitors are not experiencing strong reactions to any of the combinations. It's probably time to stop the experiment and try some bolder variations.
You can temporarily stop your experiment from displaying combinations and collecting data by clicking the Pause link on your Experiment Page.
If you'd like to stop the experiment entirely, you can do so at any time, regardless of whether or not Website Optimizer has identified a clear winner. The experiment will continue as long as you want it to.
