Introduction

The StoreFront Visual Builder allows you to create server-side A/B tests (experiments) directly within your StoreFront content. This means you can use the same drag-and-drop tools and layout elements you already use in Visual Builder to design, duplicate, and test content. The experiment engine is powered by UltraCart Analytics, providing advanced testing capabilities at no additional cost.

Experiments help you test variations of content, layouts, or entire pages to determine what performs best with your customers.

Tip: Use descriptive variation labels (e.g., “CTA Button — Green” or “Hero Header — New Tagline”) instead of generic ones like “Variation 1.” This ensures your reporting remains meaningful even months later.

Features

• Fully integrated with StoreFront Visual Builder

• Server-side rendering of test content

• Optimization across multiple performance metrics

• Automatic traffic shifting toward the winning variation

• Automatic cleanup and promotion of the winner

• Detailed analytics and performance statistics

• Search engine and internal IP traffic automatically filtered out

• Free for all StoreFront users

Types of Experiments

The UltraCart StoreFront Experiments system supports three experiment types:

1. On-Page Content Experiments – Test small content changes such as CTA buttons, product images, or headlines.

2. URL Experiments – Test performance between entire landing pages or funnels using different URLs.

3. Theme Experiments – Test entire StoreFront themes (layouts, checkout flow, etc.) against each other.Experiment - On Page Content

On-Page Content Experiments

Adding an Experiment

1. Open the Visual Builder and navigate to the page section where you want to insert an experiment.

   Open

   

   Next, open the hierarchy sidepanel.

2. Click the + icon and search for Experiment in the element type search box:
   
   

   Open

   

3. Select the Experiment element to insert it into your hierarchy.

4. When the Experiment settings sidepanel opens, go ahead and follow these steps to fill out the name, duration days, objective metric, optimization type, and optionally the notes. DO NOT CLICK START EXPERIMENT YET!.  When you are done click on the collapsed Hierarchy side panel as shown below in step 6.
   
   

   Open

   

   
   Complete the experiment settings in the side panel:

   • Name

   • Duration (in days)

   • Objective metric

   • Optimization type

   • Optional notes

Prerequisite: Do not click Start Experiment yet. You’ll first need to create and label your variations.

On the hierarchy panel click the plus button on the Experiment as shown below.

Open

This will create our first Experiment Variation element.  Give the variation a name and select "Original" for the Experiment Variation.  DO NOT CLICK PROMOTE CONTENT!   When finished click back to the hierarchy.

Open

Click the plus button on the Experiment object again to add the second variation.  Now that we select Experiment Variation as "1" instead of Original this time.

Important Note about the Importance of Descriptive Variation Naming

While the example uses generic naming of variations, such as ‘Variation 1 content’, you will want to give each variation a descriptive name, so that you can easily identify which variation is which.

After the experiment ends, the variations will be removed from the VB experiment element. For example, the following experiment that has ended, the Graph & Statistics for the experiment appears like this:

Open

As you can see, you will benefit from labelling each variation descriptively!

Open

At this point we have an Experiment with two variations that will be tested.  The hierarchy looks like this:

Open

Continue adding content below each variation.  In this example we are testing two different headlines so the hierarchy looks like this:

Open

When we have the content at the desired state, click on the settings icon for the Experiment.  Click the Start Experiment and once the Experiment ID is shown click the Save button on your page.

Open

Experiment - URL

If you have two different URLs (such as landing pages) that you want to test against each other the URL test is good solution.  Under the StoreFront → Content → Experiments click on the "Add URL Experiment" button as shown below.

Open

This will open a dialog where you configure the details about the experiment.  You need to specify at least two URLs to test against.  These URLs should be on your StoreFront like the example below.

Open

Once the experiment is created the system will give you a router URL like the example above.  This URL is what you should target traffic to (such as from your PPC advertising).  The router will automatically split the traffic to the landing pages according to the configuration.

Open

Experiment - Theme

A theme experiment allows you to test one StoreFront theme against another.  This test can be useful if you have two entirely different checkout styles that you want to test against each other.  To get started with a theme experiment click on the "Add Theme Experiment" button as shown below:

Open

Once you click the button a dialog will appear that will allow you to configure the experiment.  At least two themes must be selected.  

Open

Only one theme experiment can be active at a time.  If you try to create a new experiment before ending the previous one the system will give you an error message.

Viewing the Results

The progress and results for your Experiments are visible under StoreFront → Content → Experiments.  

Open

Each experiment can be expanded to show additional details.

Open

Manually Promotion of an Experiment Variation

You can manually promote an experiment from the Storefront Experiments by expanding the experiment (click the ‘carrot’ that appears along the right side of the experiment:

Open

Then scroll down to the experiment variations, then click the blue ‘promote’ button for the variation you wish to promote:

Open

How does UltraCart report the experiment session data?

StoreFronts utilizes UltraCart Analytics engine to track the session event data.  This stream of data is processed with an approximate 45 minute lag.

1. The content is potentially visible to the customer (i.e. if you're testing something mobile then a desktop user is not going to report anything)

2. The content has actually seen the content.  If you're testing something in the footer and the customer never scrolls down then we won't report anything.

Both of these are critically import points that lead to the experiment data being highly accurate.

We will also omit traffic that originates from the following sources:

1. Search engine crawlers

2. Traffic that originates from IP addresses that are logging into your UltraCart account.  We don't want users in your call center or organization to skew the results of your experiments.

NOTE: Orders containing a test credit card number are also excluded.

How does the traffic split between each variation?

Traffic starts out with a equal split between the variations. 

If equal weighting is set to No:

Over time traffic will shift towards the winner.  Adjustments are made once a day to the traffic allocation until by the end of the experiment the winner has 100% of the traffic.

If equal weighting is set to Yes:

Each variation receives the same percentage of traffic throughout the entire test.  At the end of the experiment the winner receives 100% of the traffic.

What happens when an experiment finishes?

UltraCart will automatically learn of the winning variation, promote that variation's content and clean up the rest of your template automatically.  The automated email will contain a .CJSON attachment with a complete copy of the experiment hierarchy for archival purposes.  Below is an example of the email that you will receive.

Open

This allows you to create experiments in a single working session, have them run, and clean themselves up.  There is no manual cleanup process to engage in.  Think of experiments like fire and forget cruise missiles that always hit their intended target.

What if I have existing content that I want to test?

We recommend using the wrap tool located in the hierarchy.  When you wrap the existing content with an Experiment element the system will automatically make the existing content the original variation.  You can then duplicate the original experiment variation to variation 1, 2, 3, etc. and make edits to that content.  This working pattern allows you to quickly test minor changes to large blocks of content without having to recreate everything.

How do I force a particular variation to show?

Make sure that you are logged into UltraCart.  If you're not seeing the admin bar at the top of your StoreFront then you need to login first to make this work.

Each variation in your experiment has a number from 0 (original), 1 (first variation), 2 (second variation), etc.  If you pass the parameter expvid on the number with the number you want to view it will pull that particular variation during the page render.  Example

http://www.mysite.com/?expvid=1

If you look at the Developer Tools → Server Log there will be entries in the log labeled "EXPERIMENT" which will indicate that the variation specified by expvid was chosen.

Frequently Asked Questions

Question: We'd like to configure an experiment, but only want the original content to initially display while our team tests the experiment variation, how can we do that?
Answer:  The experiment engine operates at any point in the visual builder hierarchy.  If you're going to test two different checkouts then it would be:

• container (checkout-single)

  • experiment

    • experimentvariation (0 = Original) 

      • All the content for checkout 1

    • experimentvariation (1 = Variant)

      • All the content for checkout 2

If the experiment has not been started, then the original variation will always be displayed.  However, your logged in users (your testing team) can appending &expvid=1 to the URL see the new experiment section, even though it's not yet active to customers.  (Once the experiment is started then it would begin running at a 50/50 display rate.