A/B testing
The A/B Test page compares VIBE with Shopify native search. During a test, storefront sessions are split between the two variants.
What the test compares
Section titled “What the test compares”A test runs two variants in parallel:
- VIBE Search (Variant A) gives shoppers the full VIBE search experience.
- Shopify Search (Variant B) gives shoppers your theme’s native search instead. For this group, the VIBE widget is turned off, so they use your store’s built-in search exactly as it would behave without VIBE.
Both groups are measured on the same metrics, so you can compare them directly.
How the traffic splits
Section titled “How the traffic splits”While a test runs, VIBE assigns each visitor session to one of the two groups, aiming for an even 50/50 split. The assignment is made per session and stays consistent for that session, so a shopper never flips between VIBE and native search mid-visit. Because the split depends on who arrives during the test, the two groups are rarely exactly 50.0/50.0; the Sessions percentage on each card shows the actual split you collected.
Start a test
Section titled “Start a test”Open the A/B Test page. If no test is running and you are not in the cooldown period, you see a Start A/B Test button. Select it, and VIBE confirms that 50% of traffic will use Shopify native search for comparison. The page then shows a Running badge and starts collecting data. You can have only one test running at a time.
While a test runs
Section titled “While a test runs”The page shows:
- Searches collected toward the 4,000 target, and the day counter.
- Estimated finish date, based on your store’s current search rate.
- Split health, confirming the 50/50 split is within expected range.
- The two variant cards, VIBE and Shopify, with their live metrics.
How the test ends
Section titled “How the test ends”The test ends automatically when it collects 4,000 searches (it always runs at least 7 days), or at 14 days, whichever comes first. When it ends, VIBE returns all traffic to VIBE Search, saves a permanent results snapshot, and emails you a summary. You can also stop the test early at any time using Stop test.
A completed test shows the Test period, why it ended, whether the sample was large enough for a reliable verdict, both variant summaries, and when you can run another test. Select View full results for the detailed comparison.
Stop a test early
Section titled “Stop a test early”Select Stop test at any time. VIBE confirms that all traffic now uses VIBE Search. The control group is turned off and every shopper returns to the VIBE experience. Stopping a test early begins the 90-day cooldown before you can run another.
Cooldown
Section titled “Cooldown”After a test ends (automatically or early), there is a 90-day wait before you can start another. During the cooldown, the page shows how many days remain instead of the Start button. The cooldown prevents rapid stop-and-restart, which would split traffic unevenly and make the comparison unreliable.
Reading the results
Section titled “Reading the results”While the test runs, the results page shows live progress. After the test ends, it shows the final snapshot.
After the test ends:
- Verdict: which variant won (shown only when the sample is statistically reliable).
- Projected monthly revenue impact (shown only when the sample is reliable).
- Funnel comparison: add-to-cart rate and conversion rate side by side.
- Full metric table: every tracked metric for VIBE Search, Shopify Native, and the Difference:
| Metric | What it means |
|---|---|
| Sessions | Distinct sessions in the group during the test. |
| Searches | Searches performed by the group. |
| Searches per session | Average searches per session. |
| Click-through rate | Share of searches that led to a result click. |
| Add to cart rate | Share of searches that led to an add-to-cart. |
| Conversions | Converting result clicks attributed to the group. |
| Conversion rate | Converted clicks as a share of the group’s searches. |
| Revenue | Revenue attributed to the group’s converted clicks. |
| Revenue per session | Average attributed revenue per session. |
| Products per session | Average number of distinct products viewed in a session. |
| Average time to view | Average time before a shopper opens a product. |
| Viewed products | Number of distinct products viewed by the group. |
If the sample is not large enough to be reliable, the verdict and revenue impact are withheld and the page explains why. If native search wins, review your Explore controls and Merchandising rules before concluding.
Test status
Section titled “Test status”The badge tells you where a test stands: Not started (none has run yet), Running (collecting data now), or Completed (finished, with results to review). A completed test’s results stay visible during the cooldown.
Test recommendations
Section titled “Test recommendations”- Let the test run to completion when possible. It ends at 4,000 searches or 14 days.
- Avoid major changes during a test. Catalog, control, or rule changes affect only the VIBE variant and reduce comparability.
- If native search wins, review your Explore controls and Merchandising rules before concluding, since those shape how VIBE ranks and surfaces products.
Troubleshooting
Section titled “Troubleshooting”Some shoppers see the old Shopify search during a test. Is that a bug? No, that is the test working. While a test runs, about half of shoppers are in the control group and see your theme’s native search by design, so the two can be compared. To preview VIBE yourself regardless of group, use Search Preview, or stop the test to put everyone back on VIBE.
When does the test end automatically? When it collects 4,000 searches (minimum 7 days), or at 14 days, whichever comes first.
When can I trust the result? The verdict and projected revenue impact are only shown when the sample is statistically reliable. If the sample is too small, the page says so.
Why can’t I start a test? You can run only one test at a time, and there is a 90-day cooldown after a test ends. During the cooldown the page shows how many days remain instead of the Start button.