What to A/B Test First?
FERMÀT's Behavior-Led Approach for Ecommerce

Traditional A/B testing starts with a guess. FERMÀT starts with what shoppers actually did.

Why do most A/B tests fail to move revenue?

In most programs, someone believes a bigger button or a new hero image will lift conversion. A designer mocks it up, engineering builds it into the next release, and a few weeks later the test reads out.

Three problems are built into that process:

1. The hypothesis comes from opinion, so many tests measure ideas that were never tied to what shoppers struggled with.
2. Every variant waits on design and engineering, so most teams run a handful of tests a quarter.
3. The result is an average across all traffic. A layout that helps first-time mobile visitors can lose with returning buyers, and the blended number hides both.

When a test wins, you learn what won. You rarely learn why.

How does FERMÀT approach A/B testing with AI?

FERMÀT starts from shopper behavior. Every session is mapped into plain-language actions, so agents can spot patterns like a cohort that opens the size guide three times and then leaves. That pattern becomes the hypothesis.

From there, the loop runs in three steps:

→ Detect: agents surface the pattern and the revenue tied to it.

→ Recommend: agents propose the next best change, ranked by expected revenue lift.

→ Generate: with no engineering ticket, an agent builds the new experience for that behavioral cohort from the brand's guidelines. The team reviews the draft before anything goes live.

The new experience is tested on live traffic against the brand's current site, and the result feeds back into detection.

What does that change in practice?

Each test is aimed at a specific shopper problem, so the result explains itself. If the cohort that kept checking fit starts converting, you know why.

Speed changes too. An apparel brand put FERMÀT-built experiences against its own site in nine head-to-head experiments on live traffic, and FERMÀT won every one. The top result nearly doubled conversion rate (+98%), and several others landed between +50% and +97% at 100% confidence.

Does statistical rigor still matter?

Yes. FERMÀT experiments run as controlled splits, with enough traffic and run time to reach significance. What changes is where hypotheses come from and how quickly a variant gets built.

The takeaway

The next phase of testing will be judged on two things: how good the hypotheses are, and how quickly they become live experiences.

It's worth asking your team how many of last quarter's tests started from something a shopper actually did vs. a hypothesis based on an opinion.