How to A/B Test Emails in Klaviyo to Improve Open Rates and Revenue

TL;DR: A/B testing in Klaviyo lets you pit two versions of a campaign or flow email against each other so the winner gets sent to the rest of your list. Test one variable at a time (usually subject lines first), wait for statistical significance at 90% win probability, then roll the winner into future sends. Done well, this can lift open rates by double digits and add real revenue to your Shopify store.

Most Shopify brands send emails based on gut feel. A subject line "sounds good," a send time "feels right," a button color "looks fine." Then the campaign goes out, the open rate is mediocre, and nobody knows why.

A/B testing fixes that. It replaces guesswork with evidence by sending two versions of the same email to small slices of your list, measuring which one performs better, and then sending the winner to everyone else. Marketers who A/B test achieve 83% higher ROI than those who don't, yet only 39% of email marketers regularly test their subject lines. That gap is one of the biggest unclaimed wins in email marketing today.

If you're using Klaviyo on Shopify, the tools are already built in. This guide walks you through how to A/B test emails in Klaviyo for both campaigns and flows, what to test first, and how to read results without fooling yourself.

What does an A/B test in Klaviyo actually do?

An A/B test in Klaviyo splits your audience into groups, sends a different version of the email to each group, then automatically (or manually) declares a winner based on the metric you choose. For campaigns, Klaviyo sends each variation to a small percentage of your list (commonly 20% each), waits for results, then sends the winning variation to the rest.

So if you have 50,000 recipients and run a subject line test with 20/20 split, 10,000 people get version A, 10,000 get version B, and Klaviyo decides a winner roughly six hours later. The remaining 30,000 get the winner automatically. According to Klaviyo's documentation, you can test subject lines, message content, or send times for campaigns, and subject lines or message content for flow emails.

The mechanics are simple. The discipline is harder.

What you should A/B test in Klaviyo first

Not every variable is worth your time. Some tests move the needle. Others waste sends and teach you nothing.

Start with subject lines. They drive open rates, and open rates gate everything else. Teams that systematically test subject lines see 23% higher average open rates over 90 days compared to teams that pick one and never vary it. Adding personalization to a subject line alone can increase opens by 26%.

After subject lines, move to these high-leverage variables:

• Preview text. Often ignored, but it acts as a second subject line in the inbox.
• CTA button copy. Tiny change, big swing. Adding a single focused CTA to one campaign boosted clicks by 371% and sales by 1,617% in one tested example.
• Email length. Short and punchy vs. long and story-driven. The right answer depends on your brand and audience.
• Hero image vs. plain text. Image-heavy emails sometimes lose to clean, conversational copy.
• Offer framing. "$10 off" vs. "10% off" vs. "Free shipping" often produces very different click and conversion rates.
• Send time. Run this only after you've tested content, because audience and content effects can swamp timing signals.

If you're picking your first test, start with subject lines on your next promotional email campaign. It's the lowest-effort change with the highest immediate visibility.

How do you A/B test an email campaign in Klaviyo?

Open the campaign in Klaviyo, click the email under Content, then in the right sidebar click "Create A/B test." You'll see fields for two variations where you can change either the subject line or the message content (not both at once). Configure your winning metric and test size, schedule the campaign, and Klaviyo handles the rest.

Here's the click path in detail:

1. Go to Campaigns and create a new campaign or open a draft.
2. Build your first email as normal.
3. In the Content section, hover over the email and click Create A/B test.
4. In Variation B, edit either the subject line or the email body (one at a time).
5. Open Test settings at the bottom. Set your test size (Klaviyo recommends 20% per variation as a starting point, leaving 60% to receive the winner).
6. Choose your winning metric: Open rate, Click rate, or Placed order rate. For subject line tests, use open rate. For content or CTA tests, use click rate or placed order rate.
7. Set the test duration. The default is six hours, which is plenty for most Shopify lists over 20,000 subscribers.
8. Schedule and send.

When the test wraps, Klaviyo picks the winner automatically and sends it to the remaining 60% of your list. You can also override by clicking Choose as winner before the timer ends, though that's rarely a good idea.

How do you A/B test a flow email in Klaviyo?

Open the flow, click the email step you want to test, then click Create A/B test in the right sidebar. You'll see Variation A and B side by side. Edit the subject line OR the email content in Variation B, set the winning metric (only open rate or click rate are available for flow tests), then turn the test live. Klaviyo serves both variations to incoming triggers and declares a winner once the statistical thresholds are met.

Flow A/B tests work differently from campaigns because flows trickle. New subscribers, abandoned carts, and post-purchase triggers come in over days and weeks. There's no fixed timer. Instead, Klaviyo's flow A/B test logic requires at least 500 recipients per variation and a 90% win probability before declaring a statistically significant winner. Until then, both variations keep running.

This is where flow testing earns its keep. Flow emails generate nearly 41% of email revenue from just 5.3% of sends. A 10% lift on your abandoned cart open rate compounds month after month with zero added work. Across the Shopify stores we manage at CartStrings, optimized email automations routinely outperform campaigns by 3x on click rates.

If you want to test timing inside a flow (say, "send the first reminder at 4 hours vs. 6 hours after abandonment"), you can't use the built-in A/B test feature. Instead, drop a conditional split before the time delay, set it to a 50% random sample, and compare downstream metrics manually.

How do you know if your A/B test result is statistically significant?

Klaviyo considers a campaign A/B test statistically significant when the win probability reaches 90% or higher. For flows, the bar is the same 90% plus a minimum of 500 recipients per variation. If your test ends below those thresholds, Klaviyo will flag the result as "not statistically significant," which means the difference you're seeing could be random noise rather than a real effect.

Win probability is the math behind the headline. It tells you the chance that the "winning" variation actually beat the other one, accounting for sample size and natural variance. A subject line that wins 52% to 48% with 200 recipients per variation is basically a coin flip. The same 52/48 split with 20,000 recipients per side is almost certainly real.

If you want a rough planning number: most subject line tests need at least 1,000 recipients per variation to detect meaningful differences, and content or conversion tests often need 10,000 or more. To calculate precisely, you need four inputs: your baseline metric, the minimum lift you want to detect, your statistical power (usually 80%), and significance level (usually 95%). At a 2% baseline conversion rate and 30% target lift, you need roughly 9,800 profiles per variation.

The takeaway: if you have a list under 5,000 active subscribers, focus on subject line tests for opens (where the bar is lower) and stop trying to call winners on click or conversion tests that will never reach significance.

Common A/B testing mistakes that quietly waste your sends

Plenty of brands "do A/B testing" but never improve. Almost always, it traces back to one of these errors.

Testing more than one variable at a time. If you change the subject line and the hero image, you have no way to know which one drove the lift. Klaviyo's own guidance is explicit: test one variable per round.

Ending tests too early. A variation that's "winning" at hour two often regresses by hour six. Let the test run its full duration, and don't pause it mid-flight to "fix" something.

Ignoring seasonality and external factors. Black Friday traffic can mask underperformance. A summer sale email tested against a holiday email isn't a fair fight. Run tests inside comparable windows.

Forgetting to document. If you don't log the hypothesis, the result, and the takeaway, you'll forget what you learned. Three months later you'll be retesting the same thing. Keep a simple spreadsheet with date, test, variations, winner, and lift.

Calling a winner with too small a sample. A 60% win probability isn't a win. It's noise. Wait for 90% or extend the test, or accept the test was inconclusive and move on.

Turning test results into real revenue

A single winning subject line is worth a one-time bump. A culture of testing compounds.

Bake A/B testing into your sending rhythm. Pick one variable to test on every campaign and at least one flow email per month. Roll winners into your default templates so the next campaign starts from a higher baseline. Treat your top-performing subject line formulas as a library, not a one-off.

This is also where audit work pays off. If you've never reviewed your flows, the biggest wins aren't in any single test, they're in the structural decisions: are you sending the right offer, to the right segment, at the right step? A focused Klaviyo account audit often surfaces 5 to 10 high-impact tests before you've optimized a single line of copy.

The bottom line

A/B testing in Klaviyo isn't a productivity hack. It's how you separate what you believe about your customers from what's actually true. Start with subject lines, run one variable at a time, wait for 90% win probability, and document everything. Within a quarter you'll have a library of proven patterns and a real lift in open rates and revenue.

If you're not sure where to start or your tests keep coming back inconclusive, that's usually a signal of bigger problems with segmentation, list size, or flow setup. We help Shopify brands fix that every day at CartStrings, including running structured testing programs that average 32% email-attributed revenue. Book a call if you want a second set of eyes on your Klaviyo account.

Frequently Asked Questions

What's the minimum list size for A/B testing in Klaviyo?

There's no hard minimum, but subject line tests need around 1,000 recipients per variation to detect meaningful differences. Content and conversion tests usually need 5,000 to 10,000 per variation. If your list is under 5,000 active subscribers, stick to subject line tests and accept that some results will be inconclusive.

How long should a Klaviyo A/B test run?

For campaigns, six hours is Klaviyo's default and works well for lists over 20,000. Smaller lists may need 12 to 24 hours. For flows, the test runs until Klaviyo collects at least 500 recipients per variation and reaches 90% win probability, which can take days or weeks depending on flow volume.

Can I A/B test send times in a Klaviyo flow?

Not with the built-in A/B test feature, which only supports subject line and content testing for flows. To test timing inside a flow, drop a conditional split before the time delay, configure a 50% random sample, and compare downstream open, click, and conversion rates manually.

What's the best metric to choose for an A/B test?

Match the metric to the variable. Subject line and preview text tests should use open rate. Email content, CTA copy, or button design tests should use click rate. Offer tests (discount type or amount) should use placed order rate when sample size allows.

Why does Klaviyo say my A/B test isn't statistically significant?

Klaviyo flags results as not significant when the win probability is below 90% or the sample size is too small. The two fixes are running the test longer to gather more data, or accepting that the variations are likely too similar to produce a measurable difference, and trying a bolder change next time.