.avif)
TL;DR: A/B testing subject lines in Klaviyo sends two subject line versions to small slices of your list, measures which one earns more opens, then sends the winner to everyone else. It works best with at least 50 recipients per version and a 90% win probability. Because Apple Mail Privacy Protection inflates open rates, pair open-rate tests with clicks and revenue so you can trust the result.
A/B testing subject lines in Klaviyo is a split test that sends two subject line versions to slices of your list, then sends the higher-performing one to the rest, so Shopify brands can raise open rates with data instead of guesses. Most store owners know they should test. Few do it in a way that produces a real answer.
They test one subject line against another on a list of 300 people, see a two-point difference, and call it a win. That is not a win. That is noise.
The good news: Klaviyo builds the whole test for you. You pick two subject lines, set how much of your list sees the test, and Klaviyo picks the winner and sends it to everyone else. The hard part is not the buttons. It is knowing what counts as a real result, what to test, and which numbers you can still trust in 2026. This guide walks through the setup, the exact thresholds Klaviyo uses, and the open-rate trap that quietly ruins most tests.
What is A/B subject testing?
A subject line A/B test sends two versions of your subject line to separate groups of subscribers, tracks which one earns more opens, and sends the winner to the rest of your list. In Klaviyo, it is built into the campaign builder, so you never need a separate tool or a spreadsheet.
Your subject line is the first thing a subscriber sees. It decides whether your email gets opened or ignored. Small wording changes can move opens by double digits, and those extra opens turn into extra clicks and extra sales. Testing takes the guesswork out. Instead of arguing about which line sounds better, you let your real customers vote with their behavior.
Here is a simple example. You send version A, "New arrivals just dropped," to 20% of your list, and version B, "You'll want to see this before it sells out," to another 20%. Klaviyo watches the opens, declares a winner, and sends that line to the remaining 60%. You learned something real, and you never risked your whole list on a hunch.
How to set it up in Klaviyo
Klaviyo's A/B test lives inside the normal campaign builder. Here is the short version, based on Klaviyo's own campaign A/B testing guide:
- Go to Campaigns, click Create campaign, and choose Email.
- Pick the list or segment you want to send to.
- Write your first subject line and build the email.
- Above the subject line field, click Create A/B test. Klaviyo copies your email into a second, identical variation.
- Change only the subject line on variation B. Leave everything else the same.
- Choose your winning metric, your test size, and your test window.
- Send.
Name each variation something you'll recognize later, like "Free shipping vs 20% off." Klaviyo sends the two versions to your test group, watches the metric you picked, and then sends the winner to the rest. If you run a paid account, Klaviyo's Subject Line AI can even generate the second version for you, swapping in personalization, an emoji, or a shorter line.
If writing and testing campaigns every week is more than your team can handle, our Klaviyo campaign management handles the whole cycle for you.
What should you test?
Test one thing at a time. For subject lines, the highest-impact variables are length, personalization like a first name, emoji versus no emoji, a question versus a statement, and the offer itself. Change two things at once and you'll never know which one moved the number.
Klaviyo groups tests by the metric they affect. To lift opens, test the subject line, the preview text, and the from name. Klaviyo's benchmark research shows subject lines around seven words can reach open rates near 30%. Industry studies show personalized subject lines can lift opens by up to 26%, and emojis raise opens in many business-to-consumer tests, though not every audience responds the same way. That is the point of testing. What works for one store's audience can flop for another's. You can run the same tests inside flows, not just campaigns. Testing subject lines in your welcome and abandoned cart email automations often pays off more than campaign tests, because those emails send forever. For more starting points, see our other Klaviyo guides.
When is a test significant?
Klaviyo calls a campaign test statistically significant when at least 50 people have received each version and the win probability hits 90% or higher. Below that, the result is only "promising" or not reliable, and you should rerun it before you trust it.
Per Klaviyo's statistical significance guide, there are four buckets: significant at 90% or higher win probability, promising at 75 to 89%, not significant, and inconclusive. A test gets flagged not significant when the win probability stays under 75% even after 1,800 people received each version. The takeaway: sample size matters more than most people think. On the send side, you set what share of your list sees the test. A common setup gives 20% to version A and 20% to version B, then Klaviyo picks a winner, often after about six hours, and sends it to the remaining 60%. One more rule: don't retest near a holiday, when buying behavior is abnormal and the result won't repeat.
Do open rates still work?
Not on their own. Apple Mail Privacy Protection auto-opens emails for many subscribers, which inflates open rates and can hand your A/B test a false winner. If more than about 45% of your opens come from Apple Mail, open-rate tests need a higher bar or a different metric.
This is the part most guides skip. When an email lands in Apple Mail, Apple's servers pre-load your images, including Klaviyo's tracking pixel. Klaviyo records an "open" even if the person never looked. So a subject line test judged purely on opens might be measuring Apple's servers, not your customers. Klaviyo tags these with an "Apple Privacy Open" property you can filter out, by setting Apple Privacy Open equals false, to see truer opens. The safer move for high-stakes tests is to judge by clicks, placed orders, or revenue per recipient. Those reflect real behavior. Open rate is still a fine tiebreaker, but it should not be the only judge. A practical habit: run the test on open rate to pick a direction, then watch clicks and placed orders on the full send to confirm the winning line actually drove sales. Clean data also starts with a healthy list, which is why deliverability and testing go hand in hand.
Test more than the words
The biggest wins rarely come from swapping one word for another. They come from testing the angle. "Your cart is waiting" versus "10% off the thing you left behind" is a real test. "Hi" versus "Hey" is not. Test offers, urgency, curiosity, and value. Those move revenue, not just opens.
A few angle tests worth running: a discount versus free shipping, a benefit versus a deadline, a plain statement versus a curiosity gap, and a product name versus a problem it solves. Each one teaches you something about what your buyers actually respond to, and that lesson carries into every future email, not just the one you tested.
Two more rules keep your tests honest. First, small lists can't produce real answers. If fewer than 1,000 people will see each version, Klaviyo may just send to everyone, and even when the test runs, the result rarely reaches significance. Below that size, test the offer with your whole list over several sends and look for patterns instead. Second, don't test forever. Once a subject line pattern wins twice, use it and move on to a bigger question.
Across the Shopify stores we manage at CartStrings, the brands that treat testing as a habit, not a one-off, are the ones sitting near a 39% average open rate. A Klaviyo audit is usually where we find the quickest subject line wins.
Bringing it together
A/B testing subject lines in Klaviyo is simple to start and easy to get wrong. The platform handles the mechanics. Your job is to test one variable at a time, give each version enough people to hit 90% win probability, and stop trusting open rate as your only judge now that Apple Mail inflates it. Test the offer, not just the wording. Rerun promising results before you bet on them. And once a pattern wins, bank it and test something bigger.
Do that consistently and your open rates climb, your clicks follow, and your email revenue grows without sending more. If you'd rather hand the testing, writing, and analysis to a team that does this every day, book a call with CartStrings.
Frequently Asked Questions
What sample size do I need?
Klaviyo needs at least 50 recipients per version to call a test significant, but that is the floor, not the goal. For a reliable read on a subject line, aim for several hundred to a few thousand per version. If fewer than 1,800 people saw each version and the win probability is under 75%, Klaviyo flags the result as not significant.
How long should tests run?
It depends on your winning metric. Open-rate tests can settle in a few hours, and Klaviyo often picks a winner around the six-hour mark. Click and order tests need longer, usually 24 to 72 hours, because those actions take time. Don't end a test early just because one version jumps ahead in the first hour.
Can small lists A/B test?
Not reliably. If fewer than 1,000 people will see each version, Klaviyo may send the campaign to everyone instead of splitting it, and any result you get rarely reaches significance. With a small list, test the offer across several full sends and watch for repeatable patterns rather than trusting a single split test.
Test by open or click rate?
Use open rate as the winning metric for subject line tests, since the subject line's job is to earn the open. But because Apple Mail Privacy Protection inflates opens, confirm the winner with clicks or revenue before you roll it out widely. If open rate and click rate disagree, trust the click.
How many lines can I test?
For a clean, fast answer, test two subject lines head to head. Klaviyo supports more variations, but every extra version splits your audience thinner and needs a bigger list to reach significance. Start with two, learn what wins, then test a new idea against the winner next time.
