Quick wins for a faster PC:
Scan for outdated or missing drivers - takes under a minuteDriver Scan →Repair Windows errors before they cause bigger problemsFix Now →To run a useful email A/B test, compare versions that differ in one variable, randomly assign eligible recipients, choose the success metric before sending, and decide in advance how much of the audience to test and how long to run it. Treat the result as a winner only when the platform’s criteria and the evidence support that decision; otherwise, record it as inconclusive.
1. Define the question and the decision
Start with a specific hypothesis that connects a change to an outcome, and decide what you will do with the result. For example: “For this audience, does a benefit-focused subject line increase qualified clicks compared with our current subject line?” This is an example, not a reported result.
As an Amazon Associate I earn from qualifying purchases.
Write down the audience, the variable, the primary metric, and the action you will take if one version performs better. Setting these before the campaign goes out helps prevent choosing a favorable metric after seeing the results.
2. Change one variable at a time
Common variables include the subject line, sender name, email content, and send time. For a subject-line test, keep the sender, message, offer, audience rules, and timing the same. For a send-time test, keep the content and subject line equivalent so that timing is the main difference. Klaviyo advises changing one factor at a time and keeping other content unchanged when testing a content element (Klaviyo’s A/B testing instructions); Mailchimp also lists subject line, From name, content, and send time as test variables (Mailchimp’s A/B test guide).
#1 Best Overall
Testing several changes at once can tell you which complete version did better, but it cannot reliably show which change caused the difference. If you need to learn which individual element matters, isolate it in a separate test.
3. Choose a metric that matches the campaign goal
Pick one primary outcome before sending. The best measure is the one closest to the decision you need to make—not simply the metric that is easiest to improve.
- Subject line, preview text, or sender identity: Klaviyo recommends open rate for these tests. If the real goal is engagement or sales, consider whether clicks or a downstream conversion measure would answer the business question better.
- Email body, call to action, layout, or offer: Click rate can show which version prompts more interaction. For a purchase-focused question, use conversion or placed-order outcomes if your account and test type support them and the measurement window is appropriate.
- Send time: Compare equivalent versions at different times, then use the result to inform future scheduling.
Open rate is not interchangeable with clicks or sales. Klaviyo warns that Apple Mail Privacy Protection can prefetch tracking pixels and inflate recorded opens. If many opens come from Apple Mail, interpret open-based results cautiously; Klaviyo suggests using a custom report with an MPP property in that situation. See its campaign testing guidance for the metric and reporting details.
Free tools Windows power users keep installed
One-click scans. No signup required.
4. Set up the audience, variations, and test size
Use the same eligible campaign audience for each version and make the variations easy to identify. Both Mailchimp and Klaviyo describe dividing audience members among test variations; Mailchimp describes random selection. Keep audience rules, offer, and other campaign conditions consistent unless one of those is the variable being tested.
If your platform supports a staged test, decide what share of recipients should receive each version and how long the test should run before selecting a winner. Klaviyo lets users adjust test size and duration; when a test uses less than the full audience, the remaining recipients can receive the selected winner. Check your account’s current feature and metric eligibility before relying on a particular setup, because platform access can depend on plan or account conditions.
There is no universal list size that guarantees a useful result. The sample needed depends on factors such as the baseline event rate, the smallest lift worth acting on, the number of variations, and the error tolerance you are willing to accept. The cited platform guidance does not provide a single sample-size formula suitable for every campaign.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.5. Send the test and evaluate uncertainty
After sending, use the platform’s result categories as guidance about the strength of evidence, not as a universal guarantee that a test was adequately powered. Klaviyo’s published rule tags a campaign result as statistically significant when each variation has at least 50 recipients and the win probability is at least 90%. These are Klaviyo-specific product criteria, not general statistical thresholds for all platforms or all testing situations (Klaviyo’s explanation of campaign significance).
The Tool Desk
Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →Outbyte PC Repair FREERepair Windows errors before they cause bigger problemsFix Now →Klaviyo also distinguishes significant, promising, not significant, and inconclusive results. A promising result is not the same as a confirmed winner. If the evidence is inconclusive, record that honestly and consider repeating the test when it is useful. Do not keep extending or reinterpreting a test merely to obtain a preferred outcome.
6. Apply the result appropriately
For a staged test that covers only part of the audience, follow the platform’s process for sending the chosen winner to recipients who have not yet received a version. Mailchimp describes this approach for its subject-line workflow, and Klaviyo describes sending a selected winner to recipients who have not received the under-100% test (Mailchimp’s subject-line guidance; Klaviyo’s campaign testing instructions).
If the test already went to the full audience, the result cannot change that past send. Use what you learned to inform later campaigns, while keeping other conditions comparable enough to make the learning useful.
Platform features to verify before launch
Mailchimp says A/B test access depends on plan, while Klaviyo notes that some metrics and features have eligibility constraints. Confirm the current options in your account before preparing the campaign; product access and labels can change. Documentation is available from Mailchimp and Klaviyo.
Recommended Free Tools
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
