---
title: "A/B testing: definition, method and example"
url: "https://coldmeetings.com/glossary/a-b-testing.md"
canonical_url: "https://coldmeetings.com/glossary/a-b-testing"
last_updated: "2026-10-19"
type: "conceptual"
summary: "A/B testing sends two versions of one message to equal groups to learn which works better. See how to run one fairly and read the result."
related: ["https://coldmeetings.com/glossary/call-to-action.md", "https://coldmeetings.com/glossary/reply-rate.md", "https://coldmeetings.com/glossary/cold-email.md"]
---

# A/B testing: definition, method and example

> **For AI agents:** the complete index of ColdMeetings pages is [llms.txt](https://coldmeetings.com/llms.txt). Every page is also served as markdown at its URL plus .md.

A/B testing sends two versions of one message to equal groups to learn which works better. See how to run one fairly and read the result.

**What is A/B testing?**

A/B testing sends two versions of a message, differing in one element, to similar groups at the same time to see which version gets the better result. It replaces opinion with a comparison you can read.

Group: 3. Outreach.

## The parts

1. **One change**: Vary only one thing, such as the subject line or the closing ask. If two things differ you cannot tell which one mattered.
2. **Equal, random groups**: Split the list so both groups look alike. Sending one version to your best prospects spoils the comparison.
3. **Same time**: Send both versions together, so the day or hour is not the reason for a difference.
4. **A result you can judge**: Choose the measure in advance, such as reply rate, and wait for enough messages before deciding.

## Worked example: two subject lines at a bookkeeping firm

Alder and Finch, a fictional bookkeeping firm, tests two subject lines on 150 delivered messages each. The body of the email is identical.

1. Version A: 6 replies from 150. Reply rate: 4%.
2. Version B: 9 replies from 150. Reply rate: 6%.
3. Difference: 2 percentage points in favor of B, which is 3 replies out of 150.
4. With so few replies, the gap could be chance. The firm repeats the test on a fresh group before changing its template.

Reply rate by subject line (invented figures)

- Version A: 4%
- Version B: 6%

## How to apply it

1. Pick one element to test and write the question the test should answer.
2. Split the list randomly into two equal groups.
3. Send both versions at the same time and note the measure you will judge by.
4. Repeat on a new group before you make the winner your standard.

## Common mistakes

- Changing the subject line and the body together, so the cause of a difference is unknown.
- Declaring a winner from a handful of replies.
- Testing endlessly instead of keeping the winner and moving to the next question.

## Related terms

- [Call to action](https://coldmeetings.com/glossary/call-to-action.md): A call to action (CTA) is the one step you ask the reader to take next, such as replying, picking a time or confirming a detail.
- [Reply rate](https://coldmeetings.com/glossary/reply-rate.md): Reply rate is the portion of delivered emails answered by the recipient.
- [Cold email](https://coldmeetings.com/glossary/cold-email.md): A cold email is a first message to a person you have not spoken with, sent because their role or company suggests you could help.

## Keep reading

- [Cold email subject lines](https://coldmeetings.com/blog/cold-email-subject-lines.md)
- [Cold email metrics](https://coldmeetings.com/blog/cold-email-metrics.md)
- [Sequences](https://coldmeetings.com/product/sequences.md)

## Questions

### How many messages does an A/B test need?

More is better. Small groups swing widely by chance, so treat a result from a few dozen replies as a hint and confirm it with a second round.

### What should I test first?

Start with the element most likely to change the outcome, often the opening line or the ask.

### Can I test more than two versions?

Yes, but each extra version needs its own group, so you need a longer list for the same confidence.

### Which measure should I use?

Use the one closest to your goal. For outreach that is usually reply rate or meetings booked, not opens.

Questions about outbound for your own company? Email hello@coldmeetings.com.

## Related

- [Call to action: definition and example](https://coldmeetings.com/glossary/call-to-action.md)
- [Reply rate: definition, formula and example](https://coldmeetings.com/glossary/reply-rate.md)
- [Cold email: definition, parts and example](https://coldmeetings.com/glossary/cold-email.md)