The Email Subject Line Tester Olympics: Comparing the Top Free Testing Tools

by Etienne Garbugli on February 6 • no comments.

✉️ Sign up for Updates

🥈

Who

🥇

Comes

🥉

First

If you do a lot of email marketing, you understand the importance of subject lines.

A great subject line can drastically widen your email marketing funnel. It can be the difference between 10 and 60% of your subscribers considering your offer.

The problem though is that, before you start testing, you won’t know for sure if your subject line will perform.

This is where subject line testing tools come in.

There are several subject line testing tools on the internets to help you pre-validate and benchmark your subject line copy, but… how do you know which one to trust?

Introducing The Email Subject Line Tester Olympics

I’ve been using the same email subject line tester for a few years now. It seems good, but is it really the best choice?

I decided to put it to the test by comparing it to the other free subject line testing tools out there.

For this, I give you… the email subject line tester Olympics: 😮

The Competitors

CoSchedule’s Email Subject Line Tester

CoSchedule is an all-in-one marketing management platform. Their email subject line tester is available both as a free tool, and as part of their management platform. From what I was able to see, it analyzes the capitalization, the word and character count, the use of words positively or negatively affecting the open rate, and whether or not the subject line contains numbers and emojis.

The Email Subject Line Tester Olympics – CoSchedule
CoSchedule – Subject Line Analysis

The Email Subject Line Grader by Net Atlantic

Net Atlantic is an email marketing agency. They have their own email marketing platform, and a subject line tester available for free. The Email Subject Line Grader analyzes word mixture and balance, word and character count, word types (power or action words), and makes subject line optimization recommendations.

Moosend’s Subject Line Tester

Moosend is an email marketing automation platform. Refine, their subject line testing tool, is available for free outside of their platform. Refine makes open rate predictions and compares them to industry averages. The tool looks for personalization, emojis, exclamation marks, brand names, and calculates character counts.

The Email Subject Line Tester Olympics – Moosend's Refine
Moosend – Industry & Subscriber Count Settings

Send Check It’s Email Subject Line Tester

Send Check It is Subject-Line-Testing-as-a-Service (I made that up, sorry 😉). Although they charge businesses for full email testing services and API access, their email subject line tester is free. Their tester evaluates subject line scannability, reading grade level, sentiment, capitalization, personalization, punctuation, and looks for specific spam words.

SubjectLine.com by Worldata

Worldata is a direct marketing agency. Their subject line tester is actually a lead magnet for their business. It analyzes subject line length and characters, punctuation, urgency, capitalization, and several words it judges as negatively-impacting open rates.

The Methodology

To win 🥇 at the email subject line tester Olympics, competitors need to match the ranking of pre-tested subject lines. The subject lines we’ll use as comparison were tested before on thousands of recipients for upsell.

They were selected out of 3 performance buckets (High, Average, Low) and ranked from best to worst-performing below:

High-Performing Subject Lines

  • Can I help? (60% open rate)
  • Hey < name >, I came up with something for you (55% open rate)
  • You’re VIP, So Let Us Save You a Little Money (54% open rate)
  • I know a better way to do that… (44% open rate)

Average-Performing Subject Lines

  • < name > – Are you ready to go Pro? (34% open rate)
  • Last Chance For Your VIP Subscription (31% open rate)
  • < name > – Are you ready to kick it up a notch? (28% open rate)
  • Hang on < name > – it’s gonna get crazy in here (28% open rate)

Under-Performing Subject Lines

  • Do you have time to waste? (22% open rate)
  • Go for Gold (15% open rate)
  • Congratulations. You’re about to jump ahead (10% open rate)
  • Hey < name >, dive in. The water is fine. (7% open rate)
Related Post:  Email Marketing Teardown – Pinterest

I expect the winning subject line tester to rank the subject lines more or less in the above performance buckets.

Obviously, there’s more to email opens than subject lines. The email subject line testing tools won’t take into consideration the preview text, the sender’s name, the sender’s address, the context, the relationship, the brand, and the trust built by the organization that sent the original emails.

Because of that, I don’t expect the results to be a perfect representation of reality.

Without further due…

Comparing The Results

CoSchedule

CoSchedule scored the subject lines from 63 to 39. It ranked them as follows:

  1. Congratulations. You’re about to jump ahead (10% open rate)
  2. You’re VIP, So Let Us Save You a Little Money (54% open rate)
  3. Last Chance For Your VIP Subscription (31% open rate)
  4. I know a better way to do that… (44% open rate)
  5. < name > – Are you ready to go Pro? (34% open rate)
  6. Do you have time to waste? (22% open rate)
  7. Go for Gold (15% open rate)
  8. Can I help? (60% open rate)
  9. Hang on < name > – it’s gonna get crazy in here (28% open rate)
  10. Hey < name >, I came up with something for you (55% open rate)
  11. < name > – Are you ready to kick it up a notch? (28% open rate)
  12. Hey < name >, dive in. The water is fine. (7% open rate)

The Email Subject Line Grader

Net Atlantic’s subject line tester scored the subject lines from 100 to 58. It ranked them as follows:

  1. Congratulations. You’re about to jump ahead (10% open rate)
  2. Do you have time to waste? (22% open rate)
  3. Can I help? (60% open rate)
  4. < name > – Are you ready to kick it up a notch? (28% open rate)
  5. Last Chance For Your VIP Subscription (31% open rate)
  6. Hey < name >, dive in. The water is fine. (7% open rate)
  7. < name > – Are you ready to go Pro? (34% open rate)
  8. You’re VIP, So Let Us Save You a Little Money (54% open rate)
  9. Hang on < name > – it’s gonna get crazy in here (28% open rate)
  10. Hey < name >, I came up with something for you (55% open rate)
  11. I know a better way to do that… (44% open rate)
  12. Go for Gold (15% open rate)

Moosend

Moosend’s subject line tester ranked our subject lines as follows. It gave them expected open rates between 34.94% and 25.39%:

  1. Congratulations. You’re about to jump ahead (10% open rate)
  2. Go for Gold (15% open rate)
  3. Hey < name >, dive in. The water is fine. (7% open rate)
  4. You’re VIP, So Let Us Save You a Little Money (54% open rate)
  5. Hey < name >, I came up with something for you (55% open rate)
  6. I know a better way to do that… (44% open rate)
  7. < name > – Are you ready to kick it up a notch? (28% open rate)
  8. Hang on < name > – it’s gonna get crazy in here (28% open rate)
  9. Last Chance For Your VIP Subscription (31% open rate)
  10. Can I help? (60% open rate)
  11. < name > – Are you ready to go Pro? (34% open rate)
  12. Do you have time to waste? (22% open rate)

Send Check It

Send Check It scored the subject lines from 99 to 79. It ranked them as follows:

  1. < name > – Are you ready to kick it up a notch? (28% open rate)
  2. < name > – Are you ready to go Pro? (34% open rate)
  3. Do you have time to waste? (22% open rate)
  4. Hey < name >, dive in. The water is fine. (7% open rate)
  5. Hey < name >, I came up with something for you (55% open rate)
  6. I know a better way to do that… (44% open rate)
  7. Can I help? (60% open rate)
  8. Go for Gold (15% open rate)
  9. Hang on < name > – it’s gonna get crazy in here (28% open rate)
  10. Last Chance For Your VIP Subscription (31% open rate)
  11. Congratulations. You’re about to jump ahead (10% open rate)
  12. You’re VIP, So Let Us Save You a Little Money (54% open rate)
Related Post:  Email Marketing Teardown – MailChimp

SubjectLine.com

Worldata’s subject line tester scored the subject lines from 88 to 47. It ranked them as follows:

  1. Do you have time to waste? (22% open rate)
  2. Last Chance For Your VIP Subscription (31% open rate)
  3. I know a better way to do that… (44% open rate)
  4. Can I help? (60% open rate)
  5. < name > – Are you ready to go Pro? (34% open rate)
  6. Go for Gold (15% open rate)
  7. Hey < name >, dive in. The water is fine. (7% open rate)
  8. Hang on < name > – it’s gonna get crazy in here (28% open rate)
  9. Hey < name >, I came up with something for you (55% open rate)
  10. < name > – Are you ready to kick it up a notch? (28% open rate)
  11. You’re VIP, So Let Us Save You a Little Money (54% open rate)
  12. Congratulations. You’re about to jump ahead (10% open rate)

As you can see above ☝️, the results significantly varied from one tool to the next. Several subject lines appeared both as high and low performers.

A Few Takeaways

  • None of the subject line testers came close to a 100% accuracy. The groupings didn’t line up for any of them, no matter how I looked at it;
  • The best email subject line tester in terms of categorization got 5 subject lines out of 12 right. On average, the other subject line testers got 4, so it’s not a big gap;
  • There were huge swings for the same subject lines; there weren’t a lot of clear/obvious patterns in terms of the analysis;
  • All of the email subject line testing tools got ‘Hey < name >, I came up with something for you’ wrong;
  • The testing tool that made the least big mis-categorizations (e.g. High-Performers as Under-Performers, or the reverse) got 2 out of 12 wrong. The average, however, was just one more (3);
  • Different tools seem to give different weight to question-based subject lines, personalization, capitalization, and emojis. The weighing factors seem to skew the results a bit;
  • One of the tools got almost all of the subject lines in reverse;
  • The more elements the subject line testers seem to analyze, the better the results seem to be.

So, Which Email Subject Line Tester Comes First?

To get 100% accuracy in results, we’d need to test a lot more than 12 subject lines. Ideally as well, the subject lines would have been pre-tested on the exact same segments with the same email header.

That said, and in spite of the fact that none of the email subject line testing tools gave perfect or near-perfect results, I’m ready to declare a winner…

The 3 best subject line testing tools are:

Although Send Check It missed some of the highest-performing subject lines, and doesn’t seem to like title-case subject lines, it was the best at identifying bad subject lines.

SubjectLine.com properly categorized more subject lines, but it made bigger mistakes while CoSchedule mis-categorized a lot of the subject lines.

The tools still have a long way to go in terms of giving accurate projections, but I would expect companies with a strong focus on email testing tools like Send Check It to make the most progress moving forward. Send Check It it is! 👍

Working in SaaS? I’m putting together a SaaS Email Marketing Masterclass. It launches in March 2019. Sign up today for early access.

Your Site Traffic, Conversions & Emails, Monitored for You

Highlights helps bloggers & marketers visualize, analyze and improve their content, emails and landing pages.

Etienne Garbugli
Étienne is a three-time startup founder (Flagback, HireVoice and Highlights), and the author of Lean B2B: Build Products Businesses Want. At Highlights, Étienne is responsible for customer success.

Leave a Reply

Your email address will not be published. Required fields are marked *