The Most Misunderstood Statistic In Email Delivery

Will the real complaint rate please stand up?

The spam complaint rate is one of the most critical statistics in email delivery. Unfortunately, it’s also one of the most misunderstood.

It can happen if your software is measuring your complaint rate differently than the ISPs.

You see a low complaint rate reported in your email software, and think, “Great, people love my email. Almost no complaints!” Usually, people are struggling with an Inbox delivery problem and think, “it can’t be because of my complaints; the complaint rate is low.” The shocking thing is that your complaint rate is low because of the poor Inbox delivery: your subscribers are not getting your email so they can’t complain.

Your software & the Internet Service Providers (ISP)

Most email sending software shows you this calculation:

While the ISPs care about this calculation:

What’s the Difference?

Difference #1: The Denominator

The first big difference is what the ISPs use as the denominator versus what most email software uses as the denominator. The ISPs use the number of messages delivered to the Inbox as the basis for their spam complaint calculations while often email sending software uses the larger number of messages sent that didn’t bounce.

The ISPs care about the percentage of their users who complained about emails in their Inbox. When an email is delivered to a user’s Spam folder, it can’t be complained about, so it’s not factored into the ISP’s complaint ratio. Because of this, it’s possible to deliver most of your email to the Spam folder and still have a low overall complaint rate.

Consider these two senders:

Both senders have the same overall spam complaint rate of 0.1%, but sender B has a complaint problem while sender A does not.

Sender B’s actual Inbox spam complaint rate is very high, at 0.8%. This problem is masked in the overall spam complaint rate by the email delivered to the Spam folder where it can’t generate complaints.

So we often hear an email marketer say, “I have low complaints – less than 0.1% – so complaints can’t be my problem”. Yet they have a high Inbox spam complaint rate.

Difference #2: Individual ISPs

The other difference between how ISPs and most software look at complaint rates is that each ISP only cares about the spam complaint rate for its users, but most software shows one aggregate number for all ISPs.

We suggest finding software that solves this by showing the overall spam complaint rate broken down by each ISP.

Here’s an example:

Let’s say you send a campaign where:

Your total messages successfully sent (not bounced) = 195,390
Your total spam complaints = 304

Depending on your software, you might see the overall spam complaint rate as:

Overall complaint rate = 304 = 0.16%

Which, if this were an Inbox spam complaint rate, it would be pretty good! However, your real problem is being masked here.

As a GreenArrow user, you’ll see we break down the complaint information by domain. We can also provide the context of the Inbox monitoring data (with GreenArrow Monitor); you can begin to discern the issues within the ISPs.

In GreenArrow Engine, you’ll see this:

Which tells you this:

At AOL: We saw all of the seed messages from GreenArrow Monitor go into the Inbox, so likely all of the email went into the Inbox, and the complaint ratio listed is the true Inbox complaint ratio or close to it. If some messages were delivered to the Spam folder, then the Inbox complaint ratio would be even higher than 0.7%. Because 0.7% is high for an Inbox complaint ratio, we know that there is a complaint problem.
At Hotmail: We saw none of the seed messages to GreenArrow Monitor go to the Inbox and also no complaints. This likely means all mail is going to the Spam folder, and nobody has the chance to complain. They also don’t have the chance to buy or respond to your call-to-action. So the complaint rate of 0.0% only confirms that Hotmail is putting all or most of the email into the Spam folder.
At Yahoo: We saw about half of the seed messages to GreenArrow Monitor go to the Inbox. This means that the Inbox complaint ratio is greater than 0.21%.

We don’t know exactly how much greater; it could be a factor of 1.5 or a factor of 10. If exactly half of the messages went to the Spam folder, then the Inbox complaint ratio would be 0.42%. Our team doesn’t recommend calculating the math like this, as GreenArrow Monitor data is an indicator of how much of the mail went to the Inbox, not an exact measurement.

On this data alone, we would be suspicious of a complaint problem. But we don’t need to know precisely, because the results from AOL have given us enough information to know that there is a complaint problem.

At other domains: There were 83,749 messages sent to domain names not listed in the above report, and they did not generate any complaints. Most of these are likely at ISPs that don’t support feedback loops. The lack of feedback loops contributes to the dilution of the overall complaint ratio and makes it diverge from the Inbox complaint ratio.

Avoid the Trap, Know Your Data

Reviewing and understanding all available data can help avoid the trap of underestimating your complaint rates and put you on the track to better Inbox deliverability. What data do you use to manage your email performance?

Thanks to Tina Doppler, Matt Rideout, Jeff Anderson, and Nicole Schmied for reading and contributing to drafts of this post.