How ISP Rate Limiting Is Ruining Your Relationship

We need to talk.

I think we need some space. I like you, but I need time to process our relationship. 

If Internet Service Providers (ISPs) like Gmail and Hotmail could talk, sometimes they might sound just like this. There are plenty of reasons why they might put on the brakes and begin to limit the rate at which they accept email (i.e., rate limiting). But why do the ISPs rate limit senders? And how can you respond without landing in the digital dog house? 

Here we’ll explore two specific rate limiting techniques used by ISPs and what they look like from a sender’s perspective. Consider it a kind of relationship therapy in which you will get to know your ISPs better. Let’s get started, and remember—this is a judgment-free zone.

It’s Not You; It’s Me—Unless It’s You

Rate limiting can take a number of forms and happen for many reasons, so let’s begin with what happens when a sender tries to deliver an SMTP email to an ISP’s mail server. There are four basic possible outcomes:

1. The server could accept the message. Hooray! Delivery successful.
2. The server could defer the delivery attempt. This means, “I can’t accept this message now, but you’re welcome to try again later.” This is by far the most frequently used rate limiting mechanism.
3. The server could bounce the message. This means, “I can’t accept this message. You should not try sending it again in the future.” Ouch.
4. The server could refuse to establish the SMTP connection in the first place. This is often implemented through a firewall rule and normally won’t happen unless there’s blacklisting in place, or the ISP is experiencing technical issues on their end.

Of course, not all deferrals, bounces, and refusals are the result of rate limiting. There are a lot of potential reasons for an ISP to defer or bounce a delivery attempt, so how can you determine what your ISPs are telling you? The SMTP reply and/or enhanced status codes they send can provide valuable insight into what the issues are. GreenArrow’s Statistics also does an excellent job of classifying each bounce and deferral and telling you what it means. (See box.)

The Upside-Down Economics of Email

But why do the ISPs rate limit in the first place? To understand that, let’s look at how the costs of email delivery differ from snail mail (aka postal mail).

With snail mail, the cost is paid primarily by the sender. For example, think of your favorite magazine subscription. It might cost an advertiser $1.00 (or more) to print and mail each paper magazine. And while the U.S. Postal Service spends time and resources delivering the mail, this cost is largely offset by the sender’s purchase of postage.

However, the costs of email are paid primarily by the receiver—in this case, the ISP. It might cost an email sender a few hundredths of a penny to send each individual email, especially at higher volumes. But each delivery uses an ISP’s system resources (bandwidth, disk space, etc.), and unlike the Postal Service, there are no stamps to help offset said cost.

This economy of scale is so advantageous to senders that many attempt to take advantage of it. Business models that would never be viable via postal mail, including bulk spam and certain types of scams, become profitable when email economics are applied. And as long as the cost of sending stays low, spammers will remain incentivized to continue their nefarious ways.

Now, most of us don’t like spam, and that includes the ISPs who end up on the losing side economically if spammers drive their users away. So what can be done? Rate limiting; that’s what.

(Reader beware: To get a better idea about Rate Limiting, we, unfortunately, have to take a better look at the darker side of email: Spamming. The following sections may not be suitable for those readers who have a particular aversion to spam, spammer tactics, or who have been previously traumatized by spam. )

Tactic #1: Greylisting

One of the simplest yet most effective tactics used by ISPs use to reduce the amount of spam that they and their customers receive is called greylisting, and even the most reliable senders can get greylisted. So what is it?

If an ISP receives an email from a sender that it doesn’t recognize, it may temporarily refuse to accept any email from that sender. This “greylisted” status typically doesn’t last long—a few minutes, maybe—and if that same sender tries again, the message will likely be accepted. But even this relatively short delay can be extremely helpful in mitigating spam.

This works because many spammers only ever make a single delivery attempt to deliver mail to a particular email address. In spamming, volume is king.  So many spammers give up after a single deferral in order to keep the cost to send each message to a minimum. Since spammers typically care little about reliability, many have poorly written email sending systems that aren’t even capable of retrying sends of deferred messages. Plus, those extra few minutes allow the ISPs to run a quick “background check” on the reputation of the sender’s IP address(es), and filter any future mail if it turns out they are a known spammer.

How can you build trust with the ISPs? Start by giving them the space they need. For example, GreenArrow automatically retries delivery of greylisted mail according to its retry schedule until the message is either delivered, bounces, or expires in the queue. The second delivery attempt occurs after a delay of about 6 minutes, 40 seconds, long enough for most ISPs to re-establish communication.

It’s also important to take greylisting into account when configuring Routing Rule VirtualMTAs. For example, if you’re using a routing rule to split mail among a pool of IP addresses, then selecting a randomization type of “Message constant” or “Email address constant” creates a configuration which responds well to greylisting. The “Fully random” randomization option, by contrast, could result in a configuration in which numerous attempts are required, especially if mail is being split between so many different IP addresses that the same IP address rarely sends the same message twice.

(You can see more details on GreenArrow’s randomization options in our Routing Rule VirtualMTAs documentation. You can also check out RFC 6647 if you’d like to learn more about greylisting.)

Tactic #2: Policy Limits

Greylisting is easy to implement, which makes it particularly popular with smaller ISPs. But some spam does get around it, which is one reason why larger ISPs typically take into account a large number of factors, which we collectively refer to as “Policy Limits.”

These Policy Limits restrict the speed that senders can deliver email, by restricting how many connections may be opened at a time and/or how many messages may be delivered over a given interval from a particular sender or IP address. These limits allow for a more sophisticated logic, including treating different senders, well, differently. For example, Gmail applies a different policy limit to a reliable sender than they will to a sender who has sent spam in the past.

Some of these limits are published. For example, Hotmail’s policies page indicates that senders should not open more than 500 simultaneous connections to their inbound email servers at a time.

But most policy limits are not publicly disclosed or static. Take, for example, deferral message (DYN:T1) from AOL. When we check the error code listing page, here is the explanation we get:

The IP address you are sending from has been temporarily rate limited because it is not Whitelisted, unexpected increase in volume, or poor IP reputation.

Yikes! That’s three very different things that could have triggered this deferral.

One thing to note is that policy limit deferrals typically don’t take into account the content of the message being sent. This is because the policy decision is made before the message’s body is transmitted. Once the sending IP address’ reputation passes muster, the content is sent, and additional spam filtering mechanisms determine whether to send the message to the Inbox or Spam folder.

GreenArrow’s default throttling settings are optimized to comply with the policy limits that ISPs typically assign to new IP addresses that are beginning their warm up process with a neutral reputation. As you establish a reputation as a good sender, these policy limits will typically be relaxed, and more mail will be allowed through.

In addition to adjusting throttling settings, which control how fast GreenArrow attempts to deliver email under normal circumstances, we also have another helpful tool called Dynamic Delivery. When deferral and/or bounce rates become elevated, Dynamic Delivery can automatically scale back GreenArrow’s sending speed. And Dynamic Delivery stats are also useful for spotting trends, like how fast messages are being accepted by each ISP.

When Your ISP Doesn’t Feel So Good

As humans, our capacity for conversation is reduced when we get sick or get in an accident. Just like your date might suddenly run to the bathroom, if they fell ill, an ISP could also put rate limiting in place if they’re having system capacity issues.

Things like software bugs, hardware failures, or misconfigurations can all take a portion of an ISP’s infrastructure offline, leaving them unable to cope with the rate at which emails are coming in. The obvious solution is for the ISP to run to the bathroom… I mean, to temporarily reduce the load on system resources by deferring incoming mail.

Granted, most senders don’t even notice when this happens—if an ISP that makes up 1% of a mailing list is having system capacity issues for an hour, it’s not a big deal. But what should you do if a key ISP falls ill? The answer is simple. First, plan ahead, and allow for margin for deferrals in general. Second, work with a deliverability consultant if you would like guidance on what to do in your specific situation.

Getting Rate Limited? We Can Help!

Normally, rate limiting is something that becomes less of an issue over time as you and your IP addresses establish a positive sending reputation with the ISPs. But, just like most relationship difficulties, rate limiting never quite goes away. Even senders with impeccable reputations get rate limited from time to time. This is why having software that responds intelligently to each new situation is so valuable. The better you respond to rate limiting, the better your reputation will be. And that’s what every ISP is looking for—honest, reliable senders who are willing to listen to what they want and give them space when they need it.

Want to become that kind of sender? Consider requesting a free demo to find out how GreenArrow can help. We can’t turn things around overnight, because a good relationship takes time to build. But if you’re willing to work with your ISPs and make a few changes, you might just be surprised at how close you’ll become.