December 2020 Gmail Outages

The Problem

Gmail and G Suite suffered global outages on December 14 and 15, 2020. During those outages, Google generated rejected SMTP delivery of email with “The email account that you tried to reach does not exist” errors when the email addresses were actually valid. This was due to errors in Gmail’s infrastructure.

A consequence of these outages is that if you sent emails while the outage was happening, you likely ended up with subscribers that were removed from your list due to a hard bounce – but the subscriber email address is actually still valid. This is because Google was incorrectly sending out these no-such-user failures.

Google documented the outages on this page, which indicates that they occurred:

From 10:16 pm December 14 to 12:38 am December 15 (UTC)
December 15, from 7:38 pm to 11:57 pm (UTC)

The windows that Google reported line up with when we saw subscribers get improperly activated during our investigation. In the instructions below on how to reactivate subscribers, we added an extra 1-hour buffer to each end time to allow time for asynchronous bounce delivery.

The Fix:

In response to this problem, GreenArrow developed a bounce recovery tool to enable reactivating subscribers who were improperly removed from your list due to this outage.

See the Bounce Recovery page for general information on the tool. The rest of this page focuses on using it in response to the December 2020 Gmail outages.

If you’re a Cloud customer, we’ll run the tool for you, then send you a summary of how many addresses were reactivated. We’ll also offer you the option of obtaining a list of impacted addresses.

If you’re an On-Premises customer, then the instructions below are for you, with the following caveats:

If you have a server management package, you’re welcome to open a ticket with our support team, requesting assistance.
If you have multiple GreenArrow installations, please read the bounce recovery tool’s documentation on multi-server installations before proceeding with this document.

How to Reactivate Subscribers for On-Premises customers

We recommend undoing the effect of all hard bounces received from any domain in the above time windows that contain the string 5.1.1 The email account that you tried to reach does not exist.. We don’t recommend limiting this to just gmail.com because this problem also effected G Suite, which provides email service for many domains. Any email addresses that actually should have been removed from your list will be removed the next time you send.

Here’s how to reactivate subscribers that were improperly deactivated during the outage. We recommend completing the steps for both Studio and SimpleMH, even if you think you’re only using one product or the other, because if you truly aren’t using a product, then the step in question would update 0 records anyway:

SSH into your GreenArrow server as the root user or with equivalent permissions.
Upgrade your GreenArrow installation to version 4.217.0 or later.

If you prefer not to perform an update, then alternatively, you can save a copy of this file to a new location on your GreenArrow server and run it in place of the greenarrow commands in the remaining steps. The filename must be greenarrow. For example:
```
wget https://www.greenarrowemail.com/docs/downloads/greenarrow-gmail-december-2020-bounce-recovery -O /tmp/greenarrow
chmod 700 /tmp/greenarrow
```

Find out how many potentially incorrect bounces were received:

greenarrow bounce_recovery list --start "2020-12-14 22:16 UTC" --end "2020-12-15 01:38 UTC" --bounce-regexp="5\\.1\\.1 The email account that you tried to reach does not exist\\."
greenarrow bounce_recovery list --start "2020-12-15 19:38 UTC" --end "2020-12-16 00:57 UTC" --bounce-regexp="5\\.1\\.1 The email account that you tried to reach does not exist\\."

If the last line of output from each command is 0 record(s) were found, you weren’t impacted by this issue and can skip the rest of the instructions.

Remove invalid entries from the SimpleMH Bad Address Suppression list:

greenarrow bounce_recovery reactivate_simplemh --start "2020-12-14 22:16 UTC" --end "2020-12-15 01:38 UTC" --bounce-regexp="5\\.1\\.1 The email account that you tried to reach does not exist\\."
greenarrow bounce_recovery reactivate_simplemh --start "2020-12-15 19:38 UTC" --end "2020-12-16 00:57 UTC" --bounce-regexp="5\\.1\\.1 The email account that you tried to reach does not exist\\."

Reactivate Studio subscribers that had their status set to bounced:

greenarrow bounce_recovery reactivate_studio --start "2020-12-14 22:16 UTC" --end "2020-12-15 01:38 UTC" --bounce-regexp="5\\.1\\.1 The email account that you tried to reach does not exist\\."
greenarrow bounce_recovery reactivate_studio --start "2020-12-15 19:38 UTC" --end "2020-12-16 00:57 UTC" --bounce-regexp="5\\.1\\.1 The email account that you tried to reach does not exist\\."

Remove invalid entries from Studio suppression lists that are automatically populated with bounced addresses:

greenarrow bounce_recovery unsuppress_studio --start "2020-12-14 22:16 UTC" --end "2020-12-15 01:38 UTC" --bounce-regexp="5\\.1\\.1 The email account that you tried to reach does not exist\\."
greenarrow bounce_recovery unsuppress_studio --start "2020-12-15 19:38 UTC" --end "2020-12-16 00:57 UTC" --bounce-regexp="5\\.1\\.1 The email account that you tried to reach does not exist\\."

If you’re using the Event Notification System, then you may have an external application that deactivated subscribers due to receiving bounce events from GreenArrow. You can repeat the command run in step 1 to obtain a list of addresses to reactivate in CSV format. You can optionally add on the --include-sendid argument to include SendIDs in the output. For example:

greenarrow bounce_recovery list --start "2020-12-14 22:16 UTC" --end "2020-12-15 01:38 UTC" --bounce-regexp="5\\.1\\.1 The email account that you tried to reach does not exist\\." --include-sendid
greenarrow bounce_recovery list --start "2020-12-15 19:38 UTC" --end "2020-12-16 00:57 UTC" --bounce-regexp="5\\.1\\.1 The email account that you tried to reach does not exist\\." --include-sendid