Processing Events on Dedicated Servers

• Table of Contents
• Introduction
• Software Upgrade Considerations
• Procedure
  • Update Secret Constants
  • Configure Event Tracking Metadata Storage
  • Create a Bounce Mailbox
  • Configure the Bounce Email Address on the Sending Servers
  • NAT Considerations
  • Configure Faster Delivery of Bounces
    • Configure SMTP Connection Reuse
    • Raise the Throttle Limits
    • Tune the Bounce-Queue
    • Truncate Bounces at 5KB
    • Treat Bounces as “Local”
  • Configure URL Domains
  • Configure the Event Servers to Record Events
  • Setup Feedback Loops
• Testing
• Restrictions

Introduction

This page describes how to configure GreenArrow to split the load of processing SimpleMH email into two groups of servers:

• Sending Servers, which send emails.
• Event Servers, which process clicks, opens, unsubscribes, bounces and spam complaints.

You can have one, or multiple Sending Servers and Event Servers.

This document uses the term All Servers to refer to all Sending Servers and Event Servers.

This configuration only works for SimpleMH email.

Software Upgrade Considerations

When upgrading a fleet of GreenArrow servers, it’s important that you upgrade all of the Event Servers before upgrading any Sending Servers.

The reason for this is that data in messages generated by the Sending Servers (for example, a click tracking link) needs to be processable by the Event Servers. Since GreenArrow strives for backward compatibility, the Event Servers of the newer version of GreenArrow will be able to consume data generated from either the older or newer version of GreenArrow.

Procedure

Complete the following steps.

Update Secret Constants

Secret constants are used to create anti-tampering security codes for links and bounce messages. All Servers need to have the same “secret constants”:

1. Choose one Sending Server, get the secret constants:

   greenarrow_config secret_constants --view
   

2. On All Servers, set the secret constants to match:

   greenarrow_config secret_constants --constant1=INSERT_HERE --constant2=INSERT_HERE
   

3. On All Servers: print the secret constants and verify they all match:

   greenarrow_config secret_constants --view
   

Configure Event Tracking Metadata Storage

By default, GreenArrow uses a local database to assist with event tracking. This is not compatible with processing events on dedicated servers.

Review Event Tracking Metadata Storage and configure your default_event_tracking_metadata_storage to use either stateless or external.

Create a Bounce Mailbox

On the Event Servers, add a new bounce mailbox.

On the Sending Servers, ensure that the bounce mailbox’s domain does NOT exist as an Incoming Email Domain.

The bounce domain’s MX records should point to the Event Servers.

Configure the Bounce Email Address on the Sending Servers

On the Sending Servers, either:

1. Edit the Mail Classes that you are going to use with this bounce address:

   • If the simplemh_validate_bounce_address directive is not disabled, add the external bounce email address using the bounce_address_external directive, and then select it from the “Bounce Address” selection list

   • If the simplemh_validate_bounce_address directive is disabled, click “Override Default Bounce Address” and enter the bounce email address into the “Bounce Address” field.

   The simplemh_validate_bounce_address directive is not disabled by default.

2. Use the X-GreenArrow-BounceMailboxOverride header to override the bounce address. This header takes precedence over the Mail Class setting.

NAT Considerations

If your servers are behind NAT, the MX record for your bounce email address domain may point to an IP address that your Sending Server can’t connect to. If this is the case, you will need to configure static routing for this domain name using the smtp_route directive.

For example, assuming your Event Server’s IP is 10.0.55.2, and your bounce address domain is bounce-domain.example.com:

1. Add this route to the Sending Servers in /var/hvmail/control/greenarow.conf:

   ip_address * {
     domain bounce-domain.example.com {
       smtp_route 10.0.55.2
     }
   }
   

2. Load the new configuration:

   greenarrow_config reload
   

3. Verify that the Sending Servers can send emails to each bounce address domain on the Event Servers.

Configure Faster Delivery of Bounces

Configure SMTP Connection Reuse

On the Sending Servers:

1. Add the following to /var/hvmail/control/greenarow.conf, replacing domain bounce-domain.example.com with your actual bounce address domain::

   ip_address * {
     domain bounce-domain.example.com {
         reuse_connections yes
     }
   }
   

2. Load the new configuration:

   greenarrow_config reload
   

Raise the Throttle Limits

On the Sending Servers, add the bounce email address domain to all throttle templates and set the following configuration for it:

• Max Concurrent Connections: 100
• Max Messages per Hour: 10000000

NOTE: GreenArrow isn’t realistically going to send 10 million messages per hour over 100 connections. The reason for the 10 million message per hour limit is to set the ceiling high enough so that if bounces get created in bursts, the Sending Server can efficiently add them to the delivery backlog.

Tune the Bounce-Queue

Run these commands on the Sending Servers:

cd /var/hvmail/control
echo 100 > queue.bounce.concurrencyremote
echo 5000 > queue.bounce.limit.messages
echo 50000 > queue.bounce.ramdisk.inodes
echo 200 > queue.bounce.ramdisk.size
hvmail_init restart

Truncate Bounces at 5KB

Run this command on the Sending Servers:

echo 5120 > /var/hvmail/control/bouncemaxbytes
hvmail_init restart

Treat Bounces as “Local”

On the Event Servers, add all of the Sending Server names (as configured in /var/hvmail/control/me) to the /var/hvmail/control/opt.treat_qsbf_bounces_as_local_server_names file. Enter one name per line.

Configure URL Domains

Create each URL Domain in the following way.

On the Sending Servers:

1. Add the URL Domain to GreenArrow’s configuration.

2. Edit the Mail Classes that you are going to use with this URL domain, enable “Track Clicks and Opens”, and select the URL domain from the drop-down menu.

On the Event Servers

1. Create the URL domain’s DNS records so that they point to the Event Servers.

2. If you want to use HTTPS links for the URL Domain, then install each TLS certificate on the Event Servers.

Configure the Event Servers to Record Events

The Event Notification System is the only way you will be able to see the clicks, opens unsubscribes, bounces, and spam complaints that take place. This is because the Event Servers which process these events didn’t send the original emails, so they don’t have any Send Statistics Pages to populate.

Configure the Event Notification System on each Event Server.

Setup Feedback Loops

1. Create a spam complaint mailbox on each Event Server.

2. Register for feedback loops using the spam complaint mailbox created in the previous step.

Testing

We recommend testing the system before starting production sends. The following sequence is designed to catch common misconfigurations.

First, check for DNS misconfigurations:

1. Verify that each URL domain has A or CNAME records pointing to the desired Event Servers.
2. Verify that each incoming email domain used for bounce and spam complaint mailboxes has MX records pointing to the desired Event Servers.

Next, for each Sending Server / Event Server combination:

1. Send an HTML email that uses click tracking and contains an unsubscribe link from the Sending Server.
2. Load images in the email. Verify that the images load and that an engine_open event is created.
3. Click on one of the links in the email. Verify that it redirects to the correct URL and that an engine_click event is created.
4. Click the unsubscribe link, and verify that you’re redirected to the correct URL. If you’re using SimpleMH unsubscribe link tracking, verify that an engine_unsub event is created.
5. Send an email containing the raw headers of the email you sent in step one to a spam complaint mailbox on the Event Server. Verify that an scomp event is created.
6. Send an email from the Sending Server to an address that bounces. Verify that a bounce_all event is created. Depending on the type of bounce, and how GreenArrow’s bounce processor is created, this could also result in bounce_bad_address event being created.

Restrictions

To perform split event processing, GreenArrow adds information about the subscriber to every message. This information is later used by the event server when processing incoming events to identify the subscriber to whom the message belongs. This is different to the traditional event processing mechanism that leverages information stored locally on the sending server to identify the subscriber.

A GreenArrow server that has used “traditional” event processing can be configured to do split event processing. However, care must be taken to avoid losing events for messages sent with the traditional method.

Messages sent before the change will not include the information that event servers can use to identify the subscriber, which means that events for these messages have to be processed on the sending server. This forces restrictions on URL domains and bounce mailboxes used before the change.

URL domains used on messages before the change can’t be moved to event servers, or can’t be moved immediately - the A records for these domains should resolve to the sending server.

The same restriction applies to bounces and spam complaints. The sending server must be the mail server for domains used for bounces and spam complaints in messages sent before the change, at least for a reasonable period of time.

To avoid these issues use other domains for the new configuration while keeping the old domains on the sending server.

Other restrictions apply to all servers that use split event processing:

1. Each server requires a GreenArrow license, this includes servers dedicated to event processing
2. Send statistics will still be available on the sending server, however, these statistics will not reflect any of the events processed remotely. This means that bounces, clicks, opens, unsubscribes, and spam complaints will not be reflected in the sending server’s statistics
3. Event servers will not have statistics for events that they process - they will, however, record events and deliver event notifications if they are properly configured
4. This method works for messages sent with SimpleMH. GreenArrow Studio does not support split event processing.
5. The Repeat Bounce Counting System does not work properly when using split event processing. Each Event Server only knows about the bounces that it has processed, and does not know about the emails sent on any of the Sending Servers. They each make the best decision they can with the data they have.