Mailgun-Compatible Webhooks

• Table of Contents
• Overview
• Configuration
  • Events of each webhook type
  • Which messages give webhooks
• Requests
  • Verify the signature
  • Responses
• Event types
  • Suppressed events
• Field mapping
• Differences from Mailgun

Overview

An event delivery destination with the compat_mg event_delivery_format sends GreenArrow events as Mailgun webhooks. Each request has the JSON document and the signature that Mailgun sends, so software that was written to receive Mailgun webhooks (the “consumer”) can receive GreenArrow events without a change.

A compat_mg destination sends only the events of messages that have the Click Tracking ID that the Mailgun compatibility API writes. See Which messages give webhooks.

Webhooks are registered in greenarrow.conf, not through Mailgun’s Webhooks API. GreenArrow does not have a Mailgun-compatible Webhooks API.

Configuration

Mailgun lets you register one URL for each webhook type. In GreenArrow, make one event_delivery_destination for each URL. event_delivery_compat_mg_webhooks names the webhook types that go to that URL. A destination can name more than one webhook type, to send them to one URL.

For example, a consumer that has a URL for permanent failures, a URL for temporary failures and a URL for spam complaints needs these three destinations:

general {
  log_header "From", "To", "Subject"
}

event_delivery_destination mailgun-permanent-failure {
  event_delivery_format                 compat_mg
  event_delivery_url                    https://api.example.com/webhooks/permanentFailure
  event_delivery_compat_mg_signing_key  </var/hvmail/control/mailgun_webhooks_signing_key>
  event_delivery_compat_mg_webhooks     permanent_fail
  event_delivery_disable_batching       yes
}

event_delivery_destination mailgun-temporary-failure {
  event_delivery_format                 compat_mg
  event_delivery_url                    https://api.example.com/webhooks/temporaryFailure
  event_delivery_compat_mg_signing_key  </var/hvmail/control/mailgun_webhooks_signing_key>
  event_delivery_compat_mg_webhooks     temporary_fail
  event_delivery_disable_batching       yes
}

event_delivery_destination mailgun-complained {
  event_delivery_format                 compat_mg
  event_delivery_url                    https://api.example.com/webhooks/complained
  event_delivery_compat_mg_signing_key  </var/hvmail/control/mailgun_webhooks_signing_key>
  event_delivery_compat_mg_webhooks     complained
  event_delivery_disable_batching       yes
}

A compat_mg destination has these rules:

• It must be an HTTP destination (event_delivery_url). The format is not valid for a logfile, pipe or SQL destination.
• event_delivery_compat_mg_signing_key is required. It is the key of the signature. The consumer must have the same key.
• event_delivery_compat_mg_webhooks is required.
• event_delivery_disable_batching must be yes. A Mailgun webhook has one event.
• event_delivery_events and event_delivery_delivery_attempt_config are not permitted. The webhook types select the events.
• event_delivery_anonymize_clicktrackingid and event_delivery_anonymize_localparts are not permitted. This includes the value no. GreenArrow builds each webhook from the Click Tracking ID, and the consumer matches the webhook to its records by the recipient.
• The global log_header directive (in the general context) must name the headers From, To and Subject. The names are compared without regard to case. A log_header in a source context does not count. log_header_remove does not count, because it removes the header from the message. The webhook copies these headers from the delivery_attempt event. Note that this directive logs these headers for all mail on the server.

A configuration that breaks one of these rules does not load. The error names the destination and the rule.

Events of each webhook type

Each webhook type selects these GreenArrow events for the destination:

• A destination gets a delivery_attempt event only for the statuses in this table. The defaults of event_delivery_delivery_attempt_config do not apply. For example, a temporary_fail destination gets each deferral of a message, not only the first. A connmaxout delivery attempt and a delivery to a local mailbox give no webhook.
• A permanent_fail destination gets each bounce_all event, but sends a webhook only for an asynchronous bounce (a bounce message that arrived after the delivery). A synchronous bounce gives no webhook, because its delivery_attempt event gave one. So each failed delivery gives one webhook.
• The GreenArrow Studio forms of the events are not selected.

Which messages give webhooks

A compat_mg destination sends the events of a message only when the message’s Click Tracking ID is a JSON object that has a string member m. The Mailgun compatibility API writes this Click Tracking ID for each message that it accepts. GreenArrow checks only the Click Tracking ID, so a message from another source that has this Click Tracking ID also gives webhooks. These are its members:

GreenArrow removes the events of other messages from the destination’s queue and does not send them, because the consumer cannot match them to its records. This includes GreenArrow Studio campaigns, messages with a Click Tracking ID that is not such an object, and messages with no Click Tracking ID. A Click Tracking ID whose M, t or v has another type than the table shows is also not such an object.

The events of other messages go into the destination’s queue before GreenArrow removes them. If your server also sends mail outside of the Mailgun compatibility API, you can use event_delivery_filter_mail_class to keep the events of that mail out of the queue.

Requests

Each request is an HTTP POST of one event, with the Content-Type application/json. The JSON document has two members: signature and event-data.

{
  "signature": {
    "timestamp": "1529006854",
    "token": "a8ce0edb2dd8301dee6c2405235584e45aa91d1e9f979f3de0",
    "signature": "464b0644ffb4690f66615f470cd8e1f911824e8b1ce9b7f87cc0f964e99f1fa2"
  },
  "event-data": {
    "event": "failed",
    "severity": "permanent",
    "id": "8a0f9b02-5ea1-4b8f-8d9e-2c1b3a5e7d64",
    "timestamp": 1529006851.123,
    "recipient": "[email protected]",
    "recipient-domain": "example.com",
    "envelope": {
      "sender": "[email protected]",
      "sending-ip": "192.0.2.10"
    },
    "delivery-status": {
      "code": 550,
      "enhanced-code": "5.1.1",
      "message": "192.0.2.25 does not like recipient./Remote host said: 550 5.1.1 The email account that you tried to reach does not exist/Giving up on 192.0.2.25./",
      "mx-host": "mx.example.com",
      "attempt-no": 1
    },
    "message": {
      "headers": {
        "message-id": "[email protected]",
        "from": "Sample Sender <[email protected]>",
        "to": "[email protected]",
        "subject": "Your order has shipped"
      },
      "size": 2048
    },
    "user-variables": {"customer_id": "1234"},
    "tags": ["shipping"]
  }
}

Verify the signature

The signature object lets the consumer verify that the request comes from GreenArrow:

• timestamp is the time when GreenArrow sent the request (not the time of the event), in seconds past the Unix epoch, as a string.
• token is a random string of 50 hexadecimal characters.
• signature is the hexadecimal HMAC-SHA256 of timestamp followed by token. The key is the destination’s event_delivery_compat_mg_signing_key.

This is the same signature that Mailgun sends, so a consumer that verifies Mailgun’s signature needs no change. To verify a request, compute the HMAC-SHA256 with the same key and compare it to signature.

For example, with the signing key key-abc123 and the request above, this command writes the digest 464b0644ffb4690f66615f470cd8e1f911824e8b1ce9b7f87cc0f964e99f1fa2, which is equal to signature:

printf '%s' '1529006854a8ce0edb2dd8301dee6c2405235584e45aa91d1e9f979f3de0' | openssl dgst -sha256 -hmac 'key-abc123'

The same check in Python:

import hashlib
import hmac

def verify(signing_key, timestamp, token, signature):
    expected = hmac.new(
        key=signing_key.encode(),
        msg=(timestamp + token).encode(),
        digestmod=hashlib.sha256,
    ).hexdigest()
    return hmac.compare_digest(expected, signature)

Each request has a new timestamp, token and signature. This includes a retry. So a consumer can keep the tokens that it receives, and refuse a request with a token that it received before.

A destination has one signing key. To change the key, change it in greenarrow.conf and in the consumer.

Responses

• Mailgun uses 406 to refuse an event. On a destination with another format, 406 is a failure, and GreenArrow sends the event again later.
• GreenArrow writes a refused event to the event processor log. The event delivery metrics count it as a failed delivery (event_delivery_failed_total).
• A compat_mg destination does not follow a redirect. A 3xx response is a failure, and GreenArrow sends the event again later. A redirect can change the POST to a GET that does not have the webhook.

Event types

• The severity of a delivery attempt comes from its status, not from the SMTP status code. In Mailgun, temporary means that the message will be tried again. GreenArrow tries a message again after a deferral. It does not try it again after a failure or a failure_toolong (the message expired in the queue), even when the SMTP status code starts with 4.
• An asynchronous bounce is always permanent, because GreenArrow does not try the message again. The GreenArrow bounce type does not change the severity. It gives delivery-status.bounce-type.
• GreenArrow does not send these Mailgun events: accepted (in this version), rejected, stored, and the mailing list events.

Suppressed events

GreenArrow removes these events from a compat_mg destination’s queue without a request, as if they were delivered. They are not counted as delivered or failed, and are not sent again:

• An event that has no webhook in the table above: a synchronous bounce_all event, or a delivery_attempt event with the status connmaxout.
• An event of a webhook type that the destination does not name in event_delivery_compat_mg_webhooks. The destination does not select these events, so they usually do not get into its queue.
• An event of a message whose Click Tracking ID is not the one that the Mailgun compatibility API writes (see Which messages give webhooks).

Field mapping

The GreenArrow fields are described in Types of Events. These rules apply:

• A field is left out when the event has no value for it. An object with no members (envelope, delivery-status, message, message.headers or client-info) is left out.
• envelope.sender is an exception. It is present on each delivery_attempt and bounce_all event that has a sender, also when the sender is empty: the null sender (<>) is a valid envelope sender. A bounce_all event has sender and outmtaid_ip, and a scomp event has outmtaid_ip, only when include_mtaid_info_at_delivery was enabled when the message was delivered.
• user-variables and tags are always present. When the Click Tracking ID has no v, user-variables is {}. When it has no t, tags is [].
• timestamp always has a fractional part, for example 1529006854.0.
• The status text is the message of a delivery attempt. On an asynchronous bounce, it is the bounce_text before the copy of the original message, without the white space at its start and end. The copy starts at the qmail line --- Below this line is a copy of the message, at a MIME part with the content type message/rfc822 or text/rfc822-headers, or at the Exchange line Original message headers:. So the original message is not sent to the consumer, and a number in it is not read as a status code.
• delivery-status.code is the first three-digit SMTP reply code (2xx, 4xx or 5xx) in the status text that is followed by a space, a hyphen or the end of the text. delivery-status.enhanced-code is the first enhanced status code (for example 5.1.1) in the status text. A number that is part of a longer run of digits and periods, such as an IP address, is not a code. When the status text has no code, the field is left out. For example, the text 550 5.1.1 The email account that you tried to reach does not exist gives "code": 550 and "enhanced-code": "5.1.1".
• delivery-status.attempt-no is the number of the delivery attempt of the message to this recipient, starting at 1. See attempt_number in Types of Events.
• delivery-status.bounce-type is left out when bounce_type has another value or no value.
• message.headers.message-id is never GreenArrow’s msguid. The white space at the start and end of the value is removed, and then one < at the start and one > at the end: the Message-ID header <[email protected]> gives [email protected], as in Mailgun.
• message.headers.from, to and subject are the values of the headers, as Mailgun sends them. from includes the display name, for example Sample Sender <[email protected]>. These fields are also an exception to the first rule: a logged header with an empty value gives an empty string. When a header occurs more than one time, the first value is used. A folded header is unfolded, and its encoded words (RFC 2047) are decoded. When the header has an encoded word in a character set other than UTF-8, ISO-8859-1 or US-ASCII, no encoded word of the header is decoded.
• Only delivery_attempt events have logged headers. So from, to and subject are only on delivered and on a failed webhook of a delivery attempt. The other webhooks do not have them.
• These Mailgun fields are left out: reason, log-level, flags, geolocation, campaigns, envelope.targets and envelope.transport.

Differences from Mailgun

• complained, and failed from an asynchronous bounce, have no message.headers.from, to or subject. Mailgun sends them on these events. GreenArrow does not log headers on these events. (Mailgun also sends only message-id on opened, clicked and unsubscribed.)
• The fields in the last rule of Field mapping are left out.
• GreenArrow does not send some Mailgun events. See Event types.
• Webhooks are registered in greenarrow.conf. There is no Mailgun-compatible Webhooks API.