GreenArrow Email Software Documentation

QMQP Streaming Protocol

Introduction

As an alternative to implementing a multi-threaded sending SMTP client library to send high volumes of mail, GreenArrow Engine supports the QMQP Streaming Protocol, which has many of the same performance benefits but is much easier to implement.

The QMQP Streaming Protocol is asynchronous. Messages that the client wishes to send are written (“streamed”) out to the server without waiting for any replies. This means that a network round trip latency is not required for each message sent. The server queues messages to GreenArrow Engine and sends back to the client a list of replies giving the status of each message. Each message is sent with a unique identifier, and the reply contains that unique identifier so that they may be matched together.

Netstrings

This protocol makes heavy use of netstrings as defined here: http://cr.yp.to/proto/netstrings.txt

What does the client send?

The client sends one or more message blocks.

To create a message block, the client encodes each of the following elements into a netstring, concatenates them, then encodes the result into a single netstring:

  • The string M (ASCII character decimal 77).
  • A unique identifier for this message. This will be returned to the client with the success or failure of delivery of this message.
  • A safe 8-bit text message. The client encodes the message as the byte string firstline\012secondline\012...\012lastline. The last line is usually, but not necessarily, empty.
  • The envelope sender address (Return-Path).
  • One or more recipients, each encoded as their own netstring.

Sample PHP code:

<?php
function netstring($data)
{
   return strlen($data) . ":" . $data . ",";
}

function create_message_block($id, $message, $sender, $recips)
{
   $data = netstring("M") . netstring($id) . netstring($message) . netstring($sender);
   foreach ( $recips as $pos => $this_recip ) {
      $data .= netstring($this_recip);
   }
   return $data;
}

What does the server send?

The server sends a reply block for each message block sent by the client. The reply blocks may come in a different order than the messages were sent in.

Each reply block consists of a netstring, with the following parts concatenated. Each portion is itself encoded as a netstring:

  • The string R (ASCII character decimal 82).
  • The unique identifier of the message this reply is for.
  • A string describing the result
  • A string in decimal (e.g. 23) describing how many messages are yet to be completed, as far as the server is aware of right now. The client may have sent more messages than this number, if some were still in-transit to the server.

Result String

The string describing the result is defined as follows:

The first byte of the string is either K, Z, or D:

  • K means that the message has been accepted for delivery to all envelope recipients. This is morally equivalent to the 250 response to DATA in SMTP; it is subject to the reliability requirements of RFC 1123, section 5.3.3.
  • Z means temporary failure; the client should try again later.
  • D means permanent failure.

Note that there is only one response for the entire message; the server cannot accept some recipients while rejecting others.

The remaining bytes are a description of what happened. It is expected that the description, when interpreted as UTF-8 characters, will be readable English text, and will not include formatting characters other than \040. However, these expectations are not requirements, and the client must be ready for arbitrary bytes from the server.

Descriptions beginning with \040 are reserved for future extensions. In descriptions not beginning with \040, the byte \043 must not appear except in HCMSSC codes.

QMQP Streaming Session Overview

The following stages occur during a QMQP Streaming Protocol session.

  1. Connection opened.
  2. Client sends zero or more message blocks, meanwhile server may send zero or more reply blocks.
  3. Client sends done block.
  4. Server sends zero or more reply blocks, until all messages blocks have been replied to.
  5. Server sends done block.
  6. Server closes connection.

Sample Session

There is no ending newline in the sample sessions below.

Data sent by client:

122:1:M,4:msg1,72:From: [email protected]
To: [email protected]
subject: hi

this is the message
,12:[email protected],15:[email protected],,122:1:M,4:msg2,72:From: [email protected]
To: [email protected]
subject: hi

this is the message
,12:[email protected],15:[email protected],,1:D,

Hex dump of the above:

00000000  31 32 32 3a 31 3a 4d 2c  34 3a 6d 73 67 31 2c 37  |122:1:M,4:msg1,7|
00000010  32 3a 46 72 6f 6d 3a 20  72 6f 6f 74 40 64 72 68  |2:From: root@abc|
00000020  2e 6e 65 74 0a 54 6f 3a  20 64 68 61 72 72 69 73  |.net.To: dharris|
00000030  40 64 72 68 2e 6e 65 74  0a 73 75 62 6a 65 63 74  |@abc.net.subject|
00000040  3a 20 68 69 0a 0a 74 68  69 73 20 69 73 20 74 68  |: hi..this is th|
00000050  65 20 6d 65 73 73 61 67  65 0a 2c 31 32 3a 72 6f  |e message.,12:ro|
00000060  6f 74 40 64 72 68 2e 6e  65 74 2c 31 35 3a 64 68  |[email protected],15:dh|
00000070  61 72 72 69 73 40 64 72  68 2e 6e 65 74 2c 2c 31  |[email protected],,1|
00000080  32 32 3a 31 3a 4d 2c 34  3a 6d 73 67 32 2c 37 32  |22:1:M,4:msg2,72|
00000090  3a 46 72 6f 6d 3a 20 72  6f 6f 74 40 64 72 68 2e  |:From: root@abc.|
000000a0  6e 65 74 0a 54 6f 3a 20  64 68 61 72 72 69 73 40  |net.To: dharris@|
000000b0  64 72 68 2e 6e 65 74 0a  73 75 62 6a 65 63 74 3a  |abc.net.subject:|
000000c0  20 68 69 0a 0a 74 68 69  73 20 69 73 20 74 68 65  | hi..this is the|
000000d0  20 6d 65 73 73 61 67 65  0a 2c 31 32 3a 72 6f 6f  | message.,12:roo|
000000e0  74 40 64 72 68 2e 6e 65  74 2c 31 35 3a 64 68 61  |[email protected],15:dha|
000000f0  72 72 69 73 40 64 72 68  2e 6e 65 74 2c 2c 31 3a  |[email protected],,1:|
00000100  44 2c                                             |D,|
00000102

Data sent by server:

19:1:R,4:msg1,1:K,1:1,,19:1:R,4:msg2,1:K,1:0,,1:D,

Hex dump of the above:

00000000  31 39 3a 31 3a 52 2c 34  3a 6d 73 67 31 2c 31 3a  |19:1:R,4:msg1,1:|
00000010  4b 2c 31 3a 31 2c 2c 31  39 3a 31 3a 52 2c 34 3a  |K,1:1,,19:1:R,4:|
00000020  6d 73 67 32 2c 31 3a 4b  2c 31 3a 30 2c 2c 31 3a  |msg2,1:K,1:0,,1:|
00000030  44 2c                                             |D,|
00000032

Authentication

You may perform username/password authentication in the QMQP Streaming Protocol.

To request authentication, the client encodes each of the following elements into a netstring, concatenates them, encodes the result it into a single netstring, and sends it to the server:

  • The string A.
  • The username.
  • The password.

In reply to this authentication request, the server encodes each of the following elements into a netstring, concatenates them, encodes the result into a single netstring, and sends it to the client:

  • The string A.
  • The string 1 if authentication is successful or 0 if not.

Example authentication request. There is no ending newline in the this example sent by the client or server below:

34:1:A,10:myusername,12:the_password,,

Hex dump of above:

00000000  33 34 3a 31 3a 41 2c 31  30 3a 6d 79 75 73 65 72  |34:1:A,10:myuser|
00000010  6e 61 6d 65 2c 31 32 3a  74 68 65 5f 70 61 73 73  |name,12:the_pass|
00000020  77 6f 72 64 2c 2c                                 |word,,|
00000026

Example authentication reply showing success:

8:1:A,1:0,,

Hex dump of above:

00000000  38 3a 31 3a 41 2c 31 3a  30 2c 2c                 |8:1:A,1:0,,|
0000000b

Credits

This protocol is inspired by QMQP. This document borrows some language from the QMQP specification as found here: http://cr.yp.to/proto/qmqp.html

 QMQP
Speed Considerations 

Copyright © 2012–2024 GreenArrow Email