Delivery Event Processed Logfile Format

This document defines the “processed logfile format” which contains log entries of each delivery attempt event.

You get access to this by running the hvmail_report command with the --processed-logfile option.

Logfile format

The processed logfile format contains a single line for each delivery attempt event.

Each line contains a tab-separated set of columns. Each column’s data is guaranteed to not to contain any tabs. There is no escape character or line continuation character.

Columns of the processed logfile format

The columns are as follows:

(1) timestamp

Time of delivery attempt in seconds since the Unix epoch.

(2) channel

Value	Meaning
local	delivery to locally handled email address (i.e. local mailbox)
remote	delivery to remote email address via SMTP

(3) status

Result of this delivery attempt.

Value	Meaning
success	successful delivery
failure	delivery failed due to explicit rejection
failure_toolong	delivery failed due to being in the queue longer than the allowed time (due to deferrals or connmaxout errors)
deferral	delivery deferred for later retry
connmaxout	delivery attempt prevented by throttle (this attempt would have put us over the throttle limits)

(4) is_retry

Value	Meaning
1	this is a retry delivery attempt (i.e. this message was already tried before but the attempt resulted in a deferral or a connmaxout)
0	this is the first delivery attempt for this message

(5) msguid

Internal unique ID of message. These are not repeated unless the system clock goes backwards.

(6) recipient

Recipient email address of message.

(7) sender

Return-Path of email message (also called envelope sender, MFROM address, or bounce address.)

(8) MTAid

The identifier for the VirtualMTA / Mail Route of this message. This is set at the time of message injection.

This field determines what IP address the message is delivered from.

MTAid may be altered at the time of injection by set_virtualmta on header_pattern. If the VirtualMTA is later changed on retry due to the evaluate_on_retry option, this field will still contain the VirtualMTA as determined at the time of injection.

(9) SendID

Internal send identifier of message. This determines which group this message belongs to.

(10) ListID

Mailing list identifier of this message.

(11) injected time

Time in seconds past the Unix epoch that this message was created.

(12) message

Success, failure, or deferral message for this delivery attempt.

Newlines in the message are replaced with the / character, null and ascii record separator characters (hex 1e) are replaced with an underscore (_), and tabs are replaced with spaces.

(13) outmtaid

If this delivery attempt was through SMTP (channel=remote), this is the primary key of the IP Address or Relay Server that was used to deliver this message.

You can see the primary keys by looking at the number at the end of the URL for the View page of each VirtualMTA in the GreenArrow Engine web interface, or by looking at the id columns of the mr_ipaddr or mr_relay tables in PostgreSQL.

(14) SendSliceID

Not yet documented.

(15) throttleid

This field is blank if the delivery attempt took place using an SMTP Relay Server or the default throttling rule.

Otherwise this field contains the ID of the Throttling Rule that was used when making the delivery attempt.

(16) clicktrackingid

This field contains the value of the X-GreenArrow-Click-Tracking-ID header, if present. Otherwise it is left blank.

(17) mx_hostname

The hostname to which this delivery attempt occurred (whether or not it came from an actual MX record).

(18) mx_ip

The IP address to which this delivery attempt occurred (whether or not it came from an actual MX record).

(19) from_address

The first email address in the “From” header as it was injected into GreenArrow.

(20) headers

A JSON object containing the headers that are configured to be logged by the log_header configuration directive.

Folded headers will simply contain the folding newlines/whitespace.

(21) message_size

An integer indicating the message size (in bytes) loaded before the delivery attempt was made. If the message was not loaded (for example if the SMTP session failed prior to reaching the DATA segment) then this value will be blank.

(22) smtp_timing

A CSV list of SMTP timings. See the log_smtp_timings directive for more information.

(23) bounce_code

For non-successful delivery attempts, this is the bounce code for this response. This is the same bounce_code that will be included in the delivery_attempt event.

(24) source_ip

The SMTP or HTTP client IP address that injected the message into GreenArrow.

(25) mailclass

The SimpleMH Mail Class used for this message.

(26) instanceid

The InstanceID for this message from either the X-GreenArrow-InstanceID or X-Job headers.

When the Mail Class has its Statistic Report Grouping set to per_instanceid_legacy, the instanceid will be generated automatically (if not explicitly set) in the YYMMDD format.

Example Data

Here is an example delivery attempt event:

Column number	Column name	Example Data
1	timestamp	1317299024.20073
2	channel	remote
3	status	success
4	is_retry	0
5	msguid	1317299024.4669464
6	recipient	[email protected]
7	sender	[email protected]
8	MTAid	smtp1-1
9	SendID	1234
10	ListID	5678
11	injected_time	1317299024
12	message	64.21.76.32 accepted message./Remote host said: 250 ok 1317299024 qp 4973
13	outmtaid	2
14	SendSliceID	298602
15	throttleid	35
16	clicktrackingid	1234
17	mxhostname	mx1.example.com
18	mxip	1.2.3.4
19	from_address	[email protected]
20	headers	{"Subject":["Welcome to our list!"]}
21	message_size	20123
22	smtp_timing	978,0,0,141,0,0,0,720,205,0,0,116,121,179,139,0,23,132,155,1,0,0
23	bounce_code	20
24	source_ip	127.0.0.1
25	mailclass	default
26	instanceid	1234

Other Considerations

Messages with multiple recipients

If a message is injected to multiple recipients, there will be one msguid for that message. (The msguid is unique to the message.) Since each delivery attempt for each recipient will be logged, you can have multiple log entries for each msguid. In this case, you would want to use the recipient field to differentiate between these different results.

Note: we may add a field to this logfile format that will track the number of the recipient for the message, so it would be easier to differentiate between results for different recipients. or we may change the msguid field to be specific to the recipient and not the

For example:

# date | /var/hvmail/bin/mailsubj test1 [email protected] [email protected]

# hvmail_report -q ram –last ‘1 min’ –processed-logfile

1317240923.22286 remote success 0 1317240923.67920340 [email protected] [email protected] 1317240923 64.21.76.32 accepted message./Remote host said: 250 ok 1317240923 qp 25956/ 2 35 1234

1317240924.03923 remote success 0 1317240923.67920340 [email protected] [email protected] 1317240923 64.21.76.32 accepted message./Remote host said: 250 ok 1317240924 qp 25962/ 2 35 1235

You can see how each line has the same msguid.

Human-readable output

The --processed-logfile option is useful for scripting, but isn’t easily read by humans. The --human option can be added in combination with --processed-logfile to make it a bit more human-friendly.

Here is an example of both covering the same data:

• Sample Email:

  date | /var/hvmail/bin/mailsubj test1 [email protected]
  

• --processed-logfile:

  # hvmail_report -q ram --last '1 min' --processed-logfile
  1428427817.05678	remote	success	0	1428427816.76031064	[email protected]	[email protected]				1428427816	207.99.125.72 accepted message./Remote host said: 250 ok 1428427817 qp 15219/	2	mx.example.com	1.2.3.4
  

• --processed-logfile --human

  # hvmail_report -q ram --last '1 min' --processed-logfile --human
  timestamp=<1428427817.05678> channel=<remote> status=<success> is_retry=<0> msguid=<1428427816.76031064> recipient=<[email protected]> sender=<[email protected]> MTAid=<> SendID=<> ListID=<> injected_time=<1428427816> message=<207.99.125.72 accepted message./Remote host said: 250 ok 1428427817 qp 15219/> outmtaid=<2> SendSliceID=<> throttleid=<> clicktrackingid=<> mx_hostname=<mx.example.com> mx_ip=<1.2.3.4>