GreenArrow Email Software Documentation

Service Logs

Most of GreenArrow’s services use multilog to log their output to a subdirectory of /var/hvmail/log/.

For example, PostgreSQL logs data to the /var/hvmail/log/postgres/ directory. To tail the PostgreSQL log, run:

tail -F /var/hvmail/log/postgres/current | tai64nlocal

multilog is similar to syslog in that it’s a standard for message logging. There are a few key differences to keep in mind:

  1. Timestamps are recorded using external TAI64 format. You may decode these timestamps using tai64nlocal, as shown in the above tail example. These timestamps appear in both the beginning of each log entry and in the names of most files.

  2. multilog stores the logs for each service within its own directory. Within each service’s directory, the most recently updated log file is named current. Older logs have filenames starting with an external TAI64 format timestamp showing what time that log was last updated.

  3. syslog is normally configured to retain logs for a specific amount of time. multilog, by contrast, gets configured to retain a specific amount of data. For example, multilog could be configured to retain up to 100MB of data for a service.

Log Locations

Service Name Log Location
Automatic TLS /var/hvmail/log/tls
Bounce Processor /var/hvmail/log/bounce-processor
Lite Bounce Processor /var/hvmail/log/bounce-processor2
Config Agent /var/hvmail/log/config-agent
Dynamic Delivery Dispatcher /var/hvmail/log/dd-dispatcher
Dynamic Delivery Log Reader /var/hvmail/log/dd-logreader
Event Processor /var/hvmail/log/event-processor
Event Processor 2 /var/hvmail/log/event-processor2
HTTPD /var/hvmail/log/httpd
HTTPD Monitor /var/hvmail/log/httpd-monitor
Logfile Agent /var/hvmail/log/logfile-agent
Logfile Summary /var/hvmail/log/logfile-summary
Logfile Writer /var/hvmail/log/logfile-writer
Passenger /var/hvmail/log/passenger
Postgres /var/hvmail/log/postgres
Pure-FTPd Authentication Agent /var/hvmail/log/pure-authd-studio
Pure-FTPd /var/hvmail/log/pure-ftpd
Qmail - POP3 /var/hvmail/log/qmail-pop3d
Qmail - SMTPD (port 25) /var/hvmail/log/qmail-smtpd
Qmail - SMTPD2 (default port 587) /var/hvmail/log/qmail-smtpd2
Qmail - SMTPD3 (default port 901) /var/hvmail/log/qmail-smtpd3
Redis /var/hvmail/log/redis
Redis NP /var/hvmail/log/redis-np
RPC /var/hvmail/log/rpc
Rspawn Limiter /var/hvmail/log/rspawn-limiter
SimpleMH /var/hvmail/log/simplemh
SimpleMH 2 /var/hvmail/log/simplemh2
SMTP Sink /var/hvmail/log/smtp-sink
Studio Worker /var/hvmail/log/studio-worker

Adjusting Log Retention

The amount of data that multilog retains for most services can be configured by updating the /var/hvmail/control/logconfig.multilog file. Here’s a list of the services can have their log retention configured via this file:

Service Default Retention Service Name in Configuration File
/service/hvmail-bounce-processor 10MB BOUNCE_PROCESSOR
/service/hvmail-bounce-processor2 10MB BOUNCE_PROCESSOR2
/service/hvmail-config-agent 1MB CONFIG_AGENT
/service/hvmail-dd-dispatcher 1MB DD_DISPATCHER
/service/hvmail-dd-logreader 1MB DD_LOGREADER
/service/hvmail-event-processor 10MB EVENT_PROCESSOR
/service/hvmail-event-processor2 10MB EVENT_PROCESSOR
/service/hvmail-greenarrow 10MB GREENARROW
/service/hvmail-httpd 1MB HTTPD
/service/hvmail-httpd-monitor 1MB HTTPD
/service/hvmail-logfile-agent 1MB LOGFILE_AGENT
/service/hvmail-logfile-summary 1MB LOGFILE_SUMMARY
/service/hvmail-logfile-writer 1MB LOGFILE_WRITER
/service/hvmail-passenger 250MB PASSENGER
/service/hvmail-postgres 20MB POSTGRES
/service/hvmail-pure-authd-studio 1MB PURE_AUTHD_STUDIO
/service/hvmail-pure-ftpd 1MB PURE_FTPD
/service/hvmail-qmail-pop3d 1MB QMAIL_POP3D
/service/hvmail-qmail-smtpd 10MB QMAIL_SMTPD
/service/hvmail-qmail-smtpd2 10MB QMAIL_SMTPD2
/service/hvmail-qmail-smtpd3 10MB QMAIL_SMTPD3
/service/hvmail-redis 1MB REDIS
/service/hvmail-redis-np 1MB REDIS
/service/hvmail-rpc 1MB RPC
/service/hvmail-rspawn-limiter 1MB RSPAWN_LIMITER
/service/hvmail-simplemh 1MB SIMPLEMH
/service/hvmail-simplemh2 1MB SIMPLEMH
/service/hvmail-smtp-sink 1MB SMTP_SINK
/service/hvmail-tls 1MB AUTO_TLS
/service/hvmail-studio-worker 1MB STUDIO_WORKER

Adjusting any of the default rention levels listed above is a three step process:

  1. Identify the name that the configuration file uses for the service in the above table. For example, GreenArrow’s PostgreSQL service is named POSTGRES.

  2. Add an entry to the /var/hvmail/control/logconfig.multilog file of the following form:

    POSTGRES='s2097152 n10'
    

    In the above example, POSTGRES is the service name identified in the previous step, s2097152 tells multilog to allow each log file to grow to up to 2,097,152 bytes, or 2MB in size and n10 tells multilog to retain up to 10 of these log files. In other words - retain up to 20MB of PostgreSQL’s output.

  3. Restart the multilog service by using the svc command to apply the change:

    svc -t /service/hvmail-postgres/log/
    

Service Description

Automatic TLS

/var/hvmail/log/tls

This service handles GreenArrow’s TLS Certificate Configuration, allowing it to automatically obtain TLS certificates from Let’s Encrypt.

This service’s log contains information about TLS certificate registration and renewals.

Bounce Processor

/var/hvmail/log/bounce-processor

The bounce processor is responsible for processing bounce messages received at Bounce Mailboxes and spam complaint notifications received at Spam Complaint Mailboxes.

See the Bounce Processor Concepts page for more information.

This service’s log contains information about each bounce message that is processed.

Lite Bounce Processor

/var/hvmail/log/bounce-processor2

The Lite Bounce Processor is responsible for handling bounces and FBL complaints matched by the lite_bounce_processor_address and lite_fbl_processor_address directives.

See the Lite Bounce Processor documentation for more information.

Config Agent

/var/hvmail/log/config-agent

This service periodically writes internal configuration files for use by other GreenArrow services.

This service’s log is quiet except for error messages.

Dynamic Delivery Dispatcher

/var/hvmail/log/dd-dispatcher

This service handles execution of GreenArrow Dynamic Delivery.

This service’s log is quiet except for error messages.

Dynamic Delivery Log Reader

/var/hvmail/log/dd-logreader

This service handles parsing GreenArrow delivery logs for GreenArrow Dynamic Delivery.

This service’s log is quiet except for error messages.

Event Processor

/var/hvmail/log/event-processor

This service handles execution of the Legacy Event Processor.

This service logs:

  • Information about each event delivery attempt.
  • Performance statistics of event delivery.
Event Processor 2

/var/hvmail/log/event-processor2

This service handles execution of the Event Processor.

This service logs:

  • Diagnostics every 3 seconds.
  • Performance statistics of event delivery.
HTTPD

/var/hvmail/log/httpd

This service runs the Apache HTTP Server and is responsible for all incoming HTTP and HTTPS connections.

The multilog directory for this service is quiet except for error messages.

Apache’s log files are stored in /var/hvmail/apache/logs and provide information on each incoming HTTP and HTTPS request.

HTTPD Monitor

/var/hvmail/log/httpd-monitor

This service polls (and logs) the HTTPD service’s status page every 10 seconds. This is useful for troubleshooting Apache problems.

Logfile Agent

/var/hvmail/log/logfile-agent

This service polls the delivery attempt logs in order to populate the “send summary queue” files located in /var/hvmail/log/send-summary-queue.

This service’s log is quiet except for error messages.

Logfile Summary

/var/hvmail/log/logfile-summary

This service processes the data generated by the Logfile Agent and populates the “send summary” files located in /var/hvmail/log/send-summary.

This service’s log is quiet except for error messages.

Logfile Writer

/var/hvmail/log/logfile-writer

This service polls the delivery attempt logs in order to write to the Delivery Attempt Log and delivery_attempt events.

This service’s log is quiet except for error messages.

Passenger

/var/hvmail/log/passenger

This service is an internal app server used for GreenArrow’s UI and API. The HTTPD service acts as a proxy between the public internet and Passenger.

This service logs:

  • Information about each incoming request (under normal situations).
  • Error messages/conditions (under abnormal situations).
Postgres

/var/hvmail/log/postgres

This service runs the PostgreSQL database system.

The Postgres log can be fairly chatty with diagnostics and connection details.

Pure-FTPd Authentication Agent

/var/hvmail/log/pure-authd-studio

This service runs the authentication agent for the FTP server. This service is normally disabled for Engine-only installations.

This service’s log is quiet except for error messages.

Pure-FTPd

/var/hvmail/log/pure-ftpd

This service runs an FTP server. This FTP server is provided for GreenArrow Marketing Studio users to be able to upload files to be used by import features. This service is normally disabled for Engine-only installations.

This service’s log is quiet except for error messages.

Qmail - POP3

/var/hvmail/log/qmail-pop3d

This service runs GreenArrow’s POP3 server. This allows mail clients to download messages from local mailboxes.

This service logs details about incoming connections.

Qmail - SMTPD (port 25)

/var/hvmail/log/qmail-smtpd

This service runs GreenArrow’s port 25 SMTP server.

This service logs details about incoming connections.

Qmail - SMTPD2 (default port 587)

/var/hvmail/log/qmail-smtpd2

This service runs GreenArrow’s port 587 SMTP server.

This service logs details about incoming connections.

Qmail - SMTPD3 (default port 901)

/var/hvmail/log/qmail-smtpd3

This service runs GreenArrow’s port 901 SMTP server (this service is disabled by default).

This service logs details about incoming connections.

Queue - RAM

/var/hvmail/log/ram-qmail-send

This service runs GreenArrow’s RAM queue. This is the queue of first delivery attempts and is only persisted to RAM.

This service logs details about each delivery attempt.

Queue - Bounce

/var/hvmail/log/bounce-qmail-send

This service runs GreenArrow’s bounce queue. This is the queue from which bounce messages are delivered.

This service logs details about each delivery attempt.

Queue - Disk

/var/hvmail/log/disk-qmail-send

This service runs GreenArrow’s disk queue. This is the queue from which all non-first delivery attempts are tried and is persisted to disk at /var/hvmail/qmail-disk/queue.

This service logs details about each delivery attempt.

Redis

/var/hvmail/log/redis

This service runs the Redis in-memory data store.

This service logs details about its persistence to disk.

Redis NP

/var/hvmail/log/redis-np

This service runs the Redis in-memory data store. This service is configured to not persist to disk (it is used for performant ephemeral data that we don’t want to stream to disk).

This service’s log is quiet except for a startup message and error messages.

RPC

/var/hvmail/log/rpc

This service handles internal GreenArrow RPC requests.

This service’s log is quiet except for a startup message and error messages.

Rspawn Limiter

/var/hvmail/log/rspawn-limiter

This service is responsible for outgoing SMTP delivery and managing delivery throttles.

This service logs:

  • Details about each outgoing SMTP connection (depending on configuration such as log_smtp_connections).
  • Every 5 seconds a summary of diagnostics from the delivery engine.
  • Error messages.
SimpleMH

/var/hvmail/log/simplemh

This service is responsible for processing all email injected into Engine, except for the following:

  • Messages injected from GreenArrow Marketing Studio.
  • Messages injected from the HTTP Submission API.

This service’s log is quiet except for a startup message and error messages (unless the /var/hvmail/control/opt.simplemh.debug option is enabled).

SimpleMH 2

/var/hvmail/log/simplemh2

This service is responsible for processing email injected into Engine via the HTTP Submission API.

This service’s log is quiet except for a startup message and error messages (unless the /var/hvmail/control/opt.simplemh.debug option is enabled).

SMTP Sink

/var/hvmail/log/smtp-sink

This service is an SMTP server running on port 226. This SMTP server discards all messages injected to it.

This service logs details about each incoming connection.

Studio Worker

/var/hvmail/log/studio-worker

This service runs all of GreenArrow Marketing Studio’s background processing. This service is normally disabled for Engine-only installations.

The Studio Worker log can be fairly chatty with diagnostics and background job details.