Service Logs
- Table of Contents
- Log Locations
- Adjusting Log Retention
- Service Description
Most of GreenArrow’s services use multilog to log their output to a subdirectory of /var/hvmail/log/
.
For example, PostgreSQL logs data to the /var/hvmail/log/postgres/
directory. To tail the PostgreSQL log, run:
tail -F /var/hvmail/log/postgres/current | tai64nlocal
multilog
is similar to syslog in that it’s a standard for message logging. There are a few key differences to keep in mind:
-
Timestamps are recorded using external TAI64 format. You may decode these timestamps using tai64nlocal, as shown in the above
tail
example. These timestamps appear in both the beginning of each log entry and in the names of most files. -
multilog
stores the logs for each service within its own directory. Within each service’s directory, the most recently updated log file is namedcurrent
. Older logs have filenames starting with an external TAI64 format timestamp showing what time that log was last updated. -
syslog
is normally configured to retain logs for a specific amount of time.multilog
, by contrast, gets configured to retain a specific amount of data. For example, multilog could be configured to retain up to 100MB of data for a service.
Log Locations
Service Name | Log Location |
---|---|
Automatic TLS | /var/hvmail/log/tls |
Bounce Processor | /var/hvmail/log/bounce-processor |
Lite Bounce Processor | /var/hvmail/log/bounce-processor2 |
Config Agent | /var/hvmail/log/config-agent |
Dynamic Delivery Dispatcher | /var/hvmail/log/dd-dispatcher |
Dynamic Delivery Log Reader | /var/hvmail/log/dd-logreader |
Event Processor | /var/hvmail/log/event-processor |
Event Processor 2 | /var/hvmail/log/event-processor2 |
HTTPD | /var/hvmail/log/httpd |
HTTPD Monitor | /var/hvmail/log/httpd-monitor |
Logfile Agent | /var/hvmail/log/logfile-agent |
Logfile Summary | /var/hvmail/log/logfile-summary |
Logfile Writer | /var/hvmail/log/logfile-writer |
Passenger | /var/hvmail/log/passenger |
Postgres | /var/hvmail/log/postgres |
Pure-FTPd Authentication Agent | /var/hvmail/log/pure-authd-studio |
Pure-FTPd | /var/hvmail/log/pure-ftpd |
Qmail - POP3 | /var/hvmail/log/qmail-pop3d |
Qmail - SMTPD (port 25) | /var/hvmail/log/qmail-smtpd |
Qmail - SMTPD2 (default port 587) | /var/hvmail/log/qmail-smtpd2 |
Qmail - SMTPD3 (default port 901) | /var/hvmail/log/qmail-smtpd3 |
Redis | /var/hvmail/log/redis |
Redis NP | /var/hvmail/log/redis-np |
RPC | /var/hvmail/log/rpc |
Rspawn Limiter | /var/hvmail/log/rspawn-limiter |
SimpleMH | /var/hvmail/log/simplemh |
SimpleMH 2 | /var/hvmail/log/simplemh2 |
SMTP Sink | /var/hvmail/log/smtp-sink |
Studio Worker | /var/hvmail/log/studio-worker |
Adjusting Log Retention
The amount of data that multilog
retains for most services can be configured by updating the /var/hvmail/control/logconfig.multilog
file. Here’s a list of the services can have their log retention configured via this file:
Service | Default Retention | Service Name in Configuration File |
---|---|---|
/service/hvmail-bounce-processor |
10MB | BOUNCE_PROCESSOR |
/service/hvmail-bounce-processor2 |
10MB | BOUNCE_PROCESSOR2 |
/service/hvmail-config-agent |
1MB | CONFIG_AGENT |
/service/hvmail-dd-dispatcher |
1MB | DD_DISPATCHER |
/service/hvmail-dd-logreader |
1MB | DD_LOGREADER |
/service/hvmail-event-processor |
10MB | EVENT_PROCESSOR |
/service/hvmail-event-processor2 |
10MB | EVENT_PROCESSOR |
/service/hvmail-httpd |
1MB | HTTPD |
/service/hvmail-httpd-monitor |
1MB | HTTPD |
/service/hvmail-logfile-agent |
1MB | LOGFILE_AGENT |
/service/hvmail-logfile-summary |
1MB | LOGFILE_SUMMARY |
/service/hvmail-logfile-writer |
1MB | LOGFILE_WRITER |
/service/hvmail-passenger |
250MB | PASSENGER |
/service/hvmail-postgres |
20MB | POSTGRES |
/service/hvmail-pure-authd-studio |
1MB | PURE_AUTHD_STUDIO |
/service/hvmail-pure-ftpd |
1MB | PURE_FTPD |
/service/hvmail-qmail-pop3d |
1MB | QMAIL_POP3D |
/service/hvmail-qmail-smtpd |
10MB | QMAIL_SMTPD |
/service/hvmail-qmail-smtpd2 |
10MB | QMAIL_SMTPD2 |
/service/hvmail-qmail-smtpd3 |
10MB | QMAIL_SMTPD3 |
/service/hvmail-redis |
1MB | REDIS |
/service/hvmail-redis-np |
1MB | REDIS |
/service/hvmail-rpc |
1MB | RPC |
/service/hvmail-rspawn-limiter |
1MB | RSPAWN_LIMITER |
/service/hvmail-simplemh |
1MB | SIMPLEMH |
/service/hvmail-simplemh2 |
1MB | SIMPLEMH |
/service/hvmail-smtp-sink |
1MB | SMTP_SINK |
/service/hvmail-tls |
1MB | AUTO_TLS |
/service/hvmail-studio-worker |
1MB | STUDIO_WORKER |
Adjusting any of the default rention levels listed above is a three step process:
-
Identify the name that the configuration file uses for the service in the above table. For example, GreenArrow’s PostgreSQL service is named
POSTGRES
. -
Add an entry to the
/var/hvmail/control/logconfig.multilog
file of the following form:POSTGRES='s2097152 n10'
In the above example,
POSTGRES
is the service name identified in the previous step,s2097152
tellsmultilog
to allow each log file to grow to up to 2,097,152 bytes, or 2MB in size andn10
tellsmultilog
to retain up to 10 of these log files. In other words - retain up to 20MB of PostgreSQL’s output. -
Restart the
multilog
service by using thesvc
command to apply the change:svc -t /service/hvmail-postgres/log/
Service Description
Automatic TLS
/var/hvmail/log/tls |
This service handles GreenArrow’s TLS Certificate Configuration, allowing it to automatically obtain TLS certificates from Let’s Encrypt. This service’s log contains information about TLS certificate registration and renewals. |
Bounce Processor
/var/hvmail/log/bounce-processor |
The bounce processor is responsible for processing bounce messages received at Bounce Mailboxes and spam complaint notifications received at Spam Complaint Mailboxes. See the Bounce Processor Concepts page for more information. This service’s log contains information about each bounce message that is processed. |
Lite Bounce Processor
/var/hvmail/log/bounce-processor2 |
The Lite Bounce Processor is responsible for handling bounces and FBL complaints matched by the lite_bounce_processor_address and lite_fbl_processor_address directives. See the Lite Bounce Processor documentation for more information. |
Config Agent
/var/hvmail/log/config-agent |
This service periodically writes internal configuration files for use by other GreenArrow services. This service’s log is quiet except for error messages. |
Dynamic Delivery Dispatcher
/var/hvmail/log/dd-dispatcher |
This service handles execution of GreenArrow Dynamic Delivery. This service’s log is quiet except for error messages. |
Dynamic Delivery Log Reader
/var/hvmail/log/dd-logreader |
This service handles parsing GreenArrow delivery logs for GreenArrow Dynamic Delivery. This service’s log is quiet except for error messages. |
Event Processor
/var/hvmail/log/event-processor |
This service handles execution of the Legacy Event Processor. This service logs:
|
Event Processor 2
/var/hvmail/log/event-processor2 |
This service handles execution of the Event Processor. This service logs:
|
HTTPD
/var/hvmail/log/httpd |
This service runs the Apache HTTP Server and is responsible for all incoming HTTP and HTTPS connections. The multilog directory for this service is quiet except for error messages. Apache’s log files are stored in |
HTTPD Monitor
/var/hvmail/log/httpd-monitor |
This service polls (and logs) the HTTPD service’s status page every 10 seconds. This is useful for troubleshooting Apache problems. |
Logfile Agent
/var/hvmail/log/logfile-agent |
This service polls the delivery attempt logs in order to populate the “send summary queue” files located in This service’s log is quiet except for error messages. |
Logfile Summary
/var/hvmail/log/logfile-summary |
This service processes the data generated by the Logfile Agent and populates the “send summary” files located in This service’s log is quiet except for error messages. |
Logfile Writer
/var/hvmail/log/logfile-writer |
This service polls the delivery attempt logs in order to write to the Delivery Attempt Log and delivery_attempt events. This service’s log is quiet except for error messages. |
Passenger
/var/hvmail/log/passenger |
This service is an internal app server used for GreenArrow’s UI and API. The HTTPD service acts as a proxy between the public internet and Passenger. This service logs:
|
Postgres
/var/hvmail/log/postgres |
This service runs the PostgreSQL database system. The Postgres log can be fairly chatty with diagnostics and connection details. |
Pure-FTPd Authentication Agent
/var/hvmail/log/pure-authd-studio |
This service runs the authentication agent for the FTP server. This service is normally disabled for Engine-only installations. This service’s log is quiet except for error messages. |
Pure-FTPd
/var/hvmail/log/pure-ftpd |
This service runs an FTP server. This FTP server is provided for GreenArrow Marketing Studio users to be able to upload files to be used by import features. This service is normally disabled for Engine-only installations. This service’s log is quiet except for error messages. |
Qmail - POP3
/var/hvmail/log/qmail-pop3d |
This service runs GreenArrow’s POP3 server. This allows mail clients to download messages from local mailboxes. This service logs details about incoming connections. |
Qmail - SMTPD (port 25)
/var/hvmail/log/qmail-smtpd |
This service runs GreenArrow’s port 25 SMTP server. This service logs details about incoming connections. |
Qmail - SMTPD2 (default port 587)
/var/hvmail/log/qmail-smtpd2 |
This service runs GreenArrow’s port 587 SMTP server. This service logs details about incoming connections. |
Qmail - SMTPD3 (default port 901)
/var/hvmail/log/qmail-smtpd3 |
This service runs GreenArrow’s port 901 SMTP server (this service is disabled by default). This service logs details about incoming connections. |
Queue - RAM
/var/hvmail/log/ram-qmail-send |
This service runs GreenArrow’s RAM queue. This is the queue of first delivery attempts and is only persisted to RAM. This service logs details about each delivery attempt. |
Queue - Bounce
/var/hvmail/log/bounce-qmail-send |
This service runs GreenArrow’s bounce queue. This is the queue from which bounce messages are delivered. This service logs details about each delivery attempt. |
Queue - Disk
/var/hvmail/log/disk-qmail-send |
This service runs GreenArrow’s disk queue. This is the queue from which all non-first delivery attempts are tried and is persisted to disk at This service logs details about each delivery attempt. |
Redis
/var/hvmail/log/redis |
This service runs the Redis in-memory data store. This service logs details about its persistence to disk. |
Redis NP
/var/hvmail/log/redis-np |
This service runs the Redis in-memory data store. This service is configured to not persist to disk (it is used for performant ephemeral data that we don’t want to stream to disk). This service’s log is quiet except for a startup message and error messages. |
RPC
/var/hvmail/log/rpc |
This service handles internal GreenArrow RPC requests. This service’s log is quiet except for a startup message and error messages. |
Rspawn Limiter
/var/hvmail/log/rspawn-limiter |
This service is responsible for outgoing SMTP delivery and managing delivery throttles. This service logs:
|
SimpleMH
/var/hvmail/log/simplemh |
This service is responsible for processing all email injected into Engine, except for the following:
This service’s log is quiet except for a startup message and error messages
(unless the |
SimpleMH 2
/var/hvmail/log/simplemh2 |
This service is responsible for processing email injected into Engine via the HTTP Submission API. This service’s log is quiet except for a startup message and error messages
(unless the |
SMTP Sink
/var/hvmail/log/smtp-sink |
This service is an SMTP server running on port 226. This SMTP server discards all messages injected to it. This service logs details about each incoming connection. |
Studio Worker
/var/hvmail/log/studio-worker |
This service runs all of GreenArrow Marketing Studio’s background processing. This service is normally disabled for Engine-only installations. The Studio Worker log can be fairly chatty with diagnostics and background job details. |