Email Address Retention

Table of Contents
About This Document
What’s not in this document
Do not delete or modify files
Email addresses stored by GreenArrow Engine
Email addresses stored by GreenArrow Studio
Various queues and temporary logfiles

About This Document

This document describes where GreenArrow stores recipient and subscriber email addresses and the retention policy for each of those locations.

What’s not in this document

This document does not cover everywhere that a GreenArrow user can configure an email address.

For example:

Campaign From Email
Studio user email addresses
Segmentation rules that can include email addresses
Seed Lists
Files uploaded to GreenArrow via HTTP or FTP

Do not delete or modify files

IMPORTANT: This document lists the locations of files on disk. Unless specifically indicated, deleting or modifying these files directly is not supported and may break the software.

Email addresses stored by GreenArrow Engine

MTA delivery logs

Delivery logs record the recipient email address for every delivery attempt, regardless of whether the delivery is successful.

Retention:

These logs automatically get rotated out as new data comes in.

The hvmail_set log_disk_space command is used to control this behavior.

Locations on disk:

/var/hvmail/log/ram-qmail-send
/var/hvmail/log/bounce-qmail-send
/var/hvmail/log/disk-qmail-send

Permitted modification:

No modifications are permitted.

Web server (Apache) logs

Apache logs are written for each web request and log the URI of each web request.

SimpleMH click, open, and List-Unsubscribe URIs may include a Base64 encoded copy of the email address and are logged to the web server logs when these URIs are loaded.

Some API requests may include a subscriber email address in the URI.

Retention:

This data automatically get rotated out as it ages, and new data comes in. This data is retained for up to 5 weeks by default.

There are no publicly documented retention settings for this category, but GreenArrow technical support can increase or decrease the retention period upon request.

Location on disk:

/var/hvmail/apache/logs

Permitted modification:

You may delete files matching these patterns that were last modified at least 24 hours ago:

/var/hvmail/apache/logs/access_log-*
/var/hvmail/apache/logs/error_log-*
/var/hvmail/apache/logs/ssl_request_log-*

Bounce processor logs

The bounce processor has a logfile which contains email addresses that recently received bounce, spam complaints, and List-Unsubscribe header mailto method unsubscribes.

Retention:

The Service Logs documentation contains instructions on adjusting how much data is retained in these logs before being rotated.

Location on disk:

/var/hvmail/log/bounce-processor

Permitted modification:

No modifications are permitted.

Event Processor logs

The Event Processor writes all email addresses that it delivers events about to a log file as it delivers them.

Retention:

The Service Logs documentation contains instructions on adjusting how much data is retained before being rotated.

Location on disk:

/var/hvmail/log/event-processor

Permitted modification:

No modifications are permitted.

Other service logs

Other services like PostgreSQL can also log email addresses, especially when something unusual happens, like a query taking longer than usual to run, or an error occurs.

Retention:

Each service’s log file has a default retention period.

Most service logs have a configurable retention setting described in the Service Logs document.

Location on disk:

See the Service Logs document.

Permitted modification:

No modifications are permitted.

SimpleMH and Studio Remote List recipient addresses

Both SimpleMH (unless Stateless Metadata Event Tracking is enabled) and Studio Remote Lists write recipient email addresses to files as they send mail.

These files are used to connect unique identifiers in click, open, and unsubscribe URIs and bounce messages to email addresses.

Retention:

The /var/hvmail/control/opt.click_bounce_emaillist_retention_days file is used to control the retention period of these files.

Location on disk:

/var/hvmail/var/clickthrough-tracking-emaillist

Permitted modification:

No modifications are permitted.

SimpleMH click and open tracking

SimpleMH click and open tracking write recipient email addresses to a PostgreSQL table each time a click or open occurs.

Retention:

There is no retention setting for this category.

Location on disk:

PostgreSQL table: clickthrough_clicks

Permitted modification:

Old rows may be deleted, but this will cause the click and open statistics not to represent the clicks and opens that were deleted.

Bounce bad addresses

All email addresses that:

are detected as bad through bounces (hard and/or soft bounces)
register a spam complaint
are sent to through SimpleMH and unsubscribe using a SimpleMH unsubscribe link
are sent to through SimpleMH and unsubscribe through the List-Unsubscribe header mechanism
are sent to through Studio and unsubscribe using the List-Unsubscribe header mailto mechanism

are logged in the database.

Retention:

There is no retention setting for this category.

Location on disk:

PostgreSQL table: bounce_bad_addresses

Permitted modification:

Old rows may be deleted, but this will cause SimpleMH Bad Address Suppression, if configured, to not suppress email addresses in the deleted rows.

Bad Address Suppression intermediate file

If SimpleMH Bad Address Suppression is enabled, then a list of the email addresses that are to be suppressed is written to disk. The Bad Address Suppression mechanism consults these files instead of the database for speed.

Retention:

There is no retention setting for this category, but Bad Address Suppression can be turned off, and then these files may be deleted.

Location on disk:

/var/hvmail/var/simplemh-bad-addresses.cdb*

Permitted modification:

If SimpleMH Bad Address Suppression is disabled, then you may delete these files.

Bounce processor repeat bounce tracking

The bounce processor tracks email addresses that have recently received soft bounces as part of the Repeat Bounce Counting System.

Retention:

The bounce.max_gap_days configuration controls how long the repeat bounce processing system holds onto subscriber email addresses from the most recent soft bounce.

Location on disk:

/var/hvmail/var/bounce_processor_repeat_tracker.cdb

PostgreSQL tables:

bounce_repeat_state
bounce_repeat_tracker
bounce_repeat_tracker_new

Permitted modification:

No modifications are permitted.

Bounce processor could not process directories

The bounce processor retains copies of messages that were delivered into the bounce or spam complaint mailbox but could not be processed.

Retention:

GreenArrow retains up to 2GB of bounces and 2GB of spam complaints for up to 30 days. This is not configurable.

Location on disk:

/var/hvmail/maildata/Maildir-bounce/.INBOX.CouldNotProcess/new
/var/hvmail/maildata/Maildir-feedback/.INBOX.CouldNotProcess/new

Permitted modification:

Files matching these globs may be deleted:

/var/hvmail/maildata/Maildir-bounce/.INBOX.CouldNotProcess/new/*
/var/hvmail/maildata/Maildir-bounce/.INBOX.CouldNotProcess/cur/*
/var/hvmail/maildata/Maildir-feedback/.INBOX.CouldNotProcess/new/*
/var/hvmail/maildata/Maildir-feedback/.INBOX.CouldNotProcess/cur/*

Events

The Event Notification System contains a queue of events (clicks, opens, delivery attempts, bounces, unsubscribes, etc) waiting to be delivered. Each event record contains an email address.

It is possible, but not recommended, to configure the event notification system so that events are retained indefinitely using the leave_in_queue destination type as defined in the Event Processor configuration.

Retention:

No long-term retention as long as the Event Processor is properly configured, and the leave_in_queue type is not used.

Location on disk:

PostgreSQL table: events

Permitted modification:

Rows may be deleted from the events table, but then the events represented by those rows won’t be delivered.

ram-queue and bounce-queue when MTA is stopped

Most messages get written to either the ram-queue or bounce-queue for their first delivery attempt. These queues exist in RAM.

However, when the MTA is stopped any messages in these queues are written to disk. They’re removed from disk and moved back to the ram-queue and bounce-queue when the MTA is restarted.

Retention:

See above.

Starting the MTA will cause these queues to be cleared.

Location on disk:

/var/hvmail/qmail-ram/savedqueue/
/var/hvmail/qmail-bounce/savedqueue/

Permitted modification:

When the MTA is stopped you may delete these queues with this command:

rm -rf /var/hvmail/qmail-ram/savedqueue/saved.* /var/hvmail/qmail-bounce/savedqueue/saved.*

disk-queue

The disk-queue is where GreenArrow stores messages that could not be delivered on their first attempt.

Messages may be written in batches. Each batch is removed once all messages in it are delivered or bounce.

Retention:

See the following documents:

Location on disk:

/var/hvmail/qmail-disk/queue/

Permitted modification:

No modifications are permitted.

SimpleMH archived messages

SimpleMH archives sample messages when the Archive a Sample of Messages option is turned on for a Mail Class. This option is off by default.

Retention:

No messages are retained when the Archive a Sample of Messages feature is disabled.

There is no retention setting for archived messages once they are archived.

Location on disk:

PostgreSQL table: archived_message

Permitted modification:

Rows may be deleted from the archived_message table.

Send Summary files

Send Summary files are used by Send Statistics.

Retention:

See this document for how to move Send Summary files.

There is no option for deleting data in this category.

Location on disk:

/var/hvmail/log/send-summary

Permitted modification:

Files matching the glob /var/hvmail/log/send-summary/*/*.db, which have not been updated in 30 days, may be deleted. This will cause the Engine statistics for the matching “sends” to become unavailable, and Studio statistics for the matching “sends” to be incomplete.

User mailboxes

Messages received by Incoming Email Domain users.

Retention:

Each message is stored until it’s deleted. Deletion are performed with a POP3 or IMAP client.

Deleting an entire mailbox prevents new messages from being delivered to the mailbox, but does not delete any messages already delivered. If you wish to delete a mailbox, and all of its messages, then you should:

Delete all messages with an email client.
Delete the mailbox.

Location on disk:

/var/hvmail/maildata/userid-*

Permitted modification:

No modifications are permitted.

Backups

Backups contain most of the email address locations documented in this file.

See Managed Backups for GreenArrow’s retention policies or Unmanaged Backups for how to configure backups not managed by GreenArrow.

Email addresses stored by GreenArrow Studio

Campaign statistics for Remote Lists

For email sent through GreenArrow Studio using Remote Lists, all email addresses that have the following activity are retained along with the record of that activity:

were sent to
click
open
bounce
unsubscribe
spam complaint
skipped sending due to a Special Sending Rule

(To be clear: the records for statistics for email not sent through Remote Lists do not contain email addresses.)

Retention:

See this document for how to configure Studio statistics data retention.

Location on disk:

PostgreSQL tables:

s_stat_sents_rl
s_stat_opens
s_stat_clicks
s_stat_bounces
s_stat_unsubs
s_stat_scomps
s_stat_skips

Permitted modification:

No modifications are permitted.

Logfiles for failed campaigns

When a GreenArrow Studio campaign encounters an error, a logfile of the sending is retained.

Retention:

These logfiles are retained for up to one week.

There is no setting to change this behavior.

Location on disk:

/var/hvmail/var/tmp/greenarrow-studio-sending-log-*

Permitted modification:

Files matching the shell glob /var/hvmail/var/tmp/greenarrow-studio-sending-log-* that are no longer open may be deleted.

Recipient lists for paused campaigns

GreenArrow Studio prepares the recipient list for a campaign at the beginning of the campaign and stores it to the database.

Retention:

If the campaign starts sending and is then paused, the recipient list will be retained until the campaign is resumed and completes, or the campaign is canceled.

Permitted modification:

No modifications are permitted.

Temporary files used during subscriber imports and exports

GreenArrow Studio writes email addresses temporarily to disk during subscriber imports and exports.

Retention:

If an import or export is paused, this data is retained until the import or export is resumed and completes.

When an import or export finished (successfully or in failure), this data is retained for one week.

There is no option for altering the above behavior.

Location on disk:

/var/hvmail/var/studio-tmp/cdb-*
/var/hvmail/var/studio-data/subscriber_imports/*/work_unit*.json
/var/hvmail/var/studio-data/subscriber_imports/*/*/work_unit*.json

Files matching this glob might remain on disk indefinitely:

/var/hvmail/var/studio-tmp/cdb-*

Permitted modification:

Files matching this glob that were last modified at least 24 hours ago may be deleted:

/var/hvmail/var/studio-tmp/cdb-*

Temporary files used during suppression list imports

GreenArrow Studio writes email addresses temporarily to disk during suppression list imports.

Retention:

If an import is paused, this data is retained until the import is resumed and completes.

When an import is finished (successfully or in failure), this data is retained for one week.

There is no option for altering the above behavior.

Location on disk:

/var/hvmail/var/studio-data/suppressed_address_imports/

Permitted modification:

No modifications are permitted.

Special Sending Rule Errors

When a Special Sending Rule generates errors, error messages for some of the subscribers that generated errors are entered in the campaign history log. These error messages contain subscriber email addresses.

The campaign history log is retained indefinitely.

Retention:

There is no option for altering the above behavior.

Location on disk:

PostgreSQL table: s_audit_log_entries

Permitted modification:

Rows may be deleted from the table, but once deleted they will no longer appear in the campaign history log.

Subscriber records

GreenArrow Studio stores subscriber records in PostgreSQL database tables, including subscriber email addresses.

Retention:

This data is retained until the mailing list is deleted.

There is no option for altering the above behavior.

Location on disk:

PostgreSQL tables:

s_subscribers
s_subscribers_*
s_subscriber_statuses

Permitted modification:

No modifications are permitted.

Rails application server request logfiles

GreenArrow’s web application server logs requested URIs, and those URIs can contain subscriber email addresses.

Retention:

This data automatically get rotated out as it ages, and new data comes in. This data is retained for up to 15 days.

There is no option for altering the above behavior.

Location on disk:

/var/hvmail/log/studio/*log*

Permitted modification:

You may delete files matching this pattern that were last modified at least 24 hours ago:

/var/hvmail/log/studio/*log-*

Rails application server exception reports

GreenArrow’s web application server logs exceptions when they occur. These logs could contain subscriber email addresses.

Retention:

This data is retained indefinitely.

There is no option for altering the above behavior.

Location on disk:

/var/hvmail/log/studio/exceptions/

Permitted modification:

Individual files in /var/hvmail/log/studio/exceptions/ that were last modified at least 24 hours ago may be deleted.

Suppression lists

GreenArrow Studio stores suppression lists in PostgreSQL database tables. Suppressed addresses may be stored in plaintext or as an MD5 or SHA256 hash. Each suppressed address may be a fully formed email address, or a portion of an email address, such as a domain name.

Recipient addresses may be automatically added to suppression lists when bounces, spam complaints, or unsubscribes ocur if the Automatic Suppression Lists feature is enabled.

Addresses can also be added to suppression lists via API call or GreenArrow’s web interface.

Retention:

This data is retained until either an individual email address or an entire suppression list is deleted.

There is no option for altering the above behavior.

Location on disk:

PostgreSQL table: s_suppressed_address

Permitted modification:

No modifications are permitted.

Subscriber email addresses awaiting confirmed-opt-in confirmation

GreenArrow Studio stores email addresses that are awaiting confirmed-opt-in confirmation.

Retention:

This data is retained until either:

The subscriber completes the confirmed-opt-in process.
60 days elapse.

There is no option for altering the above behavior.

Location on disk:

PostgreSQL table: s_pending_subscribers

Permitted modification:

No modifications are permitted.

Various queues and temporary logfiles

There are a number of queues in which recipient email addresses are retained for short periods of time while being processed. (This is not a complete list.)

Location on disk:

/var/hvmail/data/redis
/var/hvmail/log/send-summary-queue/
/var/hvmail/maildata/Maildir-bounce/new
/var/hvmail/maildata/Maildir-bounce/cur
/var/hvmail/maildata/Maildir-feedback/new
/var/hvmail/maildata/Maildir-feedback/cur
/var/hvmail/var/tmp/greenarrow-studio-sending-log-*
/var/hvmail/var/tmp/greenarrow-studio-transactional-email-log-*

PostgreSQL table: bounce_remove_queue