Restoring Backups

Table of Contents
Intro
Prerequisites
Non-GreenArrow Files
Restoration Procedure
Final Testing

Intro

This document describes how to restore a GreenArrow installation from backups.

Prerequisites

The restoration procedure described on this page has a number of prerequisites:

A good set of backups created by GreenArrow’s unmanaged backup script.

Each time a backup is taken, the results get written to the backup directory’s greenarrow-backup.log file. The backup directory is /var/backup by default.

If the backup completed without any detected errors, then the last line of the log will read:
```
The backup is finished. No errors were detected.
```
If any errors were detected, then the last line of the log will read:
```
Error(s) were detected. See above.
```
The backup script will output the following text immediately after each error that it detects:
```
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! WARNING !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
There was an error running backups. Please review the above output for
details and contact GreenArrow technical support if you have any questions.
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! WARNING !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
```
An experienced Linux systems administrator who can execute the restoration procedure.

The backups must be of GreenArrow Engine 4.1.144-0 or later. If an older version was backed up, then please contact GreenArrow technical support to request a restore.

Here’s a bash script you can run to check what version was backed up to the /var/backup/ directory:

if grep -q greenarrow-engine /var/backup/greenarrow-packages.txt; then
  grep greenarrow-engine /var/backup/greenarrow-packages.txt | grep -v greenarrow-engine-common
else
  grep greenarrow /var/backup/greenarrow-packages.txt | grep -v greenarrow-support || echo "I was unable to determine what version of GreenArrow is installed."
fi

Alternatively, if you haven’t run GreenArrow’s unmanaged backup script yet, you can check what version is currently installed by running:

rpm -q greenarrow 2> /dev/null || dpkg -l greenarrow 2> /dev/null || rpm -q greenarrow-engine 2> /dev/null || dpkg -l greenarrow-engine 2> /dev/null | grep "^ii " || echo "GreenArrow is either not installed, or is a pre-4.1.144-0 release."

The server should have enough disk space to hold the restored data. If the specs are identical to or greater than the server that the backup was taken on, and you didn’t have any disk space issues prior to needing the restore, then you’re covered. Otherwise, you can get an approximation of how much disk space will be needed to restore GreenArrow’s backups in two steps:
1. cd into the directory that your backups are stored in, and run the gzip -l greenarrow-files.tar.gz command to estimate how much space will be required for non-PostgreSQL files.
  
  The uncompressed column’s value will show approximately how much disk space, in bytes, will be used by these files.
2. cd into the directory that your backups are stored in, and run the /var/hvmail/postgres/default/bin/pg_restore greenarrow.dump | wc -c command to estimate how much disk space, in bytes, will be required for PostgreSQL.
The figures that are provided by the above commands do not account for a number of things, including GreenArrow’s packages, the disk space required by PostgreSQL’s indexes, and deferred email, so you should allow for significantly more disk space usage than they indicate you need. You might allow for double the disk space, for example.
Running commands on a tmux or screen session is recommended, especially for steps that may take a long time to complete. Restoring the database or files may take a long time and SSH disconnects can cause the restore process to fail. Using a tmux or screen session will prevent SSH disconnects from breaking a restore in progress.

GreenArrow technical support can perform a restore using just the files created by our unmanaged backup script, so if you’re missing any of this document’s other prerequisites, contact us for details.

Non-GreenArrow Files

GreenArrow’s unmanaged backup script backs up the /etc/ and /var/spool/cron/ directories to the operating-system-files.tar.gz file. These directories are not normally needed to perform a GreenArrow restore, but we found that these backups are sometimes useful. You’re welcome to use this file, but this document doesn’t go into details on how to do so.

Restoration Procedure

Some of the steps below are Linux distribution specific. For the sake of brevity we refer to Red Hat based Linux distributions, such as CentOS, Red Hat Enterprise Linux or Scientific Linux as “Red Hat”, and Debian-based distributions, such as Ubuntu and Debian to “Debian”.

Here’s how to restore GreenArrow from backups:

Log in to the server that you wish to restore to as the root user using the bash shell.
Install packages that are used by the restore process:

On Red Hat systems, run:
```
yum install ruby
```
On Debian systems, run:
```
apt-get install ruby wget gnupg
```
Copy the files created by GreenArrow’s unmanaged backup script to a location that’s accessible to the operating system, such as /var/backup/, then cd into that directory.
View the greenarrow-dereferenced-symlinks.txt file.

If the file is empty, then move onto the next step.

If the file lists any symlinks, then decide whether you wish to re-create them. The symlinks that are listed in this file do not exist on a default GreenArrow installation but were present at the time that the backup was taken. If you do not re-create the symlinks, then they will be dereferenced later when we run tar to restore them. tar will restore the backed up files either way. The decision of whether to create these symlinks just impacts where the data that the symlinks pointed to gets written.
Check the postgresql-tablespaces.txt file for any tablespaces which specify a location.

Here’s an example of what this file might look like. In this example, the tablespace named “data2” is the only one which has a location (/var/hvmail/postgres/default/data2) specified:
```
                     List of tablespaces
    Name    |   Owner    |              Location
------------+------------+------------------------------------
 data2      | greenarrow | /var/hvmail/postgres/default/data2
 pg_default | postgres   |
 pg_global  | postgres   |
(3 rows)
```
If there are no tablespaces which specify a location, then you can move onto the next step.

If there are any tablespaces which specify a location, then re-create their directories, and set their owner equal to the hvpostgres user and group, with 0700 permissions. For example:
```
mkdir /var/hvmail/postgres/default/data2
chown hvpostgres:hvpostgres /var/hvmail/postgres/default/data2
chmod 0700 /var/hvmail/postgres/default/data2
```
/var/hvmail/postgres/default is normally a symlink. If it needs created during this step, then you can check on where it pointed to by viewing the postgresql-symlink.txt file in your backup directory. If the file is missing, then /var/hvmail/postgres/default was not a symlink.
Complete the Prerequisites section of our Installation Guide. If you get to the “Install GreenArrow” step in the Installation Guide, you’ve gone too far.
Install GreenArrow’s packages:
```
ruby -e "$(curl -fsSL http://git.drh.net/pub/install_greenarrow_packages)" \
  -- --detect-from-backup=.
```
The above command will install the same version of GreenArrow packages that were used to make the backup.
GreenArrow packages make a number of changes to environment settings. To activate the changes, either log out, then back into the server or run the following commands. If you’re logged into the server in multiple windows, then you’ll need to complete this step in each window:
```
. /etc/profile.d/greenarrow-engine.sh
. /etc/profile.d/greenarrow-ruby.sh
. /etc/profile.d/greenarrow-studio.sh
. /etc/profile.d/greenarrow-passenger.sh
```
Complete the Tune the operating system section of our Installation Guide.
Put the system into a blank state ready for a restore:
```
/var/hvmail/libexec/hvmail_clear_data_for_restore
```
If desired, switch to Persistence Path Mode for the restored system.

Regardless of whether the backup was taken from a system that used Persistence Path Mode or not, you can now decide to use Persistence Path Mode. If you choose to, run the following command, replacing $PERSISTENCE_PATH with the absolute path you wish to use for your persisted data.

If you do not run the following command, the restored system will use Traditional Mode, where GreenArrow’s data files are mixed into a variety of paths within the /var/hvmail hierarchy.
```
mkdir -p "$PERSISTENCE_PATH"
[ -e postgresql-symlink.txt ] && \
  ln -s $(cat postgresql-symlink.txt) /var/hvmail/postgres/default
/var/hvmail/libexec/greenarrow_convert_to_persistence_path "$PERSISTENCE_PATH"
```

Restore GreenArrow’s data files. We recommend running this process in a tmux or screen session:

tar xzfp greenarrow-files.tar.gz -C /
mv /var/hvmail/postgres/default{,.restored-config-files}

If the backup is from an installation that used Postgres 8.3, then there could be a compatibility issue. The issue only exists if all of the following are true:
1. The backup is being restored to a Red Hat 8 based distribution, like CentOS 8.
2. The content of the backup’s postgresql-symlink.txt file include the string 8.3.
3. The /var/hvmail/postgres/default.restored-config-files/data/pg_hba.conf file authorizes clients with hostssl.
If all of the above is true, then the PostgreSQL clients who are authorized using hostssl will be unable to connect. To complete the restoration process, do one of the following:
1. Comment out hostssl authorizations with a #.
2. Replace hostssl with an option that doesn’t require SSL/TSL, such as host.
After the restoration is complete, you can optionally contact GreenArrow technical support and request a PostgreSQL upgrade. Upgrading to PostgreSQL 9.5 or 16 allows hostssl to be used again.
If the server that you’re restoring to has a different amount of RAM than the server that backups were taken on, then:
1. Tune the /var/hvmail/control/postgres.conf file as described in the documentation on tuning PostgreSQL memory usage. PostgreSQL is not running yet, so skip the steps that involve restarting and testing PostgreSQL.
2. Adjust the size of GreenArrow’s RAM and Bounce queues as described in the documentation on changing the amount of RAM allocated to mail queues. The GreenArrow queues are not started yet, so skip the hvmail_init restart step which restarts the queues.
If you previously tuned PostgreSQL’s memory settings and this required updating the kernel.shmmax and kernel.shmall sysctl parameters on the backed up system, also update those parameters on the new system or re-tune PostgreSQL. See the documentation on tuning PostgreSQL memory usage.
Comment out any lines in /var/hvmail/control/postgres.conf that contain ssl and listen_addresses.
Restore PostgreSQL. Depending on the size of the backup, and the speed of your hardware, this step could take anywhere from minutes to hours to complete. We recommend running this process in a tmux or screen session:
```
hvmail_ini_postgres --restore-from-backup .
```
If you receive an unable to connect to the database error, then this may mean that not enough memory could be allocated. If this happens to you, then try tuning sysctl as described in the documentation on tuning PostgreSQL memory usage.

If you are restoring due to a database corruption, it may be helpful to pause the restore process between the insertion of data and the creation of indexes (which might fail if your database is corrupted). To do this, instead run the following command:
```
hvmail_ini_postgres --restore-from-backup . --wait-before-creating-indexes
```
This will cause the hvmail_ini_postgres command to wait for you to type continue before proceeding to create indexes. This gives you an opportunity to connect to Postgres and correct the database corruption before continuing to index creation.
Prevent Studio from resuming sends automatically when we start GreenArrow:
```
touch /service/hvmail-studio-worker/down
```
This step is not needed for Let’s Encrypt certificates that are configured automatically.

If you’re using the Legacy Let’s Encrypt configuration method, then remove all that method’s certificates from the HTTP TLS configuration file. These certificates are stored outside of GreenArrow’s directories, so they are not migrated by this procedure. Take note of which certificates you’ve removed in case you wish to add them back in later. There’s a step later in this procedure for doing that.

Start GreenArrow:

systemctl start greenarrow.service
systemctl enable greenarrow.service

Check whether greenarrow.conf has dns_cache_service_run set:
```
grep dns_cache_service_run /var/hvmail/control/greenarrow.conf
```
1. If dns_cache_service_run is set to true or yes, then configure the /etc/resolv.conf file so that the only nameserver line is set to 127.0.0.1. For example:
```
echo "nameserver 127.0.0.1" > /etc/resolv.conf
```
2. If dns_cache_service_run is set to false or no, or is not specified, then configure the /etc/resolv.conf file to query one or more recursive DNS resolvers. For example, if you would like to query the Google and Cloudflare public DNS servers, you could populate the file with:
```
echo "nameserver 8.8.8.8
nameserver 8.8.4.4
nameserver 1.1.1.1" > /etc/resolv.conf
```
Verify that different nameservers aren’t configured outside of /etc/resolv.conf. If the commands below return any output, then verify that it matches your desired configuration:

On Red Hat systems, run:
```
find /etc/sysconfig/network-scripts/ifcfg-* /etc/sysconfig/network -type f | xargs grep 'DNS=' | grep -v 'PEERDNS='
```
On Debian systems, run:
```
grep dns-nameservers /etc/network/interfaces
```
If you use GreenArrow Studio, then go to Studio’s web interface, and cancel any campaigns that you do not wish to be sent. When making this decision, think about which campaigns may have finished sending after the backup that you just restored was taken. The idea here is to avoid sending any campaigns twice.
If you’re not using GreenArrow Studio’s autoresponders then skip this step.

If you use GreenArrow Studio’s autoresponders, then decide what to do with the autoresponders which were in the queue to be sent before the GreenArrow server went offline. These are autoresponders which may have been sent already, but you don’t know for certain. You have two options:

The first option is to let the autoresponders get sent out after the restore, potentially as duplicates of what was previously sent. If you select this option, then you can move onto the next step.

The second option is to NOT allow the autoresponders to be sent out. This is the safer option, in terms of not annoying subscribers with duplicated emails, but the downside is that it may cause some subscribers not to receive any copies of autoresponder(s) which they were scheduled to be sent.

If you choose the second option, then you’ll need to delete some rows from PostgreSQL’s s_autoresponder_reactions table. Here’s how to do that:
1. Login to PostgreSQL:
```
/var/hvmail/postgres/default/bin/psql -U greenarrow
```
2. Use a SELECT TIMESTAMP query to enter the time and date that the event that caused the need for a restore occurred in UTC, and verify that PostgreSQL understands what you entered. For example:
```
SELECT TIMESTAMP 'January 1, 2050 12:00 PM UTC';
```
  The above command did not make any changes. Its purpose is to verify that you have the correct time and date syntax.
3. Begin a transaction:
```
BEGIN;
```
4. Delete all rows in the s_autoresponder_reactions table that have their delayed_until value set to less than or equal to the time and date that the event that caused the need for a restore occurred in UTC. For example, if your GreenArrow server went offline on January 1, 2050, at 12:00 PM UTC, you’d run:
```
DELETE FROM s_autoresponder_reactions WHERE delayed_until <= 'January 1, 2050 12:00 PM UTC';
```
5. If the steps above completed without error, commit the transaction. This applies changes:
```
COMMIT;
```
  If there were any issues the steps above, rollback the transaction. This will undo any changes that were made to PostgreSQL since the BEGIN; query was run:
```
ROLLBACK;
```
6. Log out of PostgreSQL:
```
\q
```
This step is not needed for Let’s Encrypt certificates that are configured automatically.

Optionally, add Legacy Let’s Encrypt certificates to the HTTP Server’s TLS configuration.
Check if the /var/hvmail/control/UPGRADE_BLOCKER.txt file exists:
```
cat /var/hvmail/control/UPGRADE_BLOCKER.txt
```
This file is only present on GreenArrow installations where non-standard steps are required to perform updates of GreenArrow’s software.

If the file is missing, then you can move onto the next step.

If the file is present, then contact GreenArrow technical support to have whatever extra steps are needed completed.

If you use GreenArrow Studio, then resume its sends:

rm -f /service/hvmail-studio-worker/down
svc -u /service/hvmail-studio-worker

Final Testing

Verify that some common misconfigurations are not present. Each test gets a simple pass/fail result:
```
hvmail_check_config
```
Verify that all services are running normally. Any services with an abnormal state will be shown in red:
```
service greenarrow status
```
The hvmail-qmail-smtpd3 and hvmail-dnscache services are down by default. All other services should be reported as up unless you shut them down intentionally.
Send yourself a test email (replacing [email protected] with your actual email address):
```
date | /var/hvmail/bin/mailsubj "GreenArrow Test Message" [email protected]
```

Congratulations, you’re finished restoring GreenArrow! As a next step, we recommend following up with GreenArrow technical support to request that we review the restored GreenArrow installation for issues.