Server Migrations

Table of Contents
Intro
Prerequisites
Non-GreenArrow Files and Services
Migration Procedure
Post-Migration Configuration
Final Testing
Disable Old Server
Let Us Know You’ve Migrated

Intro

This document describes how to migrate a GreenArrow installation from one server to another.

Prerequisites

The migration procedure described on this page has some prerequisites:

An experienced Linux systems administrator who can execute the migration procedure.
The server that you’re migrating from must be running GreenArrow 4.200.0 or later. If you’re using an older release, then see these instructions.

Here’s a command you can run to check what version is installed:
```
rpm -q greenarrow 2> /dev/null || dpkg -l greenarrow 2> /dev/null || echo "GreenArrow is either not installed, or is older than version 4.200.0."
```
Most GreenArrow installations use PostgreSQL 16 or 9.5, which can run on all of our supported Linux distributions. Some installations use PostgreSQL 8.3, though, which comes with Linux distribution restrictions. To check which version of PostgreSQL you’re using, examine the contents of the PG_VERSION file. For example:
```
# cat /var/hvmail/postgres/default/data/PG_VERSION
16
```
If you’re running PostgreSQL 8.3, then the following restrictions apply:
1. PostgreSQL 8.3 is only supported by:
  - Red Hat-based distributions version 8, and 9.
  - Ubuntu distributions version 18.04, 20.04, and 22.04.
  - Debian distributions version 10 and 11.
2. PostgreSQL 8.3 does not support using TLS for incoming connections.
Click here for the documentation to upgrade PostgreSQL to the latest version.
The server should have enough disk space to hold the migrated data. The vast majority of GreenArrow’s data is located within the /var/hvmail directory and the migrated data will occupy approximately the same amount of disk space after the migration as it did before.

If the specs of the destination server are identical to or greater than the server that you’re migrating from, and you didn’t have any disk space issues before the migration, then you’re covered.
The /var/hvmail/control/UPGRADE_BLOCKER.txt file must not exist. This file is only present on GreenArrow installations where non-standard steps are required to perform updates of GreenArrow’s software. Those same non-standard steps will need to be performed during the migration. Contact GreenArrow technical support to arrange to have those steps applied.

Non-GreenArrow Files and Services

These migration instructions do not take into account any non-GreenArrow files or services that you may also wish to migrate.

Migration Procedure

Some of the steps below are Linux distribution specific. For the sake of brevity, we refer to Red Hat-based Linux distributions, such as CentOS, Red Hat Enterprise Linux or Scientific Linux as “Red Hat”, and Debian-based distributions, such as Ubuntu and Debian as “Debian”.

Here’s how to migrate GreenArrow to a new server. All steps should be completed on the new server unless otherwise indicated:

If you’re adding, removing or changing any IPs as part of the migration, then consider configuring their DNS records and feedback loop registrations in advance. Instructions for adding new IPs are here.
Authorize the new server’s root account to SSH into the old server as root.
Log in to the new server as the root user.
Install packages that are used by the migration process:

On Red Hat systems, run:
```
yum install ruby rsync
```
On Debian systems, run:
```
apt-get install ruby wget gnupg rsync
```
Make an entry in the new server’s /root/.ssh/config file, pointing the oldserver hostname to an IP on the old server that has an SSH server running. For example:
```
Host oldserver
    Hostname 1.2.3.4
    User root
    Port 22
```
This entry is needed because we’re going to run some commands later on which connect to oldserver.

If a non-default port is used for ssh, you can update that port number in /root/.ssh/config as defined above.
Take a backup of the old server. A migration normally only uses some of the files that get created by a backup, but we recommend that you hold onto the entire backup directory to increase the odds of being able to recover if something goes wrong during the migration. For extra safety, you may want to copy this directory to some media that’s not attached to either server.
Review the postgresql-tablespaces.txt file in the backup on the old server for any tablespaces which specify a location.

Here’s an example of what this file might look like. In this example, the tablespace named “data2” is the only one which has a location (/media/data2) specified:
```
                    List of tablespaces
    Name    |   Owner    |              Location
------------+------------+------------------------------------
data2       | greenarrow | /media/data2
pg_default  | postgres   |
pg_global   | postgres   |
(3 rows)
```
If there are no tablespaces which specify a location, then you can move onto the next step.

If there are any tablespaces which specify a location, then re-create their directories on the new server. For example:
```
mkdir /media/data2
```
Also, rsync each of the directories identified above from the old server to the new server. You should both run each rsync invocation now, and record it so that it can be run again later after hvmail_server_migration has run to completion. For example:
```
rsync -a --delete oldserver:/media/data2/ /media/data2
```
Retrieve your GreenArrow package repository key from the old server:
```
ssh oldserver -- \
  cat '/etc/apt/sources.list.d/greenarrow*' '/etc/yum.repos.d/greenarrow*' \
  2> /dev/null | \
  ruby -e 'puts STDIN.read.scan(%r{/key/([0-9a-f]+)/}).first'
```
The above should print something like d41d8cd98f00b204. If it does not, contact GreenArrow technical support for a copy of your GreenArrow package repository key.

You will need your GreenArrow package repository key for the next step.
Complete the Installation Guide up to and including the “Update environment settings” step of “Install GreenArrow”.

We recommend installing the latest version of GreenArrow on the new server. If you want to install the same version as was running on the old server, then you’ll need to update your package installation step of the install procedure above accordingly.

If you get to the “Initialize GreenArrow” step in the Installation Guide, you’ve gone too far.
Run the migration script for the first time. This can safely be done while GreenArrow is still running on the old server:
```
/var/hvmail/libexec/hvmail_server_migration --no-exit-on-error
```
This migration script could take anywhere from minutes to hours to run, depending on how much data there is to transfer. It’s safe to run the script multiple times, as long as only one instance is running at any given point in time.

rsync is used for most of the data transfer, so all other things being equal, the closer together two invocations of this script are to each other, the faster the second invocation will run.

Stop GreenArrow on the old server:

systemctl stop greenarrow.service
systemctl disable greenarrow.service

Run the migration script again. This will synchronize any files which changed since the last time the script was run:
```
/var/hvmail/libexec/hvmail_server_migration
```
If the migration script completes without error, then its last line of output will read No errors were detected. If the last line contains anything else, review the remaining output for issues. Once any issues have been corrected, it’s safe to re-run hvmail_server_migration.
If you identified any additional rsync invocations to run while you were inspecting postgresql-tablespaces.txt earlier, run them now.
If there are any IP addresses to move from the old server to the new one, move them now.
This step is not needed for Let’s Encrypt certificates that are configured automatically.

If you’re using the Legacy Let’s Encrypt configuration method, then remove all of that method’s certificates from the HTTP TLS configuration file. These certificates are stored outside of GreenArrow’s directories, so they are not migrated by this procedure. Take note of which certificates you’ve removed in case you wish to add them back in later. There’s a step later in this procedure for doing that.

Start GreenArrow on the new server:

systemctl start greenarrow.service
systemctl enable greenarrow.service

If you installed a version of GreenArrow on the new server that is more recent than the one installed on the old server:
1. Collect and complete all the Version specific steps listed in the “GreenArrow Updates” document that apply to the version of GreenArrow installed on the old server
2. Run the following command to apply all pending GreenArrow migrations:
```
greenarrow update
```

On Debian systems, disable the GreenArrow package repos:

sed -i 's/^deb /#deb /' /etc/apt/sources.list.d/greenarrow.list

Check whether greenarrow.conf has dns_cache_service_run set:
```
grep dns_cache_service_run /var/hvmail/control/greenarrow.conf
```
1. If dns_cache_service_run is set to true or yes, then configure the /etc/resolv.conf file so that the only nameserver line is set to 127.0.0.1. For example:
```
echo "nameserver 127.0.0.1" > /etc/resolv.conf
```
2. If dns_cache_service_run is set to false or no, or is not specified, then configure the /etc/resolv.conf file to query one or more recursive DNS resolvers. For example, if you would like to query the Google and Cloudflare public DNS servers, you could populate the file with:
```
echo "nameserver 8.8.8.8
nameserver 8.8.4.4
nameserver 1.1.1.1" > /etc/resolv.conf
```
Verify that different nameservers aren’t configured outside of /etc/resolv.conf. If the commands below return any output, then verify that it matches your desired configuration:

On Red Hat systems, run:
```
find /etc/sysconfig/network-scripts/ifcfg-* /etc/sysconfig/network -type f | xargs grep 'DNS=' | grep -v 'PEERDNS='
```
On Debian systems, run:
```
grep dns-nameservers /etc/network/interfaces
```
Check if the /var/hvmail/control/UPGRADE_BLOCKER.txt file exists:
```
cat /var/hvmail/control/UPGRADE_BLOCKER.txt
```
This file is only present on GreenArrow installations where non-standard steps are required to perform updates of GreenArrow’s software.

If the file is missing, then you can move onto the next step.

If the file is present, then contact GreenArrow technical support to have those non-standard steps applied.
Complete the Performance Tuning section of our Installation Guide.

Post-Migration Configuration

If you’re adding, removing or changing any IPs as part of the migration, then they should be added into GreenArrow’s configuration now. Instructions for adding new IPs are here.
This step is not needed for Let’s Encrypt certificates that are configured automatically.

Optionally, add Legacy Let’s Encrypt certificates to the HTTP Server’s TLS configuration.
If there are any other configuration changes desired as part of this migration, make them now.

Final Testing

Verify that some common misconfigurations are not present. Each test gets a simple pass/fail result:
```
hvmail_check_config
```
Verify that all services are running normally. Any services with an abnormal state are shown in red:
```
service greenarrow status
```
The hvmail-qmail-smtpd3 and hvmail-dnscache services are down by default. All other services should be reported as up unless you shut them down intentionally.
Send yourself a test email (replacing [email protected] with your actual email address):
```
date | /var/hvmail/bin/mailsubj "GreenArrow Test Message" [email protected]
```
Remove the entry that you made earlier for oldserver in the new server’s /etc/hosts file.

Congratulations, you’re finished migrating GreenArrow to the new server! As a next step, we recommend following up with GreenArrow technical support to request that we review the migrated GreenArrow installation for issues.

Continue to the next section to prevent the old server from coming back online, and attempting to send duplicate emails and/or use the same license key in two locations at the same time.

Disable Old Server

Remove GreenArrow’s licenses from the old server:

rm -f /var/hvmail/control/license_key /var/hvmail/control/bounce.boogie_studio_api_license_key

Delete the queued messages on the old server:

find /var/hvmail/qmail-ram/savedqueue -type f -delete
find /var/hvmail/qmail-bounce/savedqueue -type f -delete
cd /var/hvmail/qmail-disk/queue/
find info -type f -delete
find mess -type f -delete
find remote -type f -delete
find intd -type f -delete
find local -type f -delete
find todo -type f -delete
find bounce -type f -delete
find batches -type f -delete

Let Us Know You’ve Migrated

Contact GreenArrow technical support to update your server information. This is not required, but it helps prevent potential licensing issues.