TLS Certificate Configuration

• Table of Contents
• Overview
• Let’s Encrypt Integration
  • Enabling Let’s Encrypt
  • Getting a Server-Default TLS Certificate Through Let’s Encrypt
  • Getting TLS certificates for URL Domains Through Let’s Encrypt
  • Rules of Precedence for URL Domains
  • Self-Checks
  • Let’s Encrypt Certificate Renewal
  • Let’s Encrypt Expiration Notifications
  • GreenArrow Renewal Warnings
  • greenarrow lets_encrypt_status
  • greenarrow lets_encrypt_retry
  • Let’s Encrypt DST Root CA X3 Expiration
• Configuring TLS Certificates Manually
  • Issuing certificate request (CSR) and key files
• Server-Default TLS Certificate
• Deprecated Default TLS Certificates

Overview

This page describes how to configure TLS certificates for GreenArrow’s HTTPS, FTP, and incoming SMTP services.

This page also describes GreenArrow’s Let’s Encrypt integration.

We recommend configuring GreenArrow to register TLS certificates automatically with Let’s Encrypt. Automatic registration is the most straightforward approach, and it covers the needs of most GreenArrow installations.

A DNS record for the domain that’s to be registered must point to your GreenArrow server for automatic registration to work.

Let’s Encrypt Integration

Enabling Let’s Encrypt

Let’s Encrypt is a Certificate Authority that provides free TLS certificates and a protocol to automate TLS installation and renewal.

GreenArrow can automatically register TLS certificates, but that functionality is disabled by default. To enable it, you must accept the Let’s Encrypt subscriber agreement by completing the following steps:

1. Set lets_encrypt_agree_to_terms_of_service to yes and lets_encrypt_registration_email to your email address to signify your acceptance.
2. Apply changes by running greenarrow_config reload.

Getting a Server-Default TLS Certificate Through Let’s Encrypt

Once Let’s Encrypt integration is enabled, GreenArrow’s default behavior is to get the Server-Default TLS certificate through Let’s Encrypt for the domain name configured in/var/hvmail/control/me.

This can be overridden with the following methods:

• The tls_certificate_files configuration directive.
• The tls_certificate_auto_generate configuration directive.
• The deprecated TLS configuration files

Getting TLS certificates for URL Domains Through Let’s Encrypt

Once Let’s Encrypt integration is enabled, GreenArrow’s default behavior is to get a TLS certificate for every URL Domain through Let’s Encrypt.

This can be disabled by setting the url_domains_get_lets_encrypt_certs configuration directive to no.

You can check the status of certificates for each URL domain by either:

• Running the greenarrow lets_encrypt_status command
• Checking the “HTTPS” column in the URL Domains page in Engine

After this is enabled and certificates are retrieved for all URL Domains, we recommend also enabling the url_domains_always_use_tls directive to always use HTTPS for all click, open, and unsubscribe links.

Rules of Precedence for URL Domains

If TLS certificates are both obtained from Let’s Encrypt and configured in /var/hvmail/control/httpd.ssl.listen, then the following rules of precedence apply:

1. /var/hvmail/control/httpd.ssl.listen rules with the server_name option take precedence over Let’s Encrypt certs.
2. Let’s Encrypt certs take precedence over /var/hvmail/control/httpd.ssl.listen rules that don’t have the server_name specified.

Self-Checks

GreenArrow’s automatic TLS certificate registration checks whether it can connect to itself via HTTP (port 80) at the URL domain before it attempts to register a TLS certificate with Let’s Encrypt for that domain.

If the self-check fails, the error message will include “Self-check failed:” to aid in determining the source of the failure.

See lets_encrypt_certs_self_check for more information on the self-check.

Let’s Encrypt Certificate Renewal

GreenArrow attempts to renew certificates 30 days before their expiration date. If a renewal fails, GreenArrow retries one minute after the first attempt, and from that moment forward the delay grows until it reaches a maximum of 12 hours between retries.

Let’s Encrypt Expiration Notifications

The email address set in lets_encrypt_registration_email will receive an Expiration Email from Let’s Encrypt before a certificate expires.

Let’s Encrypt lets you unsubscribe from these emails if you wish, but they are a good backup signal that something has gone wrong with your certificate renewal. However, there is a case where you will be notified when there is not actually a problem:

If you have removed a domain from GreenArrow, then Let’s Encrypt will send an expiration warning email when the domain is not renewed by GreenArrow. In this case, nothing is actually wrong. When you get an expiration warning from Let’s Encrypt, we recommend that you check GreenArrow’s certificate expiration warnings (documented later on this page).

If you have many customer domains added and removed from GreenArrow, and there are too many false-positives, you may want to disable the expiration warnings from Let’s Encrypt.

GreenArrow Renewal Warnings

GreenArrow shows a warning about domains that have failed to register or renew in the following conditions:

• The domain is still in GreenArrow as a URL domain, /var/hvmail/control/me, or tls_certificate_auto_generate.
• Registration failed, either:
  • The domain has a certificate but has failed to renew for at least an hour.
  • The domain does not yet have a certificate and has failed to register at least once.
• The user is signed into either the GreenArrow Engine user interface or Marketing Studio as a system administrator.

greenarrow lets_encrypt_status

The greenarrow lets_encrypt_status command is used to check the status of Let’s Encrypt TLS certificate registrations, including expiration dates. It’s also used to identify the causes of registration issues.

Here’s an example of its output when there are two URL domains:

Domain                                             Cert Exists     Cert Expiration
----------------------------------------------------------------------------------
working.example.com                                 yes             2020-05-26
broken.example.com                                  no
  Error message:     Registration failed: Self-check failed: DNS lookup failed. Check DNS and verify that "broken.example.com" resolves to an IP address on this host.
  Last failure time: 2020-03-06 06:48:04 -0600
  Next retry time:   2020-03-06 07:48:04 -0600

Here’s what the above report means:

• working.example.com has a certificate that expires on May 26, 2020.
• broken.example.com does not have a certificate because it failed the Self-Check.

Add the flag --only-errors (i.e., greenarrow lets_encrypt_status --only-errors) to display only certificates that have not yet been attempted or have encountered an error in registration or renewal.

Running greenarrow lets_encrypt_status --json produces JSON formatted output:

{
    "domains": [
        {
            "domain": "working.example.com",
            "has_cert": true,
            "cert_expiration_time": "2020-05-28T20:23:35Z",
            "last_failure_error_message": null,
            "last_failure_time": null,
            "next_retry_time": null
        },
        {
            "domain": "broken.example.com",
            "has_cert": false,
            "cert_expiration_time": null,
            "last_failure_error_message": "Registration failed: Self-check failed: DNS lookup failed. Check DNS and verify that \"broken.example.com\" resolves to an IP address on this host.",
            "last_failure_time": "2020-03-06T12:48:04.740131Z",
            "next_retry_time": "2020-03-06T13:48:04.740128Z"
        }
    ]
}

greenarrow lets_encrypt_retry

The greenarrow lets_encrypt_retry command is used to force GreenArrow to retry registration on one or more specific domains.

greenarrow lets_encrypt_retry domain1.example.com domain2.example.com

GreenArrow immediately retries registration or renewal of the specified domains.

This command does not affect TLS certificates that are currently valid and are outside of their renewal window, so it doesn’t cause early renewals.

Let’s Encrypt DST Root CA X3 Expiration

On May 2021 Let’s Encrypt notified that on September 30 2021 the “DST Root CA X3” certificate will expire. This expiration may affect Let’s Encrypt certificate verification on IoT devices and HTTP clients running old Operating Systems or libraries.

This will not affect certificate registrations or renewals, but it can potentially cause older clients to fail to verify TLS certificates issued by Let’s Encrypt.

For more information about the DST Root CA X3 certificate and the systems that can potentially be impacted by its expiration, visit Let’s Encrypt documentation page.

To check if your subscribers or your applications will be affected by this change visit Let’s Encrypt’s Certificate Compatibility page.

If this certificate expiration will affect your applications or subscribers, consider configuring TLS certificates manually.

Configuring TLS Certificates Manually

If you prefer to configure TLS certificates manually, then you have a few options:

1. The Server-Default TLS certificate can be set with the tls_certificate_files configuration directive. This is the recommended alternative for customers who don’t want to use automatic registration.
2. Use the HTTP Server’s /var/hvmail/control/httpd.ssl.listen file.
3. Use the deprecated default TLS certificate paths.

Issuing certificate request (CSR) and key files

To issue a certificate request with openssl, run this command and follow the prompts:

openssl req -new -newkey rsa:2048 -nodes -keyout server.key -out server.csr

This will generate the private key and the CSR (Certificate Signing Request) for your TLS certificate. The output will be saved to files named server.key and server.csr respectively.

Server-Default TLS Certificate

GreenArrow uses the Server-Default TLS certificate in the following areas:

1. HTTPS requests for hostnames or IPs that do not otherwise have a matching certificate
2. The FTP server
3. TLS encryption for incoming SMTP connections
4. The GreenArrow Sink Service, which is used for speed testing

The Server-Default TLS certificate is the first certificate found in the following list:

1. The certificate defined by tls_certificate_files
2. A Let’s Encrypt certificate obtained due to the tls_certificate_auto_generate directive
3. A Let’s Encrypt certificate obtained for the domain in /var/hvmail/control/me
4. A self-signed certificate generated for the domain in /var/hvmail/control/me

Acquisition of Let’s Encrypt certificates can fail due to registration errors. If a failure occurs, GreenArrow moves onto the next item in the above list. Failures can be seen with the greenarrow lets_encrypt_status command. The last option (generating a self-signed certificate) always succeeds.

Deprecated Default TLS Certificates

Before the introduction of automatic TLS certificate registration and the tls_certificate_files configuration directive, TLS certificates were read from different files for different functions.

If a service’s deprecated default TLS certificate files are present, they take precedence over the Server-Default TLS certificate to ensure backward compatibility. We recommend replacing them with one of the other options described in the Default TLS Certificate section.

1. HTTPS requests and the GreenArrow Sink Service used:
   • /var/hvmail/control/httpd.ssl.crt
   • /var/hvmail/control/httpd.ssl.key
2. The FTP server used:
   • /var/hvmail/control/pure-ftpd.pem
3. TLS encryption for incoming SMTP connections used:
   • /var/hvmail/control/tls.servercert.pem