HTTP Server

GreenArrow Engine comes with an HTTP server to provide a web interface, and click and open tracking.

Table of Contents
Enabling and Starting the HTTP Server
IP Addresses and Ports to Provide HTTP
IP Addresses and Ports to Provide HTTPS
Default TLS Configuration
Automatic TLS Configuration
Multiple TLS Certificates with Dedicated IPs
Multiple TLS Certificates with Shared IPs Using SNI
Graceful Restarts
URI Filtering
- Let’s Encrypt URIs
Performance Tuning
Advanced Configuration Changes
Apache Server Status
Passenger Status
Processing Clicks and Opens Behind a Proxy Server
- Click and Open Tracking Domain Security
- Verifying that GreenArrow correctly detects the requested hostname
Troubleshooting

Enabling and Starting the HTTP Server

To enable the HTTP server, set the /var/hvmail/control/httpd.enabled file to 1 or true:
```
echo 1 > /var/hvmail/control/httpd.enabled
```
After enabling the HTTP server, you can start it with:
```
svc -u /service/hvmail-httpd
```
To disable the HTTP server, set the /var/hvmail/control/httpd.enabled file to 0 or false:
```
echo 0 > /var/hvmail/control/httpd.enabled
```
After disabling the HTTP server, you can shut it down with:
```
svc -d /service/hvmail-httpd
```

IP Addresses and Ports to Provide HTTP

If you’re using GreenArrow’s click and open tracking, then we recommend running GreenArrow’s HTTP server on the default port (port 80). This is because some spam filters and firewalls are more likely to score a message as spam, or block access to a URL if a non-default port is used.

List the IP address and port combinations for the HTTP service to listen on, one per line, in the /var/hvmail/control/httpd.listen file. For example, a line can contain just a port number, such as 80 to configure GreenArrow Engine’s HTTP server to listen on that port for all IP addresses assigned to the server:
```
echo 80 > /var/hvmail/control/httpd.listen
```
A line can also contain an IP address and a port combination, such as 1.2.3.4:80:
```
echo 1.2.3.4:80 > /var/hvmail/control/httpd.listen
```
To apply these changes, run:
```
svc -t /service/hvmail-httpd
```

IP Addresses and Ports to Provide HTTPS

List the IP address and port combinations for the HTTPS service to listen on, one per line, in the /var/hvmail/control/httpd.ssl.listen file. This file is in the same format as the /var/hvmail/control/httpd.listen file, unless you’re using multiple TLS Virtual Hosts.
To apply changes, run:
```
svc -t /service/hvmail-httpd
```

Default TLS Configuration

The TLS Certificate Configuration doc describes how to configure the default TLS certificate and key.

Automatic TLS Configuration

See the TLS Certificate Configuration doc.

Multiple TLS Certificates with Dedicated IPs

You can configure each IP address to use its own TLS certificate.

Do this by adding lines to /var/hvmail/control/httpd.ssl.listen that specify the key and certificate to use for a particular IP and port.

This is done by adding a line like this:

10.0.0.1:443 key=/path/to/1.key crt=/path/to/1.crt

To include a certificate chain, add a line like this:

10.0.0.1:443 key=/path/to/1.key crt=/path/to/1.crt crt_chain=/path/to/ca1.crt

For example, here is a complete /var/hvmail/control/httpd.ssl.listen that serves two certificates based on the IP address and port:

443
10.0.0.1:443 key=/path/to/1.key crt=/path/to/1.crt crt_chain=/path/to/ca1.crt
10.0.0.2:443 key=/path/to/2.key crt=/path/to/2.crt crt_chain=/path/to/ca2.crt

In this example the first line of 443 instructs GreenArrow to bind to all IPs on port 443 and the last two lines override what certificates are used.

To apply changes to the /var/hvmail/control/httpd.ssl.listen file, run:

svc -t /service/hvmail-httpd

Multiple TLS Certificates with Shared IPs Using SNI

You can configure the same IP address to use multiple TLS certificates, based on the SNI hostname provided in the TLS request.

Do this by adding lines to /var/hvmail/control/httpd.ssl.listen that specify the key and certificate to use for a particular IP, port, and incoming SNI server name.

This is done by adding a line like this:

10.0.0.1:443 key=/path/to/1.key crt=/path/to/1.crt server_name=tls1.example.com

To include a certificate chain, add a line like this:

10.0.0.1:443 key=/path/to/1.key crt=/path/to/1.crt crt_chain=/path/to/ca1.crt server_name=tls1.example.com

For example, here is a complete /var/hvmail/control/httpd.ssl.listen that serves two certificates based on the SNI hostname:

443
10.0.0.1:443 key=/path/to/1.key crt=/path/to/1.crt crt_chain=/path/to/ca1.crt server_name=tls1.example.com
10.0.0.1:443 key=/path/to/2.key crt=/path/to/2.crt crt_chain=/path/to/ca2.crt server_name=tls2.example.com

In this example the first line of 443 instructs GreenArrow to bind to all IPs on port 443 and the last two lines override what certificates are used.

To apply changes to the /var/hvmail/control/httpd.ssl.listen file, run:

svc -t /service/hvmail-httpd

Graceful Restarts

To gracefully restart the web server, rather sending it a TERM signal with svc, run the following command:

/var/hvmail/bin/hvmail_update_httpd_config --graceful

URI Filtering

The following URI prefix needs to be publicly accessible in order for Engine to properly function:

/click/
/open/
/unsub/

If $CLICKTHROUGH_URL in /var/hvmail/control/simplemh-config ends with /click.php, then /click.php/ should also be included in this list. This was the default prior to GreenArrow Engine v4.1.232.

If you want to use the embeddable HTML campaign stats API then you should allow the access to the following URI prefixes:

/greenarrowembed/assets/
/greenarrowembed/stats_sends_view_drilldown.php
/greenarrowembed/stats_sends_view_logs.php
/gaeembed/assets/
/gaeembed/stats_sends_view_drilldown.php
/gaeembed/stats_sends_view_logs.php

The following URI prefixes need to be publicly accessible in order for Studio to properly function:

/ga/open/
/ga/click/
/ga/unsubscribe/
/ga/webviews/
/ga/campaign_images/
/ga/autoresponder_images/
/ga/front/
/ga/bee-fs/

Let’s Encrypt URIs

If you want to use Automatic TLS Certificates with Let’s Encrypt then you should allow access to the following URI prefixes:

/.well-known/acme-challenge/
/.lets_encrypt_challenge_preflight

HTTP to HTTPS redirection for the Let’s Encrypt URI prefixes listed above should also be disallowed.

Performance Tuning

The greenarrow.conf configuration file can be used to tune performance-related settings.

Advanced Configuration Changes

GreenArrow Engine uses the Apache HTTP Server. To customize this Apache installation’s configuration beyond the settings described above, place the configuration in one of the following files:

Update the /var/hvmail/control/httpd.custom.conf file to make changes to the main Apache configuration.
Update the /var/hvmail/control/httpd.ssl.custom.conf file to make change to the configuration of both GreenArrow’s default HTTPS Virtual Host, and any Virtual Hosts defined in /var/hvmail/control/httpd.ssl.listen.

To avoid creating conflicting configurations, please verify that your desired customizations are not configurable via some other mechanism, such as the configuration file before editing either of these advanced configuration files.

Apache Server Status

To view the status of Apache, navigate to the /greenarrowadmin/server-status path. This will require the credentials for GreenArrow Engine.

This screen can also be reached locally on the server at http://127.0.0.1/server-status-localhost.

Passenger Status

The web application server includes a command to check its status.

# passenger-status
Version : 6.0.4
Date    : 2020-03-04 06:11:30 -0600
Instance: YwAMfniA (Phusion_Passenger/6.0.4)

----------- General information -----------
Max pool size : 4
App groups    : 1
Processes     : 2
Requests in top-level queue : 0

----------- Application groups -----------
/var/hvmail/studio (production):
  App root: /var/hvmail/studio
  Requests in queue: 0
  * PID: 6763    Sessions: 0       Processed: 193     Uptime: 17h 38m 31s
    CPU: 0%      Memory  : 153M    Last used: 1h 36m
  * PID: 6784    Sessions: 0       Processed: 2       Uptime: 17h 38m 30s
    CPU: 0%      Memory  : 106M    Last used: 1h 32m

In this screen, we can see Processes that indicates how many application processes are available for serving web requests. If pass__min_instances is lower than pass__max_pool_size, this number will vary based upon the volume of traffic. If they are equal, they will stay steady at that value.

Processing Clicks and Opens Behind a Proxy Server

When a click or open is recorded, it includes the IP address of the client. When GreenArrow is placed behind a proxy server, without further configuration, the recorded IP address would always be that of the proxy server.

Due to this, any proxy servers that are in front of GreenArrow must be configured as follows.

Set the proxy server’s IP address as trusted in the configuration files described below.
The proxy server must provide the CF-Connecting-IP, X-Forwarded-For or Client-IP headers containing the IP address of the original client.
The proxy server must provide the X-Forwarded-Host header containing the hostname requested by the original client.

To configure the proxy server (or servers) as a trusted source of the headers listed above, the following configuration must be set. The example below assumes that 1.2.3.4 is the IP address of the proxy server in front of GreenArrow. These files must contain multiple IP addresses if there are multiple proxy servers in front of GreenArrow.

For Studio:

echo 1.2.3.4 >> /var/hvmail/control/studio.trusted_proxy_ips

For SimpleMH:

echo 1.2.3.4 >> /var/hvmail/control/opt.engine.trusted_proxy_ips

The files /var/hvmail/control/studio.trusted_proxy_ips and /var/hvmail/control/opt.engine.trusted_proxy_ips may contain one or more IP addresses or CIDR ranges, separated by newlines.

Here’s an example of what the headers the proxy server at 1.2.3.4 must provide might look like:

X-Forwarded-For: 12.34.56.78
X-Forwarded-Host: ga.example.com

The headers may include multiple values, separated by commas, if there are multiple proxies in front of GreenArrow.

X-Forwarded-For: 12.34.56.78, 2.4.6.8, 1.3.5.6
X-Forwarded-Host: ga.example.com, ga1.example.com, ga2.example.com

When provided X-Forwarded-For, GreenArrow scans the list from right-to-left, looking for an untrusted IP address. It uses the first (right-to-left) untrusted IP address it comes to as the client’s IP.

When provided Client-IP, GreenArrow will use the IP address in this header, so long as the connection to GreenArrow is being made from a trusted proxy. Client-IP must contain a single IP address representing the original client’s IP. If both Client-IP and X-Forwarded-For are provided, GreenArrow uses X-Forwarded-For.

When provided X-Forwarded-Host, the first domain in the list is used, so long as the connection to GreenArrow is being made from a trusted proxy. That domain will will be used for tracking domain security (see more information on tracking domain security below).

In the above example, if the proxy servers 1.2.3.4, 2.4.6.8, and 1.3.5.6 are configured as trusted: GreenArrow will use 12.34.56.78 as the client’s IP and ga.example.com will be used for tracking domain security.

Click and Open Tracking Domain Security

GreenArrow includes anti-tamper protection in the links created for click tracking, to prevent abuse of the link redirector service. Anti-tamper protection detects if the destination URL, the subscriber, or campaign details are tampered with, and returns an error to the user if the link was modified. The domain name in the link is included in this anti-tamper protection.

Including the domain name in anti-tamper protection is useful for email service providers, where one customer could send a link to themselves, capture that link, modify it to use a different domain name belonging to the ESP or another of the ESP’s customers, and then send spam using the modified link – hijacking a domain’s reputation to which they were never supposed to have access.

Because of this, the click and open processor must know the original domain name requested by the client. Any proxy or load balancer in front of GreenArrow must provide an X-Forwarded-Host header indicating the originally requested domain name. If the proxy or load balancer is configured as trusted above, the first domain listed in that header will be used to verify the link has not been modified.

To disable inclusion of the domain name in anti-tamper protection, see the disable_validating_domain_in_tracking_security configuration directive.

Verifying that GreenArrow correctly detects the requested hostname

To help determine that GreenArrow is correctly identifying the originally requested hostname, we provide an endpoint for retrieving (a) the headers with which GreenArrow was accessed, (b) the client IP that GreenArrow will use for logging clicks and opens, and (c) the hostname that will be used for anti-tamper protection.

This endpoint can be accessed via a web browser or a command-line tool like curl using your GreenArrow Engine credentials.

$ curl -u username:password http://example.com/ga/eui/print_http_diag

{
  "modified_headers": {
    "HTTP_USER_AGENT": "curl/7.64.1",
    "HTTP_X_FORWARDED_SSL": "off",
    "HTTP_ACCEPT": "*/*",
    "HTTP_X_FORWARDED_USER": "drh",
    "HTTP_X_FORWARDED_SERVER": "example.com, 172.17.0.2",
    "HTTP_X_FORWARDED_HOST": "example.com, ga.example.com",
    "HTTP_HOST": "127.0.0.1",
    "HTTP_X_FORWARDED_PROTO": "http",
    "HTTP_X_FORWARDED_FOR": "192.136.172.244, 172.17.0.1",
    "HTTP_VERSION": "HTTP/1.1",
    "HTTP_IF_MODIFIED_SINCE": "",
    "HTTP_IF_NONE_MATCH": "",
    "HTTP_X_GAORIG_CF_CONNECTING_IP": "",
    "HTTP_X_GAORIG_CLIENT_IP": "",
    "HTTP_X_GAORIG_HOST": "ga.example.com",
    "HTTP_X_GAORIG_REMOTE_ADDR": "172.17.0.1",
    "HTTP_X_GAORIG_X_FORWARDED_FOR": "192.136.172.244",
    "HTTP_X_GAORIG_X_FORWARDED_HOST": "example.com"
  },
  "original_headers": {
    "CF-Connecting-IP": "",
    "X-Forwarded-For": "192.136.172.244",
    "X-Forwarded-Host": "example.com",
    "Client-IP": "",
    "Host": "ga.example.com"
  },
  "request_ip": "172.17.0.1",
  "request_ip_is_trusted": true,
  "client_ip": "192.136.172.244",
  "requested_hostname": "example.com"
}

This response includes the following keys:

modified_headers	The headers as they made it into GreenArrow’s application server. This includes some modifications and additional headers added by GreenArrow’s internal proxy server.
original_headers	These are the relevant headers as they were received by GreenArrow.
request_ip	The IP address that was used to directly connect to GreenArrow. If you have a proxy server in front of GreenArrow, this will be the IP address of that proxy server.
request_ip_is_trusted	This field indicates whether or not `request_ip` is considered to be a trusted proxy. If this is true, then its `X-Forwarded-For`, `X-Forwarded-Host`, and/or `Client-IP` headers will be used to calculate `client_ip` and `requested_hostname`.
client_ip	The IP address that GreenArrow determined was the client’s original IP address, based upon either `request_ip`, `X-Forwarded-For` or `Client-IP`.
requested_hostname	The hostname that GreenArrow determined was originally used by the client, based upon either `Host` or `X-Forwarded-Host`.

In the above example, we can see that GreenArrow successfully identified the requested_hostname as example.com and client_ip as 192.136.172.244. This is because the reverse proxy in front of GreenArrow was correctly configured above as a trusted proxy and that proxy is providing the X-Forwarded-For and X-Forwarded-Host headers.

If the reverse proxy server were not trusted, or if the proxy was not providing the X-Forwarded-Host header, we might see ga.example.com or 127.0.0.1 listed as the requested_hostname.

If GreenArrow is not correctly identifying the requested hostname, you can temporarily turn on disable_validating_all_tracking_security to process clicks and opens while you debug and fix the issue. We strongly recommend that you do not leave this setting on as a “fix”.

Troubleshooting

If the HTTP server fails to start, or otherwise acts incorrectly, then the first place to look at is the /var/hvmail/log/httpd/current log. Here’s an example command to monitor that log file’s output, and convert the timestamps into a readable format:

tail -F /var/hvmail/log/httpd/current | tai64nlocal

Press Ctrl-c when you’re finished viewing the file.

Apache’s log files, located in /var/hvmail/apache/logs/ can also be useful for debugging if you’re in a situation where the webserver starts but doesn’t behave as desired. That directory’s error_log file is the most likely to contain useful info. Additional information is logged to access_log for HTTP requests, and ssl_request_log for HTTPS requests.

Compare any recent changes to the above log file’s output. Hopefully that’s enough to identify a fix for the problem, but if not, review the following configuration files for issues related to the errors that are logged:

/var/hvmail/control/httpd.custom.conf
/var/hvmail/control/httpd.ssl.custom.conf
/var/hvmail/control/httpd.listen
/var/hvmail/control/httpd.ssl.listen

For example, if an error message is logged about a missing file, check if any of the above configuration files contain a reference to that missing file.

The HTTP server automatically recovers from some types of configuration errors once the configuration file is fixed. If it doesn’t automatically recover, then you can force it to restart and re-read all configuration files by running:

svc -tu /service/hvmail-httpd