Here we use LetsEncrypt (certbot) with the CloudFlare DNS plugin to generate a free, auto-renewing TLS certificate to use with Nginx.

Then we configure Nginx to use that TLS certificate and create a configuration to support multi-tenancy in our applications.

We use a special configuration to capture the value of the subdomain so we can pass it off to our PHP application (or do anything we want, like use dynamic app locations for local development - as described here: https://www.youtube.com/watch?v=SPHxW1C4G6I ).

Useful Links:

Install certbot: https://certbot.eff.org/ Certbot challenge types: https://letsencrypt.org/docs/challenge-types/ My site: fideloper.com My newsletter: https://fideloper.ck.page/

Install and Configure Certbot

We can start by installing/configuring Certbot. In our case, we'll use the Cloudflare plugin to manage DNS.

Why do we need to manage DNS? Certbot uses a DNS challenge for wildcard subdomains (instead of the HTTP challenge for non-wildcard domains).

# Install certbot on Ubuntu as per
# https://certbot.eff.org/instructions?ws=nginx&os=snap

sudo snap install --classic certbot
sudo ln -s /snap/bin/certbot /usr/bin/certbot

sudo snap set certbot trust-plugin-with-root=ok
sudo snap install certbot-dns-cloudflare

We'll run stuff as user root, so we'll configure a location to save credentials to the Cloudflare API.

You'll need to generate an API token in Cloudflare (you can lock them down to be specific to managing one domain's DNS). Docs on that are here.

Create file /root/.secrets/cloudflare.ini and add something like:

dns_cloudflare_api_token = 0123456789abcdef0123456789abcdef01234567

Then we can generate our wildcard certificate!

# Optionally add --force-renewal if
# a current certificate is generated and
# you want to over-write it
sudo certbot certonly \
    --dns-cloudflare \
    --dns-cloudflare-credentials /root/.secrets/cloudflare.ini \
    --post-hook "service nginx reload" \
    --non-interactive \
    --agree-tos \
    --email your-email-here \
    -d *.your-domain.tld \
    -d your-domain.tld

Check out the video for more details on all of this.

Let's see how to install, setup, and configure LetsEncrypt (certbot) with Nginx to get an SSL certificate in something like 30 seconds.

This will help you get and configure an TSL certificate that auto-renews itself via LetsEncrypt - you never have to think about it again!

When you install certbot, it will add a systemd timer. This timer periodically checks if the certificate needs renewing, and if so, does it! Configuration in /etc/letsencrypt keeps information about the certificates installed on the server, including post-renewal hooks (like running "service nginx reload").

In our case, we'll use certbot one-line command to obtain the certificate. We'll make sure Nginx is configured to allow requests to a .well-known directory. Finally we see how Nginx should be configured to use the generated TLS certificates (thanks to H5BP Nginx server configs for making it so easy).

Here's some resources:

• LetsEncrypt https://letsencrypt.org/
• Certbot https://certbot.eff.org/
• H5BP Nginx: https://github.com/h5bp/server-configs-nginx
• Video on setting up H5BP with Nginx: https://www.youtube.com/watch?v=d6kfuPo3Cnw&ab_channel=ChrisFidao

The Setup

We can install and configure Certbot pretty easy:

# Install certbot on Ubuntu as per 
# https://certbot.eff.org/instructions?ws=nginx&os=snap
sudo snap install --classic certbot
sudo ln -s /snap/bin/certbot /usr/bin/certbot

Then, assuming Nginx is up and running with a site on port 80, and nothing is blocking the /path/to/web-root/.well-known directory from serving files:

# Optionally add --force-renewal if
# a current certificate is generated and
# you want to over-write it
sudo certbot certonly --webroot \
    -w /var/www/app/public \
    -d someapp.xyz \
    -d www.someapp.xyz \
    --post-hook "service nginx reload" \
    --non-interactive \
    --agree-tos \
    --email your-email-here

Then you can configure Nginx to use the new TLS certificate! See the video above to a few details on that.

Fixing Nginx's default configuration.

Nginx's default configuration is fine, but could use some help!

The quickest way to improve Nginx's default configuration is to use H5BP's Nginx configuration.

The Problems with Defaults

There's a few "problems" (things we can improve) with the default configuration.

1. No caching for easily-cached static assets
2. No HTTP security headers/configuration in place
3. TLS encryption defaults could use improvement

Let's fix that!

Using H5BP

I basically just blow away the default Nginx configuration and use H5BP's:

sudo mv /etc/nginx /etc/nginx.old
git clone https://github.com/h5bp/server-configs-nginx.git /etc/nginx

Files in /etc/nginx/conf.d are loaded - your site configurations go here. There are templates in there for you to use!

The main thing to check out is h5bp/basic.conf, which then loads other configuration files. This is the default set of configuration loaded - but there is more there to check out and optionally use!

The defaults sets a great set of un-obstrusive security settings, file caching, letting LetsEncrypt (certbot) work, and more.

Check out the video for a ton more details!

Your app may need to allow users/teams/tenants/whatever the ability to have their own subdomain. If your domain is myapp.com, this means letting tenants use foo.myapp.com to access their account.

This affects your code base quite a but, but what I want to show you is a server setup for this. I'll be doing videos on this for development and production, so keep an eye on the Youtube channel for more.

In this article (and video!) we're concentrating on local development. We'll setup dnsmasq and Nginx so any subdomain used for a local test domain (e.g. myapp.test) will point to your one codebase.

Here's a quick run down of what's covered in the video - there's 3 steps to this:

1. Installing and configuring dnsmasq
2. Making your computer use dnsmasq for DNS resolution
3. Intalling and configuring nginx

dnsmasq

The tl;dr on installing and configuring dnsmasq with homebrew is this:

brew install dnsmasq

echo "address=/test/127.0.0.1" \
    | sudo tee /opt/homebrew/etc/dnsmasq.d/test.conf

# use sudo here
sudo brew services start dnsmasq

We install dnsmasq, and then configure it so that domains ending in .test resolve to 127.0.0.1. Then we use brew services to start dnsmasq. If you already have dnsmasq, you want to run restart instead of just start.

Make sure dnsmasq is started with sudo, as it needs elevated permissions to do what it's doing.

To test this, you can run:

dig foo.test @127.0.0.1

The ANSWER section should tell you that foo.test resolves to 127.0.0.1. Using @127.0.0.1 tells the dig command to use the domain name server (dnsmasq!) running at 127.0.0.1.

Using dnsmasq

We need to make our computer (MacOS in this case) use dnsmasq for its DN server (in addition to the regular DN servers it uses). To do that, we'll create a file /etc/resolver/test.

Warning: This change may not work instantly. You may need to wait a bit or even restart your computer.

Here are the commands to run:

sudo mkdir -p /etc/resolver

echo "nameserver 127.0.0.1" \
    | sudo tee /etc/resolver/test

That file just contains nameserver 127.0.0.1, which tells the operating system to find a nameserver running locally at 127.0.0.1.

To test this, run the same dig command as above, but without the @127.0.0.1 part. The dig command shouldn't need that anymore, since the OS now knows to check localhost for a domain name server first.

dig foo.test

Nginx

Lastly, we can install and configure Nginx.

brew install nginx

# no sudo here
brew services start nginx

# <Add configuration here, see below>

nginx -t
nginx -s reload

In this case, we install nginx and then start it without sudo. This make it run as our current user, allowing Nginx to read (and write to, if needed) files owned by our current user. This is good as our code bases are likely owned by our current user.

The video explains these in depth. For now, I'll just write 2 Nginx config files for you to use. One lets you use *.myapp.test for your myapp code base. The second configuration sets up a generic thing. Any *.test domain you use will map to a directory on your filesystem, allowing you to use any domain you want, knowing it will reach an app that is in a folder of the same name.

The multi-tenant setup in file /opt/homebrew/etc/nginx/servers/a-srv.conf:

server {
    location 80;

    server_name ~^(?<tenant>.+)\.myapp\.test$;

    root /Users/<you>/srv/myapp/public;
    index index.html index.htm index.php;

    location / {
        try_files $uri $uri/ /index.php?$query_string;
    }

    location ~ \.php$ {
        fastcgi_pass 127.0.0.1:9000;
        fastcgi_index index.php;
        fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
        fastcgi_param TENANT $tenant;
        include fastcgi_params;
    }
}

The multi-tenant setup in file /opt/homebrew/etc/nginx/servers/x-srv.conf:

server {
    location 80;

    server_name ~^(?<app>.+)\.test$;

    root /Users/<you>/srv/$app/public;
    index index.html index.htm index.php;

    location / {
        try_files $uri $uri/ /index.php?$query_string;
    }

    location ~ \.php$ {
        fastcgi_pass 127.0.0.1:9000;
        fastcgi_index index.php;
        fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
        include fastcgi_params;
    }
}

Notes:

1. The filenames are alphabetically ordered so the multi-tenant setup is loaded first
2. The directory mapping of domain -> code base directory for *.test domains assumes a document root of public within the code base, which may be Laravel specific

I'm not a Laravel dev

Here are Nginx configurations to use if you're not a Laravel developer, and your app listens for HTTP requests (e.g. Node, Go).

# /opt/homebrew/etc/nginx/servers/a-srv.conf
server {
    location 80;

    server_name ~^(?<tenant>.+)\.myapp\.test$;

    root /Users/<you>/srv/myapp/public;
    index index.html index.htm;

    location / {
        try_files $uri $uri/ @app;
    }

    location @app {
        proxy_set_header Host $http_host;
        proxy_set_header X-Tenant $tenant;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_pass http://127.0.0.1:8000;
    }
}

# /opt/homebrew/etc/nginx/servers/x-srv.conf
server {
    location 80;

    server_name ~^ (?<app>.+)\.test$;

    root /Users/<you>/srv/$app/public;
    index index.html index.htm;

    location / {
        try_files $uri $uri/ @app;
    }

    location @app {
        proxy_set_header Host $http_host;
        proxy_set_header X-App $app;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_pass http://127.0.0.1:8000;
    }
}

These proxy over HTTP instead of FastCGI. They use a "named" location block @app but the principles are otherwise the same.

More Details

There's more details and description in the Youtube video, definitely check it out!

Use Laravel Herd!?

You get this setup (but even better!) using Laravel Herd. If you're a Laravel developer, considering just using that.

The nginx try_files directive is actually interesting! Not, like, amazingly interesting - but it has more depth than appears at first glance.

First, Nginx almost doesn't need try_files. Without it, Nginx could serve static files just fine:

server {
    listen 80;
    server_name _;

    root /var/www/html/public;
    index index.html index.html;
}

If we support PHP, we could have something like this:

server {
    listen 80;
    server_name _;

    root /var/www/html/public;
    index index.html index.html;

    location ~ \.php$ {
        # pass off to PHP-FPM via fastcgi
    }
}

That actually works for static files and the home page of our PHP application. Once we introduce a path into our URI (e.g. example.com/foo/bar), it breaks. This is where try_files comes in.

Adding try_files

The try_files directive is going to run through each option given in order to attempt to use its directives to find a file that exists on the server.

server {
    listen 80;
    server_name _;

    root /var/www/html/public;
    index index.html index.html index.php;

    location / {
        try_files $uri $uri/ /index.php?$query_string;
    }
}

For a given URI, this will do the following:

The $uri option tells try_files to find the URI as given as a file on the disk drive relative to the root, which is /var/www/html/public in this case.

1️⃣ A URI of /css/app.css will search in /var/www/html/public/css/app.css.

2️⃣ A URI of /foo/bar will have 2 behaviors - one for if the directory exists, and one for if it does not.

First, the $uri/ option tells try_files to treat the URI as a directory and see if a directory exists. If the URI relates to an existing directory, Nginx needs to figure out what file to serve from that directory.

That's where the index directive comes into play. Since Nginx just knows a directory exist, we need index to tell Nginx which files to serve out of there (if they exist). You can have any files there. The first matched file "wins" and is served.

3️⃣ If the given URI matches neither an existing file nor directory, then try_files goes to the fallback URI - /index.php?$query_string;.

But other location blocks?

The location / block, and the try_files directive within it, actually work together with other location blocks! Here's slightly more complete configuration file:

server {
    listen 80;
    
    server_name _;
    root /var/www/html/foobar.com;

    index index.html index.htm index.php;

    location / {
        try_files $uri $uri/ /index.php?$query_string;
    }

    location ~ \.php$ {
        # pass off to PHP-FPM via fastcgi
    }

    location ~* \.(?:css(\.map)?|js(\.map)?|jpe?g|png|gif|ico)$ {
        expires    7d;
        access_log off;
        log_not_found off;
    }

    location ~ /\.(?!well-known).* {
        deny all;
    }
}

If the try_files directive resolves/finds a file that is a static asset (css, js, image), then the third location block actually handles the request. That means both location / {} and location ~* \.(<stuff>)$ {} blocks are relevant to such a request!

The same is true for when PHP files are used - the location ~ \.php$ {} block is used:

1. When index resolves to index.php in a directory
2. When the fallback /index.php?$requests_uri is used
3. When a PHP file given by the URI exists as given

In these cases, the "matched" (used?) PHP file found by try_files is handled by the location ~ \.php$ {} block, which passes the request off to PHP-FPM. This is why a 404 error (whether for a static file or a non-existent application route) is generally returned from the PHP application. All real "can't find a file on the disk" cases are passed off to /index.php, and therefore the request is sent to the PHP application proxied via FastCGI in this configuration.

PHP is, historically, stateless.

This is mostly a result of HTTP being stateless as well. Each HTTP request has no knowledge of any request before it.

PHP is much the same. Under "traditional" process models, it rebuilds its entire world on each request! There's no global state.

Language Comparison

In the video, we compare other popular languages and see how its possible to create a global variable that increases in value with each web request. In other words, there's global state that that you need to worry about.

PHP is different - even global variables are "reset" to their initial value on each request. (This is not to be confused with super globals such as $_GET, $_POST, $_SERVER, and so on).

Pros and Cons

This makes PHP much easier to use - the mental model of what your code is doing is much simpler when there's no state that might be accidentally saved between web requests.

However it also means we need to reload a lot of code on each web request. This is alleviated by PHP's opcache, but it's still not as efficient as having the framework/code loaded already when accepting a new web request.

Additionally, PHP can't make use of things like connection pooling - each web request instead needs to make a whole new connection to databases, caches, and other external services.

This, luckily, is mostly just fine - PHP is still fast!

Newer PHP models

There are newer ways to run PHP - using Swoole or RoadRunner, for example, we can run PHP as a long-running process. This behaves more like other programming languages where we need to worry about global state, but we get the benefit of not having to reload your code/framework on each request.

Laravel has made using this simple via Laravel Octane. However it's still not the mainstream way to run PHP! Given how large PHP is, I'm not sure it ever will be. It's good, but not a silver bullet - everything is a tradeoff.

Nginx Unit is a "Universal Web App Server" brought to you by Nginx. This is a web server that can "directly" communicate with your code base, helping you pass off HTTP requests to your code in a way it can understand.

It supports a bunch of languages and has a module for each supported language. That lets it treat each language specially, as needed.

For PHP support, it has a PHP module that creates PHP processes, similar(ish) to PHP-FPM, but without needing PHP-FPM. Getting rid of PHP-FPM sounds really rad to me, so I decided to see how it works.

Install PHP

We'll install PHP the usual way on any Ubuntu server - using the ppa:ondrej/php repository. This is basically the de-facto way to use PHP on Ubuntu servers. It lets us get the latest PHP and install multiple versions of PHP on the same server (if so desired).

One issue with Unit: It expects the system-set default PHP version to be used. Luckily, we can recompile it's PHP module ourself (see here) to use our ppa:ondrej/php-installed PHP.

First, we'll just install PHP as usual.

sudo add-apt-repository -y ppa:ondrej/php
sudo apt-get install -y php8.2-dev php8.2-embed \
                   php8.2-bcmath php8.2-cli php8.2-common php8.2-curl \
                   php8.2-gd php8.2-intl php8.2-mbstring php8.2-mysql php8.2-pgsql \
                   php8.2-redis php8.2-soap php8.2-sqlite3 php8.2-xml php8.2-zip
curl -sLS https://getcomposer.org/installer | sudo php -- --install-dir=/usr/bin/ --filename=composer

Three things to note:

1. We need php8.2-dev (the -dev version) of PHP to get certain PHP "header" files, allowing us to later compile Unit's PHP module
2. We need php8.2-embed, as this is the SAPI that unit uses to spin up PHP processes.
3. We do NOT need php-fpm, and so we don't install it

Everything else is all the usual "stuff" for a PHP installation, typically seen on a Forge server.

Install Nginx Unit

We can install Nginx Unit as per their docs - nothing special to do there!

I used Ubuntu 22.04, and followed their instructions for that system:

sudo curl --output /usr/share/keyrings/nginx-keyring.gpg  \
      https://unit.nginx.org/keys/nginx-keyring.gpg

echo "deb [signed-by=/usr/share/keyrings/nginx-keyring.gpg] https://packages.nginx.org/unit/ubuntu/ jammy unit
deb-src [signed-by=/usr/share/keyrings/nginx-keyring.gpg] https://packages.nginx.org/unit/ubuntu/ jammy unit
" | sudo tee /etc/apt/sources.list.d/unit.list

sudo apt-get update
sudo apt-get install -y unit

I only installed unit, opting NOT to install unit-dev nor unit-php yet, as we'll need to compile the PHP module ourself.

Manually Build Unit's PHP Module

Next we manually (so uncultured) re-build Unit's PHP module to work with ppa:ondrej/php. This was taken from the aforementioned GitHub issue which helpfully pointed out how we can make this work.

Run the following as user root:

cd /opt

# Latest version of Unit as of this release
VERSION="1.31.0"
curl -O https://unit.nginx.org/download/unit-$VERSION.tar.gz
tar xzf unit-$VERSION.tar.gz
cd unit-$VERSION

./configure --prefix=/usr --state=/var/lib/unit --control=unix:/var/run/control.unit.sock \
    --pid=/var/run/unit.pid --log=/var/log/unit.log --tmp=/var/tmp --user=unit --group=unit \
    --tests --openssl --modules=/usr/lib/unit/modules --libdir=/usr/lib/x86_64-linux-gnu
./configure php --module=php82 --config=php-config8.2
make php82
make install php82-install

# Restart 
systemctl unit restart

# Check logs to ensure PHP module is loaded
cat /var/log/unit.log

We compiled the PHP module ourself, which uses the currently-installed PHP version (taken from the ppa:ondrej/php repository). Success!

Create a Laravel App

In our case, we'll just directly create a new Laravel application on our server:

mkdir -p /var/www
cd /var/www
composer create-project laravel/laravel html

# Ensure files are owned by "unit", the user created
# by unit, so PHP can write to log files, etc
chown -R unit:unit /var/www/html

Easy enough!

Configure Unit

We can now configure Unit to run our application.

Create /var/www/unit.json with:

{
    "listeners": {
        "*:80": {
            "pass": "routes"
        }
    },

    "routes": [
        {
            "match": {
                "uri": "!/index.php"
            },
            "action": {
                "share": "/var/www/html/public$uri",
                "fallback": {
                    "pass": "applications/laravel"
                }
            }
        }
    ],

    "applications": {
        "laravel": {
            "type": "php",
            "root": "/var/www/html/public/",
            "script": "index.php",
            "processes": {}
        }
    }
}

See the video for a full explanation of this configuration.

Then tell Unit to use this configuration to run the application:

sudo curl -X PUT --data-binary @unit.json --unix-socket \
       /var/run/control.unit.sock http://localhost/config/

Head to your application (listening on port 80 currently)! Test it out with curl localhost.

Control API

You can use the Control API to retrieve and update configuration.

Let's GET our configuration:

curl -X GET \
    --unix-socket /var/run/control.unit.sock \
    http://localhost/config/

Pros of Unit

We can remove PHP-FPM! This is great, as it makes throwing our apps into a container a LOT simpler.

Unit also seems to be more efficient - I could NOT get it to break using ab to send 100 requests at a time, with 10,000 total requests.

Cons of Unit

There are some trade-offs!

First, we can't switch PHP versions without re-compiling the Unit PHP module. This means it will be hard (impossible?) to run multiple versions of PHP at the same time while using Unit.

You might also need another HTTP layer in front of Unit (Nginx, Cloudflare, Cloudfront, fly.io's HTTP layer, etc). It turns out that Unit either makes some standard-ish configuration hard, or can't do it at all. Some examples:

1. Gzip compression isn't (yet?) a thing for Unit
2. It makes protecting dot files or other routes difficult
3. It's not easy to set cache headers for static assets

Let's see how an HTTP request gets from your web server into your PHP/Laravel code base.

The moving parts are:

1. Nginx - receives web request, sends it to PHP-FPM via FastCGI
2. PHP-FPM - Takes FastCGI request, spins up processes of PHP and runs your code
3. Laravel - Takes PHP super globals, along with php://input stream, and creates an HTTP Request class

Nginx

Nginx is configured for Laravel/PHP applications with the following config:

server {
    server_name app.chipperci.com;
    root /home/forge/app.chipperci.com/public;

    index index.html index.htm index.php;

    charset utf-8;

    location / {
        try_files $uri $uri/ /index.php?$query_string;
    }

    error_page 404 /index.php;

    location ~ \.php$ {
        fastcgi_split_path_info ^(.+\.php)(/.+)$;
        fastcgi_pass unix:/var/run/php/php8.2-fpm.sock;
        fastcgi_index index.php;
        include fastcgi_params;
    }
}

This is taken from Laravel Forge, with a bunch of items removed for brevity. What we see above are the parts we care about.

The first location / {} uses try_files to attempt to match the request URI to a static file or directory within the configured root, which is /home/forge/app.chipperci.com/public in our example. If it files a static file, it serves it! Otherwise, it runs index.php.

If the file found ends in .php (or we're using the fallback index.php file), then we eventually end up in the location ~ \.php$ {} block. This passes the request off to PHP-FPM via the FastCGI protocol.

It first splits the path at whatever *.php file is in there. This lets PHP get the correct URI of the request - everything AFTER index.php usually.

We also include fastcgi_params which is the information that is populated in the $_SERVER PHP super global.

Finally we pass the request off to PHP-FPM, which in this case is listening via a unix socket file at /var/run/php/php8.2-fpm.sock.

PHP-FPM

PHP-FPM is written in C, and is hard to parse (in other words, I have no idea what's going on in there). A lot of the logic to taking a request and spinning up PHP is in the fpm_request.c file. However the basics are that PHP-FPM manages processes and runs our PHP application through its "entrypoint", the index.php file.

PHP-FPM populates the PHP super globals, any any streams needed such as php://input (that will be the body of the HTTP request if there is one).

Our Code

Your framework likely abstracts the HTTP request into a class that represents the HTTP request itself. In Laravel, that's this class, which extends the underlying Symfony HTTP request here.

Laravel uses the method createFromGlobals(), which creates an HTTP request instance based off of the global information (super globals) and gets the body of the HTTP request from php://input.

Then Laravel can use that to match against registered routes, run controller code, and generally do all the magic it does for us!

PHP-FPM usually comes with a particular setting that's set too low. This results in Gateway errors when you get too much traffic, often before the server even runs out of resources.

Let's see what that setting is, and how to rectify it.

What is PHP-FPM

PHP-FPM is an application gateway. It sits between Nginx (the web server) and your code base. Nginx will get an HTTP request, and is then configured to "proxy" the requests off to PHP-FPM using the fast CGI protocol.

When PHP-FPM receives a request from Nginx, it spins up a process (if one isn't already running), which runs an instance of your PHP application (call the index.php file, or whatever, with all the needed data - $_SERVER, $_GET, $_POST, body of the request, all that good stuff).

The Problem with PHP-FPM

The "problem" (if we want to call it that), is that PHP-FPM has a setting max_children. This is the number of processes it's will to spin up. Since each child process handles a single HTTP request at a time, in serial, the only way to handle concurrent requests is by spinning up more processes. Eventually we can hit the max number of processes configured (max_children), leading to Gateway errors (PHP-FPM refuses to handle the request, nor does it queue the request).

This can happen prematurely - before the server is actually out of resources (RAM, CPU). However it often coincides with your database being overloaded. In that case, requests may not respond in a timely manner, causing a pile-up of pending HTTP requests, eventually leading to max_children being hit and the dreaded Gateway error.

This has always bugged me - it sometimes is self-limited (when max_children is set too low) and other times is disguising a real issue (e.g. database overload).

Debugging PHP-FPM Issues

If you hit a Gateway error, the place to look to see what's happening is in the PHP-FPM log. This is often in /var/log on your linux server. On Debian/Ubuntu servers, it's the /var/log/php8.2-fpm.log (adjust that as needed for your PHP version).

You'll see errors saying "You're going to hit your max limit soon", and perhaps errors like "max_children reached". If you see those, you know you should probably increase your PHP-FPM's configured max_children.

How to Configure PHP-FPM

To configure the correct number of max_children, you need to know how many concurrent requests your server can handle.

I typically do a calculation like this:

floor(RAM available / RAM used per request)

If I have a 4gb RAM server, I'll perhaps allocate 3gb of that to the web server. If each PHP web request takes 100mb, that's 3072 / 50 or roughly 30 concurrent requests that I'll allow the server to handle. That means I can set max_children to 30!

The amount of free RAM goes down a lot when you're competing for resources within the server. If your database is on the same server, then you should allocate less RAM to PHP for serving web requests.

One Important Tip

The best tip I can give you for optimizing this is to get your database off of your web server, and into it's own dedicated server. This gives your database more resources that it likely needs, and frees up your web server to dedicate a LOT more resources to serving web requests.

A Quick Review

We use a Mux to match the incoming request to a Target. The Mux can match incoming requests against the domain used, the port requested on, the URI, and more.

Currently, our Target object lets use define a single upstream (backend) server via the AddTarget() method.

What we want is to allow a Target to have multiple upstreams. The Target can decide how to distribute incoming requests amonst those upstream servers (aka Load Balancing).

Watch Your Language

We've been calling an upstream server a bunch of things interchangably so far (upstream, backend, target). We'll need to firm this language up a bit.

So, our firmed up language:

1. Mux: Matches an incoming request to a Target
2. Target: For a matched request, route the request to an available Upstream
3. Upstreams: A collection of backend servers (each being an "upstream") a Target might send a request to

The part that's new here is that we're allowing a Target to have multiple Upstreams. The Target will be responsible for deciding which Upstream to send a request to.

Refactoring Targets

So we need the ability to send to multiple Upstreams for a matched Target.

Before we do that, let's organize the code a bit more.

I decided the Target struct should be the object responsible for choosing which Upstream to send to. Since we'll be adding logic to our Target struct, let's refactor a bit to put the Target "stuff" into its own file.

We'll add file target.go.

.
├── go.mod
├── go.sum
├── main.go
└── reverseproxy
    ├── listener.go
    ├── reverseproxy.go
    └── target.go

Then we take type Target struct {...} out of reverseproxy.go and plop it into target.go:

// File target.go

package reverseproxy

import (
    "github.com/gorilla/mux"
    "net/url"
    "sync"
)

type Target struct {
    router   *mux.Router
    upstream *url.URL
}

So far, so good. We didn't really change anything yet!

Multiple Upstreams

We're going to add to the Target to accomplish two things:

1. Allow for multiple upstreams
2. Be able to load balance amongst available upstreams

First, we'll change the upstream property to be upstreams (plural) and just make it a slice of *url.URL's. We'll add some other items as well to help with load balancing.

Then we can add a method to the Target struct to help us select an upstream server. We'll hard code a round-robin strategy - no need to abstract different strategies for now.

Here's the updated struct and its shiny, new SelectUpstream() method:

package reverseproxy

import (
    "github.com/gorilla/mux"
    "net/url"
    "sync"
)

type Target struct {
    router       *mux.Router

    // NOTE: New properties here:
    upstreams    []*url.URL
    lastUpstream int
    lock         sync.Mutex
}

// SelectUpstream will load balance amongst available
// targets using a round-robin algorithm
func (t *Target) SelectUpstream() *url.URL {
    count := len(t.upstreams)
    if count == 1 {
        return t.upstreams[0]
    }

    t.lock.Lock()
    defer t.lock.Unlock()

    next := t.lastUpstream + 1
    if next >= count {
        next = 0
    }

    t.lastUpstream = next

    return t.upstreams[next]
}

We added method SelectUpstream(). This will just return a target of our choosing. The Target struct now has property upstreams (plural), which replaced upstream (singular).

Our SelectUpstream() method returns the first upstream (*url.URL) if we only defined one. No load balancing in that case!

Otherwise we do some boring logic to ensure we loop through the given upstreams without accidentally panicing with a index out of range error.

We track the last upstream that we sent a request to via lastUpstream, and we use a Mutex to safely increment said lastUpstream variable for when requests are coming on concurrently.

You can also use atomic ints for that case, which be a bit faster. However this SO answer scared me off of them. However an atomic might be preferred here.

Not too bad, logic-wise!

Defining the Upstreams

Let's next update the easy thing - we'll change our main.go file to define one or more upstreams when we create a Target.

Instead of just passing a string "http://localhost:8000", we'll pass slices of strings []string{"http://localhost:8000", ...}:

// Plenty of stuff omitted for brevity

func main() {
    r := &reverseproxy.ReverseProxy{}

    // Handle URI /foo
    a := mux.NewRouter()
    a.Host("fid.dev").Path("/foo")
    // Add a single upstream
    r.AddTarget([]string{"http://localhost:8000"}, a)

    // Handle anything else
    // Add multiple upstreams
    r.AddTarget([]string{
        "http://localhost:8001",
        "http://localhost:8002",
        "http://localhost:8003",
    }, nil)
}

Where as before we would send AddTarget a string, now we just send a slice of strings []string. This way we can define one or more upstream servers.

Also not too bad, logic-wise!

Directing Requests

Now we're finally ready to update our code and actually do the load balancing.

This is a change in reverseproxy.go. Since the Director is responsible for directing where an incoming request is proxied to, it seems like the right place to add our load balancing logic.

We'll update the Director function:

// Director returns a function for use in http.ReverseProxy.Director.
// The function matches the incoming request to a specific target and
// sets the request object to be sent to the matched upstream server.
func (r *ReverseProxy) Director() func(req *http.Request) {
    return func(req *http.Request) {
        for _, t := range r.targets {
            match := &mux.RouteMatch{}
            if t.router.Match(req, match) {
                var targetQuery = upstream.RawQuery

                // Call our new SelectUpstream method
                upstream := t.SelectUpstream()

                // Send requests to that selected upsteam
                req.URL.Scheme = upstream.Scheme
                req.URL.Host = upstream.Host
                req.URL.Path, req.URL.RawPath = joinURLPath(upstream, req.URL)
                if targetQuery == "" || req.URL.RawQuery == "" {
                    req.URL.RawQuery = targetQuery + req.URL.RawQuery
                } else {
                    req.URL.RawQuery = targetQuery + "&" + req.URL.RawQuery
                }
                if _, ok := req.Header["User-Agent"]; !ok {
                    // explicitly disable User-Agent so it's not set to default value
                    req.Header.Set("User-Agent", "")
                }
                break
            }
        }
    }
}

Instead of referencing t.upstream, we had our target select an upstream and directed the request to that!

The relevant part:

// Call our new method
upstream := t.SelectUpstream()

// Direct our request to the given upstream
req.URL.Scheme = upstream.Scheme
req.URL.Host = upstream.Host
req.URL.Path, req.URL.RawPath = joinURLPath(upstream, req.URL)

Still not too bad! We're cruising along here.

We Did It!

If we build and run the reverse proxy, we'll see that requests are bounced between the 3 backend servers localhost:8001-8003 for incoming requests.

The separate backend for requests matching fid.dev/foo continues to work and send to the one backend server localhost:8000.

If we make requests to something else (without any upstreams being present), the error messages will show the load balancing working:

2022/11/22 18:57:55 http: proxy error: dial tcp [::1]:8002: connect: connection refused
2022/11/22 18:57:57 http: proxy error: dial tcp [::1]:8003: connect: connection refused
2022/11/22 18:57:58 http: proxy error: dial tcp [::1]:8001: connect: connection refused
2022/11/22 18:57:58 http: proxy error: dial tcp [::1]:8002: connect: connection refused
2022/11/22 18:57:58 http: proxy error: dial tcp [::1]:8003: connect: connection refused
... and so on ...

What else are we missing here? Health checks!

Let's see how to test our upstreams health next. We'll start with "passive" health checks.

We'll do just that, giving connections up to 10 seconds to finish their request before shutting down forcefully.

Let's start a little backwards this time, here's an updated main.go file.

Read the previous article for more context. What we do there is call a new Stop() method on my ReverseProxy after we send an interrupt signal:

package main

import (
    "github.com/fideloper/someproxy/reverseproxy"
    "github.com/gorilla/mux"
)

func main() {
    proxy := &reverseproxy.ReverseProxy{}

    // Let's assume we have 2 backends to proxy to
    // localhost:8000 (for our fun API requests)
    // and localhost:8001 (for everything else)

    // Match requests to "localhost/api" and "localhost/api/*"
    r := mux.NewRouter()
    r.Host("localhost").PathPrefix("/api")
    proxy.AddTarget("http://localhost:8000", r)

    // Catch-all for all other requests
    proxy.AddTarget("http://localhost:8001", nil)

    // Listen for http://
    proxy.AddListener(":80")

    // Listen for https://
    proxy.AddListenerTLS(":443", "keys/fid.dev.pem", "keys/fid.dev-key.pem")

    if err := proxy.Start(); err != nil {
        log.Fatal(err)
    }

    // Shutdown when we receive Ctrl+c (interrupt)
    c := make(chan os.Signal, 1)
    
    // We'll accept graceful shutdowns when quit via SIGINT (Ctrl+C)
    // SIGKILL, SIGQUIT or SIGTERM (Ctrl+/) will not be caught.
    signal.Notify(c, os.Interrupt)

    // Block until we receive our signal.
    <-c

    proxy.Stop() // This is the only new thing here
}

The addition of proxy.Stop() is all we changed.

Adding the Stop Method

Next we'll edit reverseproxy.go and add that Stop() method. Let's first talk about what it will do!

It turns out that http.Server has a Shutdown() method. This will handle gracefully closing listeners. If any are active, it will wait indefinitely for them to disconnect.

To make sure "indefinitely" isn't forever, we can pass it a context.Context with a timeout. We'll set that timeout to 10 seconds (I randomly chose 10 seconds).

After 10 seconds, any connections that refuse to finish are forcefully cut and the server will shut down.

It looks a bit like this:

// For a given HTTP server listening for connections
srv := &http.Server{}

srv.Serve(someListener)

// ...

// We can later shut it down gracefully, with a 10 second deadline
context, close := context.WithTimeout(context.Background(), time.Second * 10)
srv.Shutdown(context)

The Shutdown() method is blocking, so it will take up to 10 seconds to complete.

But ... multiple servers!

We need to handle shutting down multiple servers, so our Stop() function is just a tad more complex than the above logic. Here's the updated Stop() method in reverseproxy.go:

// Stop will gracefully shut down all listening servers
func (r *ReverseProxy) Stop() {
    // A context that times out in 10 seconds
    ctx, cancel := context.WithTimeout(context.Background(), time.Second*10)
    defer cancel()

    // A waitgroup allows us to block until all
    // goroutines are finished - when all servers
    // have finished shutting down.
    var wg sync.WaitGroup

    for _, srv := range r.servers {
        // This prevents a nasty and extremely common bug
        // Google for "golang range loop variable re-use"
        srv := srv

        // Tell the WaitGroup to wait for
        // +1 things to finish
        wg.Add(1)
        go func() {
            // Tell the WaitGroup we finished
            // one of the things
            defer wg.Done()

            // Wait up to 10 seconds for the
            // server to shutdown
            if err := srv.Shutdown(ctx); err != nil {
                log.Println(err)
                return
            }
            log.Println("A listener was shutdown successfully")
        }()
    }

    // Block until all servers are shut down
    wg.Wait()
    log.Println("Server shut down")
}

Let's cover what's going on there.

First we create a context. We just tell the cancel() method to run at the end of the function.

We have multiple servers to shutdown, and we want them to shutdown in parallel (not in serial!). Therefore, we'll run each Shutdown method in a goroutine.

This creates a race condition - the function will just finish immediately if we run those in a goroutine, and it's possible our entire app shuts down before the shutdown calls are complete.

To ensure we wait for all servers to finish shutting down, we'll use a sync.Waitgroup, which helps us orchestrate that. The call to wg.Wait() blocks further execution until wg.Done() is called for each wg.Add(1). If we setup 3 servers, we need to make sure all 3 are done shutting down.

This will now gracefully shutdown each server! If a connection is doing something that takes longer than 10 seconds, the context will timeout and the server will forcefully shutdown. In that case, you'll see log output similar to this:

2022/10/08 13:42:04 A listener was shutdown successfully
2022/10/08 13:42:14 context deadline exceeded
2022/10/08 13:42:14 A listener was shutdown successfully
2022/10/08 13:42:14 Server shut down

One server shutdown immediately (it didn't have any active connections), but the other had a long-running connection that exceeded the 10 second limit. It was shutdown forcefully.

And voilà, our reverse proxy will now shutdown gracefully!

It would be way more useful if we could listen for both http:// and https:// (TLS) connections, or even listen on custom ports.

To do that, we need to create multiple listeners, where each listener is a network socket to use to listen for requests (such as 0.0.0.0:443).

We'll run srv.Serve(listener) once for each listener defined.

Expanding on Listeners

Previously we started an HTTP server by just passing it a string address. This address is converted to a net.Listener and used to listen for connections:

srv := &http.Server{Addr: ":80", Handler: r.proxy}

We want to be able to listen on multiple addresses of our choosing.

To accomplish this, we can build a Listener concept into the code.

Let's start by defining a new type: type Listener struct. This will contain things we need to listen and serve http:// or https:// connections.

We can create a new file listener.go for this - Here's the updated project layout:

.
├── go.mod
├── go.sum
├── main.go
└── reverseproxy
    └── listener.go
    └── reverseproxy.go

File listener.go contains a new Listener struct and some methods to make it convenient to use:

package reverseproxy

import "net"

type Listener struct {
    Addr    string
    TLSCert string
    TLSKey  string
}

// Make creates a net.Listen object to be used
// when starting an http server
func (l *Listener) Make() (net.Listener, error) {
    return net.Listen("tcp", l.Addr)
}

// ServeTLS tells us if we should be serving TLS
// connections instead of unsecured connections
func (l *Listener) ServesTLS() bool {
    return len(l.TLSCert) > 0 && len(l.TLSKey) > 0
}

The Listener struct contains an Address compatible with net.Listen(), and optional fields for a TLS certificate/key.

Method Make() generates a net.Listener for us (or an error, if our address Addr is invalid).

Method ServesTLS() returns a boolean - essentially just saying this listener is meant to be used for TLS certificates if the certificate and key fields are used.

Now that we've abstracted the concept of a Listener, we can use it!

Implementing Multiple Listeners

We need to update our ReverseProxy object to make use of multiple listeners.

Let's see what that looks like. Here are the changes within reverseproxy.go:

type ReverseProxy struct {
    listeners []Listener
    proxy     *httputil.ReverseProxy
    servers   []*http.Server
    targets   []*Target
}

// AddListener adds a listener for non-TLS connections on the given address
func (r *ReverseProxy) AddListener(address string) {
    l := Listener{
        Addr: address,
    }

    r.listeners = append(r.listeners, l)
}

// AddListenerTLS adds a listener for TLS connections on the given address
func (r *ReverseProxy) AddListenerTLS(address, tlsCert, tlsKey string) {
    l := Listener{
        Addr:    address,
        TLSCert: tlsCert,
        TLSKey:  tlsKey,
    }

    r.listeners = append(r.listeners, l)
}

// Start will listen on configured listeners
func (r *ReverseProxy) Start() error {
    r.proxy = &httputil.ReverseProxy{
        Director: r.Director(),
    }

    for _, l := range r.listeners {
        listener, err := l.Make()
        if err != nil {
            // todo: Close any listeners that
            //       were created successfully
            //       before one returned error
            return err
        }

        srv := &http.Server{Handler: r.proxy}

        r.servers = append(r.servers, srv)

        // TODO: Handle unexpected errors from our servers
        if l.ServesTLS() {
            go func() {
                if err := srv.ServeTLS(listener, l.TLSCert, l.TLSKey); err != nil && !errors.Is(err, http.ErrServerClosed) {
                    log.Println(err)
                }
            }()
        } else {
            go func() {
                if err := srv.Serve(listener); err != nil && !errors.Is(err, http.ErrServerClosed) {
                    log.Println(err)
                }
            }()
        }
    }

    return nil
}

Let's review what's going on!

Our ReverseProxy struct gets 2 new properties

1. One is a slice of Listener objects.
2. The other is a slice of *http.Server objects.

Each listener needs its own HTTP server to accept and handle connections, so we need one of each per network socket we're listening on.

A network socket is a network interface (address) + port combo being bound to in order to listen for connections.

We add two methods (which is a bit more consistent with the stdlib's HTTP methods) - one for regular connections, and one for TLS connections. These just add to our slice of listeners.

The Start() method used to run this:

srv := &http.Server{Addr: ":80", Handler: r.proxy}
srv.ListenAndServe()

But now we have multiple listener objects! We'll need a server for each listener, and we'll use the Serve / ServeTLS methods instead of ListenAndServe.

Additionally, ListenAndServe() / Serve() / ServeTLS() are all blocking. That means these all need to run in a goroutine, so they can run concurrently.

For each listener, we create a http.Server, passing it our http.ReverseProxy handler. Each server listens on our given address defined by our Listener objects. TLS listeners get the TLS treatment.

We also handle (which is to say, "log") errors here, where as before we just ignored them.

Here's a fun fact: the Serve methods always return an error. If the error is ErrServerClosed, it just means the server was shutdown properly and will no longer accept new connections. All other errors are "true" errors.

Our Start() method now just returns an error if an unexpected error is returned. However, if the Serve methods return an unexpected error, we just log them.

I didn't go through the hoops of using an error channel to handle those errors (yet?).

Running the Server

We can now update our main.go file to try to run this.

package main

import (
    "github.com/fideloper/someproxy/reverseproxy"
    "github.com/gorilla/mux"
)

func main() {
    proxy := &reverseproxy.ReverseProxy{}

    // Let's assume we have 2 backends to proxy to
    // localhost:8000 (for our fun API requests)
    // and localhost:8001 (for everything else)

    // Match requests to "fid.dev/api" and "fid.dev/api/*"
    r := mux.NewRouter()
    r.Host("fid.dev").PathPrefix("/api")
    proxy.AddTarget("http://localhost:8000", r)

    // Catch-all for all other requests
    proxy.AddTarget("http://localhost:8001", nil)

    // Listen for http://
    proxy.AddListener(":80")

    // Listen for https://
    proxy.AddListenerTLS(":443", "keys/fid.dev.pem", "keys/fid.dev-key.pem")

    if err := proxy.Start(); err != nil {
        log.Fatal(err)
    }
}

Wait! TLS keys! I used mkcert to generate some TLS keys (and a local CA to authenticate them) for me. The domain I used is fid.dev. I edited my /etc/hosts file so fid.dev pointed to 127.0.0.1.

I should probably have just used something like fid.local or the more grim fid.localhost. Oh well!

In any case, mkcert Just Worked™ for me as described in its readme (on MacOS Monteray). I placed the keys it generated in a new directory named keys.

We gotta fix a thing

OK, if you try to run this....it will just exit immediately.

Remember how we ran our servers in goroutines? Yeah, there's nothing blocking anymore, so the program just ends (and goroutines get shut down).

We need a way to keep our program running. To do that, we can listen (and wait) for an interrupt signal (SIGINT), which is roughly Windows/Mac/Linux compatible.

This gives us the ability to block (keep the servers running) until we hit ctrl+c. Based on some random Stack Overflow answer, os.Interrupt is the only signal that will work on Windows.

With some additions, main.go now looks like this:

package main

import (
    "github.com/fideloper/someproxy/reverseproxy"
    "github.com/gorilla/mux"
)

func main() {
    proxy := &reverseproxy.ReverseProxy{}

    // Let's assume we have 2 backends to proxy to
    // localhost:8000 (for our fun API requests)
    // and localhost:8001 (for everything else)

    // Match requests to "localhost/api" and "localhost/api/*"
    r := mux.NewRouter()
    r.Host("localhost").PathPrefix("/api")
    proxy.AddTarget("http://localhost:8000", r)

    // Catch-all for all other requests
    proxy.AddTarget("http://localhost:8001", nil)

    // Listen for http://
    proxy.AddListener(":80")

    // Listen for https://
    proxy.AddListenerTLS(":443", "keys/fid.dev.pem", "keys/fid.dev-key.pem")

    if err := proxy.Start(); err != nil {
        log.Fatal(err)
    }

    // Shutdown when we receive Ctrl+c (interrupt)
    c := make(chan os.Signal, 1)
    
    // We'll accept graceful shutdowns when quit via SIGINT (Ctrl+C)
    // SIGKILL, SIGQUIT or SIGTERM (Ctrl+/) will not be caught.
    signal.Notify(c, os.Interrupt)

    // Block until we receive our signal.
    <-c
}

Now our program will run indefinitely until we send it SIGINT (interrupt).

When we hit ctrl+c, the server will shut down. It will also cut off any connections! We'll next see how to do graceful shutdowns.

It would be neat if we could configure the proxy to match some request parameters in order to decide which target (upstream/backend server - I need to decide on a vocabulary) to send the request to.

I noticed that Traefik leveraged gorilla/mux for this. Let's do some theft and "pull a Traefik" (only worse, because I'm not a team of experts).

Matching Routes

Digging into the Gorilla Mux package, we find the Match() method.

That Match() method matches a given request to a registered route. Perfect!

Gorilla/Mux wants us to pass a handler to use when a route is matched, but we'll ignore the Handler stuff. Traefik doesn't ignore that feature but our reverse proxy is different and we do. We just want the matching for now!

We can do something like this:

import "github.com/gorilla/mux"

// Register a route to match a request where
// hostname is "localhost" and path is "/foo"
a := mux.NewRouter()
a.Host("localhost").Path("/foo")

// Match an *http.Request to
// my Mux's registered route
match := &mux.RouteMatch{}
if a.Match(req, match) {
    // Do something if we have a match
}

Our goal is to add some possible "targets" (see, I decided on a word!) to our ReverseProxy and have the Director match an incoming request to the desired upstream target.

Implementing Gorilla Mux

First, we grab gorilla/mux for our project via go get -u github.com/gorilla/mux.

Then, back in reverseproxy.go, we edit some stuff:

type ReverseProxy struct {
    proxy  *httputil.ReverseProxy
    targets []*Target
}

type Target struct {
    router   *mux.Router
    upstream *url.URL
}

The ReverseProxy's Target has become targets (lower case, no longer exported). We won't add targets ourselves directly, but instead use a new AddTarget() method:

// AddTarget adds an upstream server to use for a request that matches
// a given gorilla/mux Router. These are matched via Director function.
func (r *ReverseProxy) AddTarget(upstream string, router *mux.Router) error {
    url, err := url.Parse(upstream)

    if err != nil {
        return err
    }

    if router == nil {
        router = mux.NewRouter()
        router.PathPrefix("/")
    }

    r.targets = append(r.targets, &Target{
        router:   router,
        upstream: url,
    })

    return nil
}

The method AddTarget() is added to the ReverseProxy struct.

One notable bit of logic is that if we pass nil for the router parameter, we create a catch-all router via router.PathPrefix("/").

After we register some targets, we need our Director function to be able to spin through the registered targets, and match any. If they are matched, it sends to that upstream target.

The gorilla/mux lib has the matching function, we just (ab)use it.

// Director returns a function for use in http.ReverseProxy.Director.
// The function matches the incoming request to a specific target and
// sets the request object to be sent to the matched upstream server.
func (r *ReverseProxy) Director() func(req *http.Request) {
    return func(req *http.Request) {
        // Check each target for a match
        for _, t := range r.targets {
            // We don't actually use the match variable
            // but we need to make it to satisfy the
            // Gorilla/Mux Match() method
            match := &mux.RouteMatch{}
            // The magic is here ✨
            if t.router.Match(req, match) {

                // This is all stdlib Director method stuff.
                // We adjusted it to use our matched target.
                targetQuery := t.upstream.RawQuery

                req.URL.Scheme = t.upstream.Scheme
                req.URL.Host = t.upstream.Host
                req.URL.Path, req.URL.RawPath = joinURLPath(t.upstream, req.URL)
                if targetQuery == "" || req.URL.RawQuery == "" {
                    req.URL.RawQuery = targetQuery + req.URL.RawQuery
                } else {
                    req.URL.RawQuery = targetQuery + "&" + req.URL.RawQuery
                }
                if _, ok := req.Header["User-Agent"]; !ok {
                    // explicitly disable User-Agent so
                    // it's not set to default value
                    req.Header.Set("User-Agent", "")
                }
                break // First match wins
            }
        }
    }
}

As the comments suggest, the Director function now spins through the registered Targets, and uses Gorilla/Mux to match them. It directs the proxy to the first matched upstream Target.

We don't handle the case of a catch-all fallback here (if no matches are found). We'll assume we're responsible developers for now and define the fallback in our main.go file.

Tying it Together

Since that's all setup, let's go back to that main.go file and use our new feature!

package main

import (
    "github.com/fideloper/someproxy/reverseproxy"
    "github.com/gorilla/mux"
)

func main() {
    proxy := &reverseproxy.ReverseProxy{}

    // Let's assume we have 2 backends to proxy to
    // localhost:8000 (for our fun API requests)
    // and localhost:8001 (for everything else)

    // Match requests to "localhost/api"
    // and "localhost/api/*"
    r := mux.NewRouter()
    r.Host("localhost").PathPrefix("/api")
    proxy.AddTarget("http://localhost:8000", r)

    // Catch-all for all other requests
    proxy.AddTarget("http://localhost:8001", nil)

    proxy.Start()
}

Here we register 2 targets into our reverse proxy. One captures anything starting (or equal to) /api, as long as the request was made to hostname localhost. These requests will go to upstream server localhost:8000.

The 2nd target is a catch-all route (see, we're responsible!) that sends anything else to localhost:8001.

Note that proxy.AddTarget() can return an error, but I'm ignoring those for brevity.

If we start this up, we'll see it works! You don't even need to have backend servers to test this - we'll see some logging for failed requests.

# Assuming our proxy server is running...

curl localhost/api/foo

curl localhost/whatever

I didn't have any upstream servers running, so these curl requests receive a 502 Bad Gateway response. But, we'll see the following logged from our reverse proxy:

2022/10/07 21:32:09 http: proxy error: dial tcp [::1]:8000: connect: connection refused
2022/10/07 21:32:13 http: proxy error: dial tcp [::1]:8001: connect: connection refused

They're attempting to send to the correct locations! Port 8000 for our request to localhost/api/foo and port 8001 for any other request (localhost/whatever in our case).

Now we can target multiple backend hosts, using parameters of our choosing!

package main

import (
    "log"
    "net/http"
    "net/http/httputil"
    "net/url"
    "strings"
)

func main() {
    // Define the backend server we proxy to
    target, err := url.Parse("http://localhost:8000")

    if err != nil {
        log.Fatal(err)
    }

    // Stolen from `httputil.NewSingleHostReverseProxy()`
    targetQuery := target.RawQuery
    director := func(req *http.Request) {
        req.URL.Scheme = target.Scheme
        req.URL.Host = target.Host
        req.URL.Path, req.URL.RawPath = joinURLPath(target, req.URL)
        if targetQuery == "" || req.URL.RawQuery == "" {
            req.URL.RawQuery = targetQuery + req.URL.RawQuery
        } else {
            req.URL.RawQuery = targetQuery + "&" + req.URL.RawQuery
        }
        if _, ok := req.Header["User-Agent"]; !ok {
            // explicitly disable User-Agent so it's not set to default value
            req.Header.Set("User-Agent", "")
        }
    }

    // Create the proxy object
    proxy := &httputil.ReverseProxy{Director: director}

    // Listen for http:// connections on port 80
    srv := http.Server{Addr: ":80", Handler: proxy}

    // Start the server
    srv.ListenAndServe()
}

// helper functions

func singleJoiningSlash(a, b string) string {
    // snip
}

func joinURLPath(a, b *url.URL) (path, rawpath string) {
    // snip
}

We literally just ripped out the NewSingleHostReverseProxy() method directly. Unfortunately, it calls some non-exportable helper functions, so we ripped those out too. They're mine now.

All they do is add slashes to URL's in way that ensures a slash exists if needed, and that double slashes do not exist. You can find them in stdlib http.httputil.reverseproxy.go.

Instantiating a new httputil.ReverseProxy object lets us keep all the fancy logic of the Reverse Proxy, but then modify / customize what we need later.

Some Refactoring

Let's do something fun. First, I hate having this mess of code in the main namespace. Let's make our own module and tuck away some of this trash. It'll make this a tad harder to write about, but the examples will be simpler.

Here's the new project layout:

.
├── go.mod
├── go.sum
├── main.go
└── reverseproxy
    └── reverseproxy.go

The first thing we're going to do is wrap the httputil.ReverseProxy into our own ReverseProxy struct. This helps us tuck code away into our own modules, and then later we can more easily add some functionality to it.

File reverseproxy.go can have this:

package reverseproxy

import (
    "net/http"
    "net/http/httputil"
    "net/url"
    "strings"
)

type ReverseProxy struct {
    Target *url.URL
    proxy  *httputil.ReverseProxy
}

// Start will listen on configured listeners
func (r *ReverseProxy) Start() error {
    r.proxy = &httputil.ReverseProxy{
        Director: r.Director(),
    }

    // Hard-coding port 80 for now
    srv := &http.Server{Addr: ":80", Handler: r.proxy}

    return srv.ListenAndServe()
}

// Director returns a function for use in http.ReverseProxy.Director.
// The function matches the incoming request to a specific target and
// sets the request object to be sent to the matched upstream server.
func (r *ReverseProxy) Director() func(req *http.Request) {
    return func(req *http.Request) {
        targetQuery := r.Target.RawQuery
        req.URL.Scheme = r.Target.Scheme
        req.URL.Host = r.Target.Host
        req.URL.Path, req.URL.RawPath = joinURLPath(r.Target, req.URL)
        if targetQuery == "" || req.URL.RawQuery == "" {
            req.URL.RawQuery = targetQuery + req.URL.RawQuery
        } else {
            req.URL.RawQuery = targetQuery + "&" + req.URL.RawQuery
        }
        if _, ok := req.Header["User-Agent"]; !ok {
            // explicitly disable User-Agent so it's not set to default value
            req.Header.Set("User-Agent", "")
        }
    }
}

// Helper functions moved here

func singleJoiningSlash(a, b string) string {
    // snip
}

func joinURLPath(a, b *url.URL) (path, rawpath string) {
    // snip
}

So, I created my own ReverseProxy and added a Start() method. The Start() method has some code smells in it, but it's simple to see what's going on so we'll keep it.

I also added a Director() method onto my proxy struct. This generates a Director function for us, which is used by httputil.ReverseProxy. For now, we just copied, pasted, and tweaked the stdlib Director method to get it working. There are only minor tweaks here. Specifiaclly, we don't pass it a target *url.URL, but instead use the Target *url.URL that's defined in our own ReverseProxy struct.

We also moved the helper functions into this file, and I'm still hiding their boring contents for brevity (// snip!).

The main.go file, which uses our ReverseProxy, instantiates and starts the server:

package main

import (
    "github.com/fideloper/someproxy/reverseproxy"
    "log"
    "net/url"
)

func main() {
    target, err := url.Parse("http://localhost:8000")

    if err != nil {
        log.Fatal(err)
    }

    proxy := &reverseproxy.ReverseProxy{
        Target: target,
    }

    proxy.Start()
}

Simple! But there's no features yet. Let's add some!

It turns out that the stdlib has a a decent implementation of a reverse proxy: proxy := httputil.NewSingleHostReverseProxy(backend). As the name implies, it just handles a single host for the backend. It's not very load-balancer-y but it is very reverse-proxy-y.

Let's take a quick look at how to use it:

package main

import (
    "log"
    "net/http"
    "net/http/httputil"
    "net/url"
)

func main() {
    // The backend is another HTTP service listening
    // for requests. Our reverse proxy will send (proxy)
    // requests to it.
    backend, err := url.Parse("http://localhost:8000")

    if err != nil {
        log.Fatal(err)
    }

    // The proxy is a Handler - it has a ServeHTTP method
    proxy := httputil.NewSingleHostReverseProxy(backend)

    // We listen for requests on port 80
    srv := http.Server{Addr: ":80", Handler: proxy}

    srv.ListenAndServe()
}

This reverse proxy accepts http, http/2, TLS, and gRPC connections. There's a bunch of stuff going on in there! You can check out the http.httputil.reverseproxy.go file from stdlib to see more.

Here's a few things it handles:

1. HTTP Trailer headers (needed for gRPC in conjunction with http/2)
2. Hop-by-hop headers
3. HTTP streaming
4. Upgrading HTTP connections (hijacking the TCP connection for Websocket use)

If we run the above code and send requests to http://localhost, it will attempt to proxy any request to localhost:8000. It will return 502 Bad Gateway if you don't have any web server listening on localhost:8000.

Digging In

Let's see what the NewSingleHostReverseProxy is doing.

Its main job is to return an instance of httputil.ReverseProxy. The ReverseProxy isn't itself limited to proxying to a single host, but Golang's stdlib only offers this "example" of a quick single-upstream reverse proxy. More on that later.

If we take a look at the httputil.ReverseProxy struct, there's a few interesting things. Let's take a look at the 2 most interesting.

Director

First, the Director function.

type ReverseProxy struct {
    // Director must be a function which modifies
    // the request into a new request to be sent
    // using Transport. Its response is then copied
    // back to the original client unmodified.
    // Director must not access the provided Request
    // after returning.
    Director func(*http.Request)

    // snip
}

The Director function takes the incoming HTTP request (well, a copy of it) and modifies it. It's modified in a way that makes it ready to be sent to the backend/upstream server. Part of this is setting the requests host, schema (etc) to that of the upstream server so it works when the request is sent to it.

The modified request is eventually sent over to the backend server, but that's not the Director's responsibility. The Director is just responsible for modying the incoming request.

For example, if our proxy (running locally) is sent a request via curl http://localhost:80, and we've configured a backend/upstream server ("target") of http://localhost:8000, then the Directory will take the incoming request copy, and set the Host to localhost:8000. Later, the configured Transport will use that request information to connect to the upstream server and send that request.

Default Director Function

Looking at the NewSingleHostReverseProxy() method, we see that all it does is define a Director function and returns a new ReverseProxy object with that Director.

func NewSingleHostReverseProxy(target *url.URL) *ReverseProxy {
    targetQuery := target.RawQuery
    director := func(req *http.Request) {
        req.URL.Scheme = target.Scheme
        req.URL.Host = target.Host
        req.URL.Path, req.URL.RawPath = joinURLPath(target, req.URL)
        if targetQuery == "" || req.URL.RawQuery == "" {
            req.URL.RawQuery = targetQuery + req.URL.RawQuery
        } else {
            req.URL.RawQuery = targetQuery + "&" + req.URL.RawQuery
        }
        if _, ok := req.Header["User-Agent"]; !ok {
            // explicitly disable User-Agent so it's not set to default value
            req.Header.Set("User-Agent", "")
        }
    }
    return &ReverseProxy{Director: director}
}

Pretty simple, if we continue to ignore all the Reverse Proxy code we didn't look at.

Transport

As mentioned, the Director function "directs" the copied request to the correct location - our backend target http://localhost:8080.

The thing that sends the request to the backend (and returns the response) is named Transport:

type ReverseProxy struct {
    // snip

    // The transport used to perform proxy requests.
    // If nil, http.DefaultTransport is used.
    Transport http.RoundTripper

    // snip
}

The Transport is of type http.RoundTripper, which is yet another interface:

type RoundTripper interface {
    // snipped a big ole comment
    RoundTrip(*Request) (*Response, error)
}

Since we didn't define a Transport, the DefaultTransport is used. The code within it is too complex to paste here - it has a bunch of responsibilities. It's interesting to look at, I suggest you do!

The basics of it, however, are that is makes a round trip! It figures out where the request needs to go, sends it, and then gets the response. How it sends it involves making a TCP connection, handling TLS, and more. Receiving the response may involve waiting for a streamed response to complete.

Round Tripper

The main logic for that is in the Transport's roundTrip() method. The Transport struct's method RoundTrip() is kinda/sorta hidden in http/roundtrip.go, which doesn't compile in JS/WASM contexts (hence method roundTrip() - lower case - being where the real logic is. RoundTrip() just calls roundTrip()).

The roundTrip() method makes some checks, gets a persistent connection object (also defined in that same file), and then calls roundTrip() on that connection. The persistent connection is a connection to the upstream server.

This is actually the more complex logic - concurrently writing the request to the upstream server while also reading for a response that may come before the full request is even sent.

We Don't Need to Care Yet

In any case, there's a lot going on in there! We don't need to really care right now, but I think some advanced features might have us digging into the Transport.

Next, let's make our own reverse proxy! We'll start simple, and then add some nifty features on top of what the stdlib provides for us.

Reverse Proxies are basically load balancers (don't @ me about that definition, look it up yourself).

A few popular ones written in Golang are Traefik and Caddy (remember I mentioned those back in the first article?).

Let's see if we can implement some fraction of their functionality ourselves. Onward!

It's useful if your request handlers can share information about a request.

Often the request data itself (http.Request) has everything you need, but sometimes your application has its own data. For example - the authenticated user.

To help here, one common pattern is to pass a context object through the middleware chain. A middleware can set some data, and the following middleware can see that data. This is generaly done with context objects.

Now, using contexts is generally agreed to be a good thing, but what data you save to a context is disagreed upon. The rules of thumb that I like are:

1. For HTTP requests, only put information in a context that is specific to that request
2. Don't put data into a request that lives on longer than that one request

Something that belongs in a request context: the current user, or perhaps a DB transaction used just for that request.

Something that doesn't belong in a request context: A Logger or DB connection (which is indeed different from a specific transaction).

Context and Cloning

Pontificating about programming aside, there's a few annoying things to explain about Go's context object, especially in regards to HTTP requests.

First, an http.Request object has a few pertinent methods:

1. req.Context() returns the request's context. If none was set on the request, it returns a new context.Background().
2. req.WithContext(ctx) returns a shallow copy of the request with the provided context. Requests are (should be) immutable, and contexts are definitely immutable.

This means adding a context to a request nets us a copied request object with a new context on it.

But just what the hell is a shallow copy of a request?

Here's WithContext(ctx) from stdlib, with a bit of the relevant stdlib comments (which will change after Go 1.19):

// To change the context of a request, such as an incoming request you
// want to modify before sending back out, use Request.Clone. Between
// those two uses, it's rare to need WithContext.
func (r *Request) WithContext(ctx context.Context) *Request {
    if ctx == nil {
        panic("nil context")
    }
    r2 := new(Request)
    *r2 = *r
    r2.ctx = ctx
    return r2
}

Great, so I should never actually use WithContext()?!? I asked people smarter than me (who were also confused, it wasn't just me)! One of those people went to the source to ask.

It turns out, using WithContext() is just fine for our use case. We can run newReq := r.WithContext(myShinyNewCtx) in our middleware, and pass that along as if it is our original request.

Using r.Clone() is a "deep copy". It's better suited for making a completely new copy of the request with it's own "lifecycle". For example, the built-in httputil.NewSingleHostReverseProxy() makes use of Clone() in order to take a received request, and then modify it as needed before passing the cloned & modified request to an upstream server.

Here's the Clone() method:

// Clone returns a deep copy of r with its context changed to ctx.
// The provided ctx must be non-nil.
//
// For an outgoing client request, the context controls the entire
// lifetime of a request and its response: obtaining a connection,
// sending the request, and reading the response headers and body.
func (r *Request) Clone(ctx context.Context) *Request {
    if ctx == nil {
        panic("nil context")
    }
    r2 := new(Request)
    *r2 = *r
    r2.ctx = ctx
    r2.URL = cloneURL(r.URL)
    if r.Header != nil {
        r2.Header = r.Header.Clone()
    }
    if r.Trailer != nil {
        r2.Trailer = r.Trailer.Clone()
    }
    if s := r.TransferEncoding; s != nil {
        s2 := make([]string, len(s))
        copy(s2, s)
        r2.TransferEncoding = s2
    }
    r2.Form = cloneURLValues(r.Form)
    r2.PostForm = cloneURLValues(r.PostForm)
    r2.MultipartForm = cloneMultipartForm(r.MultipartForm)
    return r2
}

It does more stuff! I'm still not sure why a "shallow" copy is safe to use with Middleware while a "deep" copy requires explictly copying some data. It seems like it's concerned with cloning specific types of the http.Request struct defined by the http module (vs "standard" types such as string, bool, or []string).

Anyway, let's do some contexting.

Adding Context

We'll stick with our example of adding information about the current authenticated user. Let's add a Middleware that "adds" the current user to the request's context.

// UserMiddleware gets the current user and adds it to a new context
func UserMiddleware(h http.HandlerFunc) http.HandlerFunc {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        ctx := context.WithValue(r.Context(), "user", "fideloper")
        newReq := r.WithContext(ctx)
        h.ServeHTTP(w, newReq)
    })
}

Contexts are immutable, so each "change" requires creating an new context based off of an old one. We grab r.Context(), which likely is just returning context.Background() as mentioned earlier.

We pass our new request object newReq along in h.ServeHTTP, leaving the old one to die a lonely death when the garbage collector comes calling.

We can add this into our Middleware stack, and then we get a user object (just a string for now) that any other middleware/handler can retrieve.

Here's the whole thing:

package main

import (
    "context"
    "fmt"
    "log"
    "net"
    "net/http"
)

// Middleware is a func type that
// allows for chaining middleware
type Middleware func(http.HandlerFunc) http.HandlerFunc

// CompileMiddleware takes the base http.HandlerFunc h
// and wraps around the given list of Middleware m
func CompileMiddleware(h http.HandlerFunc, m []Middleware) http.HandlerFunc {
    if len(m) < 1 {
        return h
    }

    wrapped := h

    // loop in reverse to preserve middleware order
    for i := len(m) - 1; i >= 0; i-- {
        wrapped = m[i](wrapped)
    }

    return wrapped
}

// Let's define some middleware!

// LogMiddleware logs some output 
// for each request received
func LogMiddleware(h http.HandlerFunc) http.HandlerFunc {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        log.Printf("%s: %s", r.Method, r.RequestURI)
        h.ServeHTTP(w, r)
    })
}

// UserMiddleware gets the current user 
// and adds it to a new context
func UserMiddleware(h http.HandlerFunc) http.HandlerFunc {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        ctx := context.WithValue(r.Context(), "user", "fideloper")
        newReq := r.WithContext(ctx)
        h.ServeHTTP(w, newReq)
    })
}

// RateLimitMiddleware limits how often
// a request can be made from a given client
func RateLimitMiddleware(h http.HandlerFunc) http.HandlerFunc {
	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        if rateLimitBreached(r) {
            writer.WriteHeader(403)
            fmt.Fprint(writer, "Rate limit reached")
            return
        }

		h.ServeHTTP(w, r)
	})
}

func main() {
    mux := http.NewServeMux()

    // Define our middleware stack
    // Last middleware will be run first
    stack := []Middleware{
        LogMiddleware,
        UserMiddleware,
        RateLimitMiddleware,
    }

    // Assign our actual HTTP Handler to a variable
    handleAllRequests := func(writer http.ResponseWriter, r *http.Request) {
        writer.WriteHeader(200)
        fmt.Fprint(writer, "HELLO!")
        fmt.Fprintf(writer, " HERE IS YOUR USER: %s", r.Context().Value("user"))
    }

    // Set our handler as a "wrapped" handler - each middleware is called
    // before finally calling the handleAllRequests http Handler
    mux.HandleFunc("/", CompileMiddleware(handleAllRequests, stack))

    srv := &http.Server{
        Handler: mux,
    }

    ln, err := net.Listen("tcp", ":80")
    if err != nil {
        panic(err)
    }

    srv.Serve(ln)
}

In addition to creating the UserMiddleware and adding it to the stack, we updated the base handler to print out information about the current user, retrieved from the request context.

That's this part:

// Assign our actual HTTP Handler to a variable
handleAllRequests := func(writer http.ResponseWriter, r *http.Request) {
    writer.WriteHeader(200)
    fmt.Fprint(writer, "HELLO!")
    fmt.Fprintf(writer, " HERE IS YOUR USER: %s", r.Context().Value("user"))
}

But I Want Types!

One thing sort of sucks: The type any.

The context.WithValue method accepts a value of type any, and r.Context().Value("foo") can return a value of type any.

This means Go's compiler (and our IDE's) can't enforce types, nor help us to know what data is being get/set in the context object. But we want type safety! That's why we use Go!

This article (and this one) covers some ways to get type saftey. I've not settled on what I like best, but here's a stab at it.

First, let's assume that our context shouldn't just receive a string representing a user. We'll instead make a User struct and some helper functions to manage it:

// Still a bit contrived,
// but bear with me
type User struct {
    Username string
}

// setUser adds a user to a context, returning 
// a new context with the user attached
func setUser(ctx context.Context, u *User) context.Context {
    return context.WithValue(ctx, "user", u)
}

// getUser returns an instance of User,
// if set, from the given context
func getUser(ctx context.Context) *User {
    user, ok := ctx.Value("user").(*User)

    if !ok {
        return nil
    }

    return user
}

These helper functions gives us a type-safe way to manage getting/setting the User to/from our context, and gives the compiler something to chew on.

Our UserMiddleware becomes this:

// UserMiddleware gets the current user and adds it to a new context
func UserMiddleware(h http.HandlerFunc) http.HandlerFunc {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        ctx := setUser(r.Context(), &User{
            Username: "fideloper",
        })
        newReq := r.WithContext(ctx)
        h.ServeHTTP(w, newReq)
    })
}

The only change there is to use the setUser function to add a new User to the context (which returns a new Context - remember, contexts are immutable).

Then we can update our base handler to use getUser to retrieve the User. I chose to return nil if no user is associated, rather than an error. You do you.

// Assign our actual HTTP Handler to a variable
handleAllRequests := func(writer http.ResponseWriter, r *http.Request) {
    writer.WriteHeader(200)
    fmt.Fprint(writer, "HELLO!")
    
    user := getUser(r.Context())

    if user != nil {
        fmt.Fprintf(writer, " HERE IS YOUR USER: %s", user.Username)
        return
    }

    fmt.Fprint(writer, " NO USER AUTHENTICATED")
}

Here's the whole thing:

package main

import (
    "context"
    "fmt"
    "log"
    "net"
    "net/http"
)

type User struct {
    Username string
}

func setUser(ctx context.Context, u *User) context.Context {
    return context.WithValue(ctx, "user", u)
}

func getUser(ctx context.Context) *User {
    user, ok := ctx.Value("user").(*User)

    if !ok {
        return nil
    }

    return user
}

// Middleware is func type that allows for chaining middleware
type Middleware func(http.HandlerFunc) http.HandlerFunc

// CompileMiddleware takes the base http.HandlerFunc h and wraps around the given list of Middleware m
func CompileMiddleware(h http.HandlerFunc, m []Middleware) http.HandlerFunc {
    if len(m) < 1 {
        return h
    }

    wrapped := h

    // loop in reverse to preserve middleware order
    for i := len(m) - 1; i >= 0; i-- {
        wrapped = m[i](wrapped)
    }

    return wrapped
}

// Let's define some middleware!

// LogMiddleware logs some output for each request received
func LogMiddleware(h http.HandlerFunc) http.HandlerFunc {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        log.Printf("%s: %s", r.Method, r.RequestURI)
        h.ServeHTTP(w, r)
    })
}

// UserMiddleware gets the current user and adds it to a new context
func UserMiddleware(h http.HandlerFunc) http.HandlerFunc {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        ctx := setUser(r.Context(), &User{
            Username: "fideloper",
        })
        newReq := r.WithContext(ctx)
        h.ServeHTTP(w, newReq)
    })
}

// RateLimitMiddleware limits how often a request can be made from a given client
func RateLimitMiddleware(h http.HandlerFunc) http.HandlerFunc {
	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        if rateLimitBreached(r) {
            writer.WriteHeader(403)
            fmt.Fprint(writer, "Rate limit reached")
            return
        }

		h.ServeHTTP(w, r)
	})
}

func main() {
    mux := http.NewServeMux()

    // Define our middleware stack
    // Last middleware will be run first
    stack := []Middleware{
        LogMiddleware,
        UserMiddleware,
        RateLimitMiddleware,
    }

    // Assign our actual HTTP Handler to a variable
    handleAllRequests := func(writer http.ResponseWriter, r *http.Request) {
        writer.WriteHeader(200)
        fmt.Fprint(writer, "HELLO!")
        user := getUser(r.Context())

        if user != nil {
            fmt.Fprintf(writer, " HERE IS YOUR USER: %s", user.Username)
            return
        }

        fmt.Fprint(writer, " NO USER AUTHENTICATED")
    }

    // Set our handler as a "wrapped" handler - each middleware is called before
    // finally calling the handleAllRequests http Handler
    mux.HandleFunc("/", CompileMiddleware(handleAllRequests, stack))

    srv := &http.Server{
        Handler: mux,
    }

    ln, err := net.Listen("tcp", ":80")
    if err != nil {
        panic(err)
    }

    srv.Serve(ln)
}

In reality I'd have several files and/or some modules of my own here to manage Users, Middleware, etc. In this examples, we're throwinng it all into one file.

But now we know how to use context objects to pass data through our middleware and handlers!

No More Web Servers

That's a wrap on web servers. The next thing we'll look into is more fun: Reverse Proxies.

There's more than one way to go about this, but they all use http.Handler. If you google it, you'll come across examples that have you nest function calls in a way that gets grim pretty quicly.

func(w http.Writer, seriously(
    who(
        wants(
            this(req)
        )
    )
) {}

However, if you google "golang chainable middleware" (or similar), you'll find a much nicer pattern! Let's explore it, and see how Handlers help us here.

Generating a Middleware

Let's see a function that generates a middleware:

// LogMiddleware returns a function that logs
// related output for each request received.
func LogMiddleware(next http.HandlerFunc) http.HandlerFunc {
	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		log.Printf("%s: %s", r.Method, r.RequestURI)
		next.ServeHTTP(w, r)
	})
}

We'll deal with http.HandlerFunc instead of http.Handler in this case. The only reason for this is that it's easier to pass a regular function (see the previous article on Handlers if you haven't read it).

Function LogMiddleware takes an instance of http.HandlerFunc and returns another http.HandlerFunc.

The function we return does some logic, and then calls ServeHTTP on the "next" http.HandlerFunc. We know the ServeHTTP method is available to call thanks to the magic of http.HandlerFunc as described in the previous article.

We can keep nesting middleware having each one call the "next" Handler. Let's write some code and make that idea concrete.

Codify the Middleware

First, we'll codify this "pattern" of generating Middleware functions as a type:

// Middleware is func type that allows
// for chaining middleware
type Middleware func(http.HandlerFunc) http.HandlerFunc

Defining this gives us the ability to enforce types, which we'll see later.

Chaining the Middleware

The LogMiddleware function logs info about the request before calling the "next" middleware. Every middleware does this until the last Handler is run (or if a middleware decides to short-circuit the process and do something else, e.g. return a "not authorized" response).

Since each middleware calls the "next" middleware, we call this a chain of middleware.

That becomes a bit clearer if you see multiple middleware in use. Let's pretend we have 2 middleware:

// LogMiddleware logs some output for each request received
func LogMiddleware(h http.HandlerFunc) http.HandlerFunc {
	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		log.Printf("%s: %s", r.Method, r.RequestURI)
		h.ServeHTTP(w, r)
	})
}

// RateLimit middleware limits how often a
// request can be made from a given client
func RateLimitMiddleware(h http.HandlerFunc) http.HandlerFunc {
	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        // Love me some handy-wavy magic
        if rateLimitBreached(r) {
            // 429 Too Many Requests
            writer.WriteHeader(429)
            fmt.Fprint(writer, "Rate limit reached")
            return
        }

		h.ServeHTTP(w, r)
	})
}

We have 2 middleware, and then we have a function that actually handles the web request:

func(writer http.ResponseWriter, r *http.Request) {
    writer.WriteHeader(200)
    fmt.Fprint(writer, "HELLO")
}

Let's whip up a helper function that will wrap an HTTP handler function (the thing doing the actual work of responding to a request) in our various middleware:

// CompileMiddleware takes the base http.HandlerFunc h 
// and wraps it around the given list of Middleware m
func CompileMiddleware(h http.HandlerFunc, m []Middleware) http.HandlerFunc {
	if len(m) < 1 {
		return h
	}

	wrapped := h

	// loop in reverse to preserve middleware order
	for i := len(m) - 1; i >= 0; i-- {
		wrapped = m[i](wrapped)
	}

	return wrapped
}

Putting it all together along with our basic web server looks like this:

package main

import (
	"fmt"
	"log"
	"net"
	"net/http"
)

// Middleware is func type that allows for
// chaining middleware
type Middleware func(http.HandlerFunc) http.HandlerFunc

// CompileMiddleware takes the base http.HandlerFunc h 
// and wraps it around the given list of Middleware m
func CompileMiddleware(h http.HandlerFunc, m []Middleware) http.HandlerFunc {
	if len(m) < 1 {
		return h
	}

	wrapped := h

	// loop in reverse to preserve middleware order
	for i := len(m) - 1; i >= 0; i-- {
		wrapped = m[i](wrapped)
	}

	return wrapped
}

// Let's define the middleware!

// LogMiddleware logs some output for each
// request received
func LogMiddleware(h http.HandlerFunc) http.HandlerFunc {
	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		log.Printf("%s: %s", r.Method, r.RequestURI)
		h.ServeHTTP(w, r)
	})
}

// RateLimit middleware limits how often a
// request can be made from a given client
func RateLimitMiddleware(h http.HandlerFunc) http.HandlerFunc {
	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        if rateLimitBreached(r) {
            writer.WriteHeader(403)
            fmt.Fprint(writer, "Rate limit reached")
            return
        }

		h.ServeHTTP(w, r)
	})
}

func main() {

	mux := http.NewServeMux()

	// Define our middleware stack
	// These run in the order given
	stack := []Middleware{
        LogMiddleware,
		RateLimitMiddleware,
	}

    // Assign our base HTTP Handler to a variable
	handleAllRequests := func(writer http.ResponseWriter, r *http.Request) {
		writer.WriteHeader(200)
		fmt.Fprint(writer, "HELLO")
	}

    // Set our handler as a "wrapped" handler.
    // Each middleware is called before finally
    // calling the handleAllRequests http Handler
	mux.HandleFunc("/", CompileMiddleware(handleAllRequests, stack))

	srv := &http.Server{
		Handler: mux,
	}

	ln, err := net.Listen("tcp", ":80")
	if err != nil {
		panic(err)
	}

	srv.Serve(ln)
}

Fairly simple, if a little verbose (just like Golang itself). Any request will be logged, then checked against a rate limit. If the rate limit is not reached, then our handleAllRequests handler is finally run.

Note that we preserve the order that middleware are run. Our base handler is run last.

Quick Review

The annoying part (but the part that makes this interesting for me to learn and write about) is how Golang's type system can be a bit obtuse.

A little review from the last article: Golang uses a type HandlerFunc func(...) to define a type that is itself a function. That function type provides method ServeHTTP. This is so we can pass a regular function and have the http stdlib convert it to an http.Handler (which wants that ServeHTTP method).

On top of that, we abuse it a bit for our Middleware, allowing us to create a chain of http.Handler's (technically http.HandlerFunc's) that can call each sibling middleware (in a specific order!) before finally calling the actual HTTP handler that returns a response.

Sidenote: Any middleware that doesn't call ServeHTTP breaks the chain. This is on purpose - it can abort the chain and respond with whatever makes sense if needed. That's why middleware are often used on routes that require authentication. No use processing the request further if we know the user needs to be authenticated to perform the action.

So, now we have middleware! Let's next see how to share information amongst our middleware.

The http.Handler is just an interface:

type Handler interface {
    ServeHTTP(ResponseWriter, *Request)
}

The http.Server has a Handler property, which wants a http.Handler:

# From stdlib http module:
type Server struct {
    // snip
    Handler Handler // handler to invoke, http.DefaultServeMux if nil
}

Most properties in the Server have a long comment describing it. This property does not, but the little comment it does have tells us it uses the DefaultServeMux if it's nil. That's the behavior we saw in the previous article.

In the example in the last article, we created a ServeMux and a Server:

mux := http.NewServeMux()
mux.HandleFunc("/", func(writer http.ResponseWriter, r *http.Request) {
    writer.WriteHeader(200)
    fmt.Fprint(writer, "HELLO")
})

srv := &http.Server{
    Handler: mux,
}

We passed a http.ServeMux to the server's Handler. That ServeMux object satisifes the http.Handler interface because it has a ServeHTTP method.

The interface doesn't care what the ServeHTTP method does (that's not the job of an interface). The Mux's ServeHTTP method happens to match an HTTP request to a user-defined http.Handler (yes, another Handler!) and calls ServeHTTP on that Handler.

That process looks a bit like the below, where the ServeHTTP method that's on the ServeMux struct matches a user-defined Handler and calls ServeHTTP on it:

// ServeHTTP dispatches the request to the handler whose
// pattern most closely matches the request URL.
func (mux *ServeMux) ServeHTTP(w ResponseWriter, r *Request) {
    // snip

    // Match the request URI to
    // a user-registered route
    h, _ := mux.Handler(r)

    // The user-defined route is
    // an instance of http.Handler
    h.ServeHTTP(w, r)
}

When I saw a "user-defined handler", I mean this thing:

mux.HandleFunc("/", func(writer http.ResponseWriter, r *http.Request) {
    writer.WriteHeader(200)
    fmt.Fprint(writer, "HELLO")
})

This is a bit confusing at first. The ServeMux satisfies the Handler interface, but it also then matches an incoming request to a registered handler. The registered handler also satisfies the Handler interface, and so we can call ServeHTTP on that handler.

That happens a few more times! It's a whole chain of Handlers! Here's roughly what the code path is:

1: An incoming request (eventually) triggers the creation of a http.serverHandler (a private, unexported object)

2: http.serverHandler has a ServeHTTP method! It's the first ServeHTTP method called in the "chain"

3: http.serverHandler contains a reference to the Server and uses it to call the server's Handler (which, remember, is often an instance of ServeMux, although it doesn't have to be)

4: Ours is a Mux, and the Mux matches the incoming requestss route to a http.HandlerFunc (the function we provided as a handler), and calls ServeHTTP on that handler!

We can look at the code a bit to see that more clearly.

The chain of Handler calls roughly looks like this, which I copied/pasted from stdlib, and then tweaked to make sense:

// A. Deep in http/server.go...

//    This is the top-level ServeHTTP call
//    serveHandler has a reference to the Server
sh := serverHandler{srv: c.server} 
sh.ServeHTTP(w, w.req)


// B. Within serverHandler.ServeHttp()...

//    A Server is a prop of the serverHandler.
//    It calls the server's Handler (ServeMux).
handler := sh.srv.Handler
if handler == nil {
    // Look, the default ServeMux!
    handler = DefaultServeMux
}

handler.ServeHTTP(rw, req)


// C. Within handler.ServeHTTP()...

//    Our handler function has been converted to a HandlerFunc
//    which gives it the ServeHTTP() method that's called here:
userDefinedHandler := mux.Handler(request)
userDefinedHandler.ServeHTTP(w, r)

🐢 It's Handlers all the way down.

HandlerFunc

In the code comments directly above I mentioned that our handler function is "converted" to a http.HandlerFunc.

Here's the handler function we defined previously:

mux := http.NewServeMux()

// We pass a regular function as a handler function
mux.HandleFunc("/", func(writer http.ResponseWriter, r *http.Request) {
    writer.WriteHeader(200)
    fmt.Fprint(writer, "HELLO")
})

The function we passed there actually satisifes Handler even though it's not explicitly named ServeHTTP and is not typed as a http.Handler! However, we actually called ServeHTTP() on it, and it ran our handler code. That's weird!

// Somehow this runs the handler code we wrote
// even tho what we passed was a regular func()
// and didn't define something with a ServeHTTP
// method on it.
userDefinedHandler := mux.Handler(request)
userDefinedHandler.ServeHTTP(w, r)

The trick is the http.HandlerFunc "conversion". Let's see how that works with an example.

It does some whack nonsense, just bear with me.

package main

import (
    "fmt"
    "net"
    "net/http"
)

// MyHandler is of type "func", with a specific function signature. Weird!
type MyHandler func(http.ResponseWriter, *http.Request)

// ServeHTTP adds a function with the correct
// signature to make it satisfy http.Handler.
// Note that MyHandler `m` is *callable* as a
// function. We can, and do, call `m(w, r)`!
func (m MyHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
    m(w, r)
}

func main() {
    mux := http.NewServeMux()

    // Our function gets turned into an instance of MyHandler,
    // which provides method ServeHTTP, and calls the func we
    // passed into mux.Handle() here.
    //
    // * Note that this method requires us to use mux.Handle,
    //   not mux.HandleFunc
    mux.Handle("/", MyHandler(func(writer http.ResponseWriter, r *http.Request) {
        writer.WriteHeader(200)
        fmt.Fprint(writer, "HELLO")
    }))

    srv := &http.Server{
        Handler: mux,
    }

    ln, err := net.Listen("tcp", ":80")
    if err != nil {
        panic(err)
    }

    srv.Serve(ln)
}

This is a bit weird unless you're pretty familiar with Golang. The pattern was new to me.

In Golang, you can just define your own types. Above, we defined (named) a type MyHandler. Its of type func! That function has a specific signature.

We then give MyHandler a ServeHTTP method! This was new to me - I'm used to creating struct types and adding methods to those, but here we created MyHandler as type func ... and then we added a method on that!

To repeat myself, this is the part I'm talking about:

// MyHandler is of type "func", with a specific function signature. Weird!
type MyHandler func(http.ResponseWriter, *http.Request)

// ServeHTTP adds a function with the correct
// signature to make it satisfy http.Handler.
// Note that MyHandler `m` is *callable* as a
// function. We can, and do, call `m(w, r)`!
func (m MyHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
    m(w, r)
}

By adding the ServeHTTP method, the MyHandler type now satisfies http.Handler interface.

What's weird is that ServeHTTP calls m(writer, r). Turns out...you can do that. Since MyHandler is a func you can just call it like a function.

func (m MyHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
    // We can actually call "m" as a function. Neat!
    // "m" is from `func(m MyHandler)` where I named 
    // the instance of this function "m"
    m(w, r)
}

After we define all of that, we pass a regular ole' function with our handling code (status 200, string "HELLO"), but we typecast that handler function as a MyHandler. As I've said before, MyHandler satisfies interface http.Handler, so we're effectively able to take our regular function and pretend it's an http.Handler.

Fun fact: This is what the stdlib does.

It turns out MyHandler is an exact copy of the following stdlib code from http/server.go, which defines type http.HandlerFunc:

// The HandlerFunc type is an adapter to allow the use of
// ordinary functions as HTTP handlers. If f is a function
// with the appropriate signature, HandlerFunc(f) is a
// Handler that calls f.
type HandlerFunc func(ResponseWriter, *Request)

// ServeHTTP calls f(w, r).
func (f HandlerFunc) ServeHTTP(w ResponseWriter, r *Request) {
    f(w, r)
}

Note the comment from the stdlib code:

HandlerFunc type is an adapter to allow the use of ordinary functions as HTTP handlers.

We allowed ourselves to write an "ordinary function" as a handler even though we technically needed to satify interface http.Handler. Instead of creating a struct (or whatever) and adding a method ServeHTTP() on it, we can just pass a function to mux.HandleFunc.

Being asked to pass a regular function is just syntactic sugar, and also confusing. It hides an important implementation detail of Golang's http module - handlers!

(Maybe the core team was as sick as Handlers we are, and hid them from us.)

What's the Point?

In the first post, I mentioned asking Caddy's community forum a question. That question came from trying to figure out where the hell ServeHTTP came from. I couldn't find the complete code path.

The whole point of me writing this is basically my excitment in finally figuring it out.

The really nifty surprise was seeing how (ab)used the http.Handler type is! It's everywhere!

Sidenote - the thing we did above might actually be useful for you to explicitly use. Here's an example of how this pattern might help with error handling in your own http applications.

Not only are Handlers all over the place, they often call each other in a chain. It's almost like a design pattern! Yes, that's a hint!

We'll next see how we can use this knowledge in some cool ways.

Eventually I got to an understanding where I could ask a coherent question on Caddy's community forum about Caddy.

Matt (creator of Caddy) was generous enough to give me an extremely solid answer, pointing out certain parts of the stdlib http module that did what I was asking about. It dawned on me that my grasp on how Golang handles HTTP was more tenuous than I thought.

So I set about seeing what I could understand! And I wrote it down. So,here's some ċ̷͍͔o̷̯͓̓̒n̸̨͍̫͒͛t̷̛̫̞̃̀̓̄ë̴͇̜͈̗͉́͗̽ń̸̞̮͈͎t̴̝͉̹̤̖͌͗.

The simplest lil' HTTP server

Golang handles HTTP natively. There's a lot that goes into that - http/1, http/2, h2c, TLS, websockets, trailer headers, upgrading connections - the list goes on!

But we're not there yet. Let's just make a very simple web server:

package main

import (
    "fmt"
    "net/http"
)

func main() {
    http.HandleFunc("/", func(writer http.ResponseWriter, r *http.Request) {
        // Send a 200 response. This is technically
        // superfluous as it implicitly sends a 200
        // when we write any response to the writer
        writer.WriteHeader(200)

        // Here we write "HELLO" to the ResponseWriter
        fmt.Fprint(writer, "HELLO")
    })

    http.ListenAndServe(":80", nil)
}

We used function http.HandleFunc to pass a URI to match, and a "handler function". The handler function takes a http.ResponseWriter which you write response data to, and a http.Request object with the request data.

By default you need to handle GET vs POST (etc) yourself by reading r.Method. Fancy libraries such as gorilla/mux help you there.

The writer.WriteHeader method just takes an HTTP status code. If we omitted it, a 200 response status would have been sent when we wrote anything to the writer, like our "HELLO" string.

The WriteHeader() method is just sending the HTTP response header, e.g. HTTP/1.1 200.

Here's something more interesting: The function passed to HandleFunc actually handles all requests. What we're passing in for a URI is technically a "URI pattern". Patterns ending in a trailing slash / are "rooted subtrees", which is Golang's insufferable way of saying that it'll match the given URI and anything after it.

The URI pattern "/" will handle any URI not otherwise matched. We don't have any other URI patterns defined, so it's a catch-all route!

But why does this work?

I set a catch-all handler, and then just told http to ListenAndServe on some port.

But those are two top-level functions defined in the stdlib http module. There's no obvious connection between those 2 actions! Shouldn't I have had to pass that handler to the server somehow?

// These two top-level functions from http
// don't appear to be "connected" in any
// way at first.
http.HandleFunc("/", func(writer http.ResponseWriter, r *http.Request) { 
    // snip
});

// In fact, we pass nil to the param
// that normally would take a handler
http.ListenAndServe(":80", nil)

It turns out that the http stdlib module has default objects and uses them if we don't define those objects ourself.

1. The http.HandleFunc() function adds a handler to a http.DefaultServerMux object, which is conveniently predefined
2. Deep in http/server.go, a serverHandler{} struct is created. It has a ServeHTTP method on it. This method checks if a handler (the mux) is defined on the Server object. If not, it uses the http.DefaultServerMux

So the simple lil' HTTP server works because of syntactic sugar. There are default objects the http module uses unless you explicitly define them.

Explicitly defining things

Let's take this exact same setup but make it more complicated. For science!

package main

import (
    "fmt"
    "net"
    "net/http"
)

func main() {
    mux := http.NewServeMux()
    mux.HandleFunc("/", func(writer http.ResponseWriter, r *http.Request) {
        writer.WriteHeader(200)
        fmt.Fprint(writer, "HELLO")
    })

    srv := &http.Server{
        Handler: mux,
    }

    ln, err := net.Listen("tcp", ":80")
    if err != nil {
        panic(err)
    }

    srv.Serve(ln)
}

We have more going on here, but with the exact same result. We took the syntactic sugar and made it less sweet.

Previously, the http module added the route and handler function to http.DefaultServerMux.

Here we create a Mux ourselves, and then register our route against it. The HandleFunc method is exactly the same, but one is "global" to the http module and one is on ServeMux objects.

// this:
http.HandleFunc("/", func(writer http.ResponseWriter, r *http.Request) {
    // snip
})

// versus this:
mux := http.NewServeMux()
mux.HandleFunc("/", func(writer http.ResponseWriter, r *http.Request) {
    // snip
})

A Mux is a "HTTP request multiplexor" and is responsible for matching incoming requests against a list of registered routes. It sends requests to the correct handler.

After that, we create an instance of http.Server, passing it the Mux as its Handler.

srv := &http.Server{
    Handler: mux,
}

That's curious, though.

If the ServeMux is a Mux, and we pass handler functions to that Mux, why is the ServeMux object referred to as a Handler within the http.Server object?

Interestingly, ServeMux is actually an http.Handler, meaning it "satisfies" interface http.Handler - it has a ServeHTTP() method on it:

// ServeMux has a method ServeHTTP()!
type Handler interface {
    ServeHTTP(ResponseWriter, *Request)
}

So http.Server's just wants any http.Handler. It doesn't actually need to be a ServeMux, but we usually use one to route specific requests to the code of our choice.

Here's another tidbit: ServeHTTP(http.ResponseWriter, *http.Request) has an equivalant signature to the handler function we passed to mux.HandleFunc(). Suspicious! Let's table that for a hot second, but keep it in mind.

// Our handler:
func(writer http.ResponseWriter, r *http.Request) {
    // snip
}

// Is basically an unnamed `ServeHTTP` method:
type Handler interface {
    ServeHTTP(ResponseWriter, *Request)
}

The ServeHTTP() method will come up a lot, the http module leans on the http.Handler interface pretty hard.

Finally, we create a network listener, and pass that to the server. The server will take the listener and Accept() new connections/data on the defined network socket for HTTP connections (port 80 on all networks, in this case).

So, all of this work boils down to a less-sweet way of doing exactly what our most basic web server did.

Along the way, we learned about the ServeMux, creating http.Server instances, and noticing that the mux is actually a Handler.

We also saw that we can create a network listener ourselves and pass it to our server.

I've hinted that Handlers are sort of interesting. Let's get into that next.

I came across this issue, which had a great idea.

In this scenario, we replace the HTTP Client used by the Stripe PHP SDK so we don't make any HTTP requests. Then we do some work to return some pre-created responses (fixtures, I suppose we'll call those).

To create the fixtures, I went to the Stripe API docs and copied/pasted the JSON responses expected for any particular calls.

In my case, there were three calls to the Stripe API:

1. Creating a customer
2. Retrieving a customer
3. Creating a Stripe Checkout session

The Test

Here's what one of the tests looked like. Obviously this could be abstracted some more, but this served my purposes just fine.

In my case, I'm testing creating a Stripe Session, which is a required step before redirecting a user to a hosted Stripe Checkout page.

Here is file tests/Feature/CreateStripeSessionTest.php:

<?php

namespace Tests\Feature;

use App\Models\User;

use Stripe\ApiRequestor;
use Stripe\HttpClient\ClientInterface;

use Laravel\Jetstream\Jetstream;

use Tests\TestCase;
use Illuminate\Support\Str;
use Illuminate\Foundation\Testing\WithFaker;
use Illuminate\Foundation\Testing\RefreshDatabase;

class CreateStripeSessionTest extends TestCase
{
    use RefreshDatabase;

    /** @test */
    public function creates_stripe_session_and_redirects()
    {
        // Create the mock HTTP client used by Stripe
        $mockClient = new MockClient;
        ApiRequestor::setHttpClient($mockClient);

        $user = User::factory()->create([
            'stripe_id' => 'cus_'.Str::random(),
        ]);

        $response = $this->actingAs($user)->post(route('create-session'), [
            'intent' => 'some-meta-data-required',
        ]);

        $response->assertRedirect($mockClient->url)
    }
}

// Mock the Stripe API HTTP Client

# Optionally extend Stripe\HttpClient\CurlClient
class MockClient implements ClientInterface
{
    public $rbody = '{}';
    public $rcode = 200;
    public $rheaders = [];
    public $url;

    public function __construct() {
        $this->url = "https://checkout.stripe.com/pay/cs_test_".Str::random(32);
    }

    public function request($method, $absUrl, $headers, $params, $hasFile)
    {
        // Handle Laravel Cashier creating/getting a customer
        if ($method == "get" && strpos($absUrl, "https://api.stripe.com/v1/customers/") === 0) {
            $this->rBody = $this->getCustomer(str_replace("https://api.stripe.com/v1/customers/", "", $absUrl));
            return [$this->rBody, $this->rcode, $this->rheaders];
        }

        if ($method == "post" && $absUrl == "https://api.stripe.com/v1/customers") {
            $this->rBody = $this->getCustomer("cus_".Str::random(14));
            return [$this->rBody, $this->rcode, $this->rheaders];
        }

        // Handle creating a Stripe Checkout session
        if ($method == "post" && $absUrl == "https://api.stripe.com/v1/checkout/sessions") {
            $this->rBody = $this->getSession($this->url);
            return [$this->rBody, $this->rcode, $this->rheaders];
        }        

        return [$this->rbody, $this->rcode, $this->rheaders];
    }

    protected function getCustomer($id) {
        return <<<JSON
{
  "id": "$id",
  "object": "customer",
  "address": null,
  "balance": 0,
  "created": 1626897363,
  "currency": "usd",
  "default_source": null,
  "delinquent": false,
  "description": null,
  "discount": null,
  "email": null,
  "invoice_prefix": "61F72E0",
  "invoice_settings": {
    "custom_fields": null,
    "default_payment_method": null,
    "footer": null
  },
  "livemode": false,
  "metadata": {},
  "name": null,
  "next_invoice_sequence": 1,
  "phone": null,
  "preferred_locales": [],
  "shipping": null,
  "tax_exempt": "none"
}
JSON;

    }

    protected function getSession($url)
    {
        return <<<JSON
{
  "id": "cs_test_V9Gq09dEmaJ2p3tydHonjbPSr3eq3mfOn52UBVbppDLVEFQfOji1uZok",
  "object": "checkout.session",
  "allow_promotion_codes": null,
  "amount_subtotal": null,
  "amount_total": null,
  "automatic_tax": {
    "enabled": false,
    "status": null
  },
  "billing_address_collection": null,
  "cancel_url": "https://example.com/cancel",
  "client_reference_id": null,
  "currency": null,
  "customer": null,
  "customer_details": null,
  "customer_email": null,
  "livemode": false,
  "locale": null,
  "metadata": {},
  "mode": "subscription",
  "payment_intent": "pi_1DoyrW2eZvKYlo2CHqEodB86",
  "payment_method_options": {},
  "payment_method_types": [
    "card"
  ],
  "payment_status": "unpaid",
  "setup_intent": null,
  "shipping": null,
  "shipping_address_collection": null,
  "submit_type": null,
  "subscription": null,
  "success_url": "https://example.com/success",
  "total_details": null,
  "url": "$url"
}
JSON;

    }
}

Some Details

There's a few things to note.

First, we tell Stripe to use our mock HTTP client (luckily it's set globally):

$mockClient = new MockClient;
ApiRequestor::setHttpClient($mockClient);

The MockClient test implements the request() method as required by the interface. Here we just "sniff" out the various requests sent to the API and send some fake responses.

To figure out the calls made for the test, I just var_dump()'ed the method parameters:

public function request($method, $absUrl, $headers, $params, $hasFile)
{
    // Figure out what API calls Laravel Cashier is making
    // for a given test
    dd($method, $absUrl, $headers, $params, $hasFile);
}

Then I made mock/fake/fixture/whatever responses for those calls based on the API reference (literally just copying/pasting the JSON).

Under the hood, Stripe takes the object parameter in the JSON response from their API and maps it to a PHP class. So returned JSON "object": "checkout.session" becomes an instance of Stripe\Checkout\Session.

The redirect my specific test includes is the URL returned as a parameter when creating a Checkout Session. This sends the user off to a hosted Stripe Checkout page.

The Offical Way

There's a more Technically Correct™ way to do this, although I discovered this after I already setup the above and didn't change it.

Stripe has a Golang based project named Stripe Mock that runs a test version of the Stripe API. This project aims to return "approximately correct API response for any endpoint".

To use this, you can download a binary (or run it in Docker), and then have the Stripe client object use that for it's API endpoint.

Here's an article on using Stripe Mock, and even how to set it up in GitHub Actions.

Locally I did something like this to play with it:

# In one terminal window
docker run --rm -it -p 12111-12112:12111-12112 \
    stripemock/stripe-mock:latest

# In another terminal window
curl -i -X POST http://localhost:12111/v1/customers \
    -H "Authorization: Bearer sk_test_123" \
    -d 'name=Chris Fidao' -d 'email=foo@example.com'

HTTP/1.1 200 OK
Request-Id: req_123
Stripe-Mock-Version: 0.109.0
Date: Thu, 22 Jul 2021 11:44:24 GMT
Content-Length: 589
Content-Type: text/plain; charset=utf-8

{
  "address": null,
  "balance": 0,
  "created": 1234567890,
  "currency": "usd",
  "default_source": null,
  "delinquent": false,
  "description": null,
  "discount": null,
  "email": "foo@example.com",
  "id": "cus_H42rveoStCxpP4E",
  "invoice_prefix": "40BEC7C",
  "invoice_settings": {
    "custom_fields": null,
    "default_payment_method": null,
    "footer": null
  },
  "livemode": false,
  "metadata": {},
  "name": "Chris Fidao",
  "next_invoice_sequence": 1,
  "object": "customer",
  "phone": null,
  "preferred_locales": [],
  "shipping": null,
  "tax_exempt": "none"
}

Pretty neat! I would use this method for testing if I had a lot of tests that hit the Stripe API.

Children, Time, Advice

On January 1, 2018, my son was just 3 months and 6 days old.

One thing you learn when you have a kid is that a LOT of the business and life advice we hear is not geared towards parents. There were so many times when I muttered "yeah, try that with a baby" to a podcast.

Finding good advice through the lense of parenthood is rare. This is probably because many of the people we listen to don't have kids of their own, or divorce that element from their advice - in either the case, advice that takes parenthood into account is rare.

There's a larger point here that I'm slowly internalizing - everyone giving business and life advice has a context through which they are giving it. This means that what works for some may not work for you. This idea (although not related to children) is covered a bit in this excellent Art of Product episode.

In any case, this whole section is an excuse to get to this sentence: young kids are fucking hard to have around, and finding time to work on my business outside of work hours (I have a job!) was a major source of stress this year. Working from home exacerbates this, but I won't give up being home during my kid's most important years.

Andrey Butov does a good job explaining the struggle of work and kids in episode 10 of our (newish) podcast.

What Happened in 2018

Here's what I can remember doing in 2018!

Released Scaling Laravel

Most of the work for Scaling Laravel was done in 2017, and then delayed a few months after my son was born. I released the course in early 2018.

This was my big "thing" for the year, and the main driver of revenue. The next largest driver was Shipping Docker, which was released in 2017.

The sales cliff for courses is steep - sales drop off almost immediately and remain low very soon after an initial launch, so the majority of revenue for a course is made up-front. Revenue from Shipping Docker was therefore low relative to Scaling Laravel.

Docker in Development

Docker is a technology that is still moving and changing rather quickly. My course Shipping Docker is about 2 years old as I write this.

This year I refreshed a free part of the course: Docker in Development. This was one of the first things I made for the original course - it was in need of a refresh since I learned and refined quite a bit afterwards!

Vessel

If you're interested in seeing my current dev workflow using Docker, check out both Part I and Part II of the refreshed Docker in Development course linked above.

For Laravel specifically, I created Vessel, which essentially is this same development workflow, but specifically for Laravel (and improved upon).

Vessel was released in late 2017, and I've continued to work on it in 2018. The goals for this project were:

1. Simplicity
2. Great documentation

I spent more time on the docs than the initial project. Good documentation is a lot of work, and I wanted to be proud of the docs I made. I really enjoyed the process of crafting them.

A lesson here: If you want your project to gain traction, the bar is fairly high. One way your project can stand out is through great documentation and a developer-friendly API.

From a business & marketing point of view, this can really help your project get traction and, if it's open source, attract great pull requests. Finding a project with light or bad documentation is super common. However, great documentation and ease-of-use is quickly becoming table stakes.

Tweet Tips

Many of you reading this probably has seen Steve Schoger's tweets promoting what eventually became Refactoring UI. Growing a Twitter audience is a great way to generate interest in your projects, and Steve really succeeded there.

Other than being "famous", one way of having an interesting twitter account is by giving as much useful information as possible. Steve Schoger, Adam Wathan, and Wes Bos have really perfected this.

If that's something you're interested in (it's certainly not the be-all-end-all of business), combining a tweet with an image and maybe a link off to an article is a great way to help grow your audience. I make heavy use of Carbon to help make my own tweet tips.

I've done some of this myself in 2018 - Here's a few examples:

1. Collection of Nginx tips
2. MySQL INFORMATION SCHEMA tip
3. Linux human-readable disk usage tip
4. MySQL mysqldump tip

These are a great way to generate interest in courses - I have a bunch of MySQL tweet tips lately to help with my upcoming MySQL Backups course.

MySQL Backups

My next course is going to be all about MySQL Backups. This course is in-process right now and is about 75% complete as I write this.

My favorite part about creating this particular course is that I found someone to help me edit my videos. This saves me an incredible amount of time and stress.

Hiring people to help with time-intensive tasks is going to be a theme for me in 2019.

This course is NOT going to be one of better revenue-generating courses. I can tell by how many people have NOT signed up to the email list relative to past courses.

That's disapointing, but totally OK. I like the content of this course a lot - this is a woefully neglected topic. Stack Overflow is often leading people astray.

Thanks to having help edit my videos, and already having infrastructure around selling and viewing the course, I'm not squandering too much opportunity cost in creating a course whose sales won't match my previous "larger" courses.

Backops.app

I've started building an application to help people automate their MySQL backups (and other stuff, eventually).

Why make an application around a course that's not my most popular? My thinking is that course popularity isn't necessarily an indicitation in who will pay a monthly subscription for a service. More importantly, I feel like I have some competitive advantages:

1. I have an audience in the devops/server space
2. I can integrate tightly with providers such as Forge (and others!) to make the experience really great
3. I know there is a market for it, and its rough size, thanks to apps like Ottomatik - which was sold to someone non-technical and outside of the tech communities I (we) hang out in over the last year or so
4. There are shortcomings in similar apps and open source tools which I feel like I can handle better

So, we'll see where that leads! The Long Slow Saas Ramp of Death is a real thing, so if I spend most of 2019 on this, I know my business revenue will drop like a rock. That might just be the risk I take this year. We'll see.

Has Opinions Podcast

My friend Dan and I started a "for-fun" podcast who's name, in typical techie fashion, is based purely on a domain I had lying around - hasopinions.wtf.

I love podcasts where the hosts shoot-the-shit and sprinkle in some interesting things about their business. That's the podcast I hope to make. It'll take time and practice to really find my voice and a focus for the pod.

I don't have grand plans for this in terms of business. I want to keep it fun. I don't have the time nor the inclination to make another thing feel stressful.

Not every aspect of your life needs to be monetized!

Youtube

I started moving videos from Servers for Hackers onto a Youtube channel to see if they'd gain any traction.

Based on subscriptions and comments, I think this was a good move overall! I'll likely continue to add Servers for Hackers content onto Youtube.

This is another thing I'll be looking to hire out, as it's a bit of a tedious process. However it's one that can be systematized through documentation or programming.

Business

Over the last 3-4 years, I've released about one large course a year.

Revenue has grown the most within the last 3 years. However, in 2018, it stayed steady with 2017 (it was just a tad lower than 2017).

What's that mean? I'm not sure! In one sense, it's a win, since it's so hard to find time to get any work done between the day job and having a kid. On another hand, there are people under similar constraints but making many multiples of anything I've ever made. That doesn't really mean anything, but as Rob Walling pointed out, entrepreneurs have a knack for turning any positive thing into a negative! I must be a great entrepreneur...

Business Plans

I do, of course, want to grow my revenue!

Continuing to grow my audience feels like the right way to grow that revenue number. This means producing more content - which is good, because it's something I like to do!

Finding time to put out a lot of free stuff, and make at least one more course in 2019 (after MySQL Backups) is a large part of my 2019 plans.

The backops.app app may interfere with that - we'll see!

How much did I make?

My side business has made more than my salary in 2017 and 2018.

Why don't I quit?

I would make way less money! Although I'm aware that this doesn't account for opportunity cost and the potential to make more if I capitalize on the "free time" found by not being employed.

In the last 3 years, my business has made "significant" money relative to my salary (the last 2 years made more than my salary, and the year before that, I made about 50% of my salary).

Because of this, my family is out of debt from student loans and vehicles, I was able to put 20% down on my mortgage (fuck off, PMI), I'm able to save for retirement, have an emergency fund, and basically follow the good advice from /r/personalfinance.

In theory I'm in a good position to go out on my own. However, I'm planning on keeping my job for the following reasons:

1. I have a lot of flexibility right now, and any future job will likely have less. I'm very lucky to have found/been found by UserScape.
2. I know a paycheck is coming, but also am finding time to make additional money. This alleviates a lot of "Founder Stress".
3. Health care in the US is a disaster and is super unfriendly to small business. Striking out on your own is a LOT scarier thanks to the rising cost of health care and how bad the quality of the open market appears to be in terms of expense and coverage.
4. "Real" work gives me an avenue to learn about topics I would never come across otherwise.

2019 Plans

Here's what I plan on working on in 2019:

• Finishing the MySQL Backups course
• Creating a free Nginx course
• Creating a free server course that's a bit more start-to-finish-hosting-of-an-application
• Creating a paid Ansible course (probably? AWS is a strong contender, but would be a HUGE course)
• Working on backops.app

Competing with all of this is time spent in the day job, raising a kid, and potentially working on a SaaS app. - which is way more than just coding.

Numbers to Track

I haven't tracked any particular metrics over the last few years (I wish I did!). Here's a few things that I'm going to start to track, and hopefully back-fill where I can:

1. Newsletter Subscribers
2. Twitter Followers
3. Youtube Subscribers
4. Podcast Downloads (although the pod is really just for fun)

You can see they are geared towards audience growth, and not revenue. Revenue will hopefully follow, but I refuse to set a revenue goal.

Having a business of my own started by finding out I enjoyed helping people by digging through code and learning about servers. I want that focus to remain - in other words, I want to enjoy my work.

Things I Like

Here's some podcasts and resources I've really enjoyed in 2018.

• Justin Jackson has been putting out great articles and podcasts about his journey making his SaaS. The journey is hard and he's not afraid to say so.
• Art of Product podcast - Ben and Derrick's journey to build their apps has been super interesting to follow. They're great guys making interesting products.
• Tropical MBA Podcast - They've recovered from their momentary period of "rich founder problems" (which can be forgiven since they sold their business), and are now in full swing with really interesting episodes around being an entrepreneur.
• Bootstrapped FM - Yes, it's my boss (and Andrey!). But it's the perfect "shoot the shit" podcast. Make more episodes, guys.

Here's a video explaining the process in Laravel 5.3:

1. app/Http/Kernel.php
2. app/Console/Kernel.php

Within each, I over-rode the __construct method from the parent class, in order to edit the $bootstrapper array - the array of class names telling Laravel which to load and bootstrap for Http vs Console requests.

Each of these loaded in the ConfigureLogging bootstrap class. This class (Illuminate\Foundation\Bootstrap\ConfigureLogging) hard-coded the laravel log file name.

App\Http\Kernel.php

namespace App\Http;

use Illuminate\Routing\Router;
use Illuminate\Contracts\Foundation\Application;
use Illuminate\Foundation\Http\Kernel as HttpKernel;

class Kernel extends HttpKernel {
    // boiler plate removed

    // over-ride parent __construct method
    public function __construct(Application $app, Router $router)
    {
        parent::__construct($app, $router);

        // Replace default logger with HelpSpot Logger
        $loggingKey = array_search('Illuminate\Foundation\Bootstrap\ConfigureLogging', $this->bootstrappers);
        $this->bootstrappers[$loggingKey] = 'App\ConfigureLogging';
    }
}

App\Console\Kernel.php

<?php

namespace App\Console;

use Illuminate\Contracts\Events\Dispatcher;
use Illuminate\Console\Scheduling\Schedule;
use Illuminate\Contracts\Foundation\Application;
use Illuminate\Foundation\Console\Kernel as ConsoleKernel;

class Kernel extends ConsoleKernel {
     // boiler plate removed

    // over-ride parent __construct method
    public function __construct(Application $app, Dispatcher $events)
    {
        parent::__construct($app, $events);

        // Replace default logger with HelpSpot Logger
        $loggingKey = array_search('Illuminate\Foundation\Bootstrap\ConfigureLogging', $this->bootstrappers);
        $this->bootstrappers[$loggingKey] = 'App\ConfigureLogging';
    }
}

App\ConfigureLogging

Then we can make our own class that extends the base ConfigureLogging class and tweaks it as needed:

<?php

namespace App;

use Illuminate\Log\Writer;
use Illuminate\Contracts\Foundation\Application;
use Illuminate\Foundation\Bootstrap\ConfigureLogging as BaseLoggingBootstrapper;

class ConfigureLogging extends BaseLoggingBootstrapper
{
    /**
     * Configure the Monolog handlers for the application.
     *
     * @param  \Illuminate\Contracts\Foundation\Application  $app
     * @param  \Illuminate\Log\Writer  $log
     * @return void
     */
    protected function configureSingleHandler(Application $app, Writer $log)
    {
        $log->useFiles(
            $app->storagePath().'/logs/my-app.log',
            $app->make('config')->get('app.log_level', 'debug')
        );
    }

    /**
     * Configure the Monolog handlers for the application.
     *
     * @param  \Illuminate\Contracts\Foundation\Application  $app
     * @param  \Illuminate\Log\Writer  $log
     * @return void
     */
    protected function configureDailyHandler(Application $app, Writer $log)
    {
        $config = $app->make('config');

        $maxFiles = $config->get('app.log_max_files');

        $log->useDailyFiles(
            $app->storagePath().'/logs/my-app.log', is_null($maxFiles) ? 5 : $maxFiles,
            $config->get('app.log_level', 'debug')
        );
    }

    /**
     * Configure the Monolog handlers for the application.
     *
     * @param  \Illuminate\Contracts\Foundation\Application  $app
     * @param  \Illuminate\Log\Writer  $log
     * @return void
     */
    protected function configureSyslogHandler(Application $app, Writer $log)
    {
        $log->useSyslog(
            'my-app',
            $app->make('config')->get('app.log_level', 'debug')
        );
    }
}

And there we have it - our log file will be named my-app.log instead of laravel.log.

It's up to you to learn your tools so you can adapt them to your needs. It's not up to the tool to adapt to your needs.

If available tools make it too hard to meet your needs, your alternative option is to make your own tool.

Frustration in available tooling is my theory on how things like Rails and Laravel get created.

That being said, I think the more interesting question is why some of us seem to always adapt available tools while others seem to always create their own tools.

Personalities

I know some of those who have made very popular tools. I've always wished to be one of these people. However I've never reached the tipping point where I went to make something new. I've always been able to adapt current tools to suit my needs.

I've noticed that those who make their own tool are different here - they get frustrated at the process of learning someone else's tool.

Over time, I've come to divide this distinction of personalities into adapters and makers. It should be noted that I imagine these to be on a spectrum, not binary positions.

Adapters

Adapters learn their tools and adapt their usage so they can complete their objectives.

Adapters tend to work through the frustrations that come with learning a new tools, either because they enjoy the learning process or sheer determination to not be defeated (apologies if "sheer determination" is a tad hyperbolic).

One thing of note, however: It is common for the ideal solution to suffer compromise due to limitations in available tools. You know this is happening when you need to explain why your code won't be a stakeholder's (your?) exact vision. This is a trade-off of not spending the time on making something bespoke.

Makers

Makers are less willing to suffer the frustrations that come with learning a tool. They have a vision of how something should work. While learning a tool, they see how compromised that vision becomes. Rather than pushing through, they make their own tool.

They may spend countless hours on this. Many of us would see this as a waste of time; Indeed it ultimately might be.

However, those with extraordinary (read that as "above average", not "god-like") vision may make something great.

This, perversely, often involves some deep learning. While makers may not have the perserverence for deep learning of someone else's tool (we call this "human nature"), they are more willing to shoulder that burden in the building of their own thing.

While not necessarily directly related to building a tool, this seems to be a trait shared amongst entrepreneurs.

Glory

It's easy to glorify the makers. The very successful ones are few and enjoy the fruits (and labors) of minor to major internet celebrity.

Conversely, successful adapters are more likely to have a better job relative to their less-successful peers in adaptation. This might mean doing better within a company, or doing better in finding new employment. I imagine this is due to a network effect, where their success positively effects the success of what they work on, and thus they themselves become respected within their social/professional circles.

Fear

However, I fear that conflating the maker personalities with the celebrity version of success is damaging. I can just imagine the Hacker News response, each commenter desperately trying to prove how they are victoriously fitting the Maker Mold.

Not that I give my opinions that much weight. Just as the HN crowd try to see something of themselves in others' success, I often find myself imagining my ideas as read and respected by "the masses". Reality is appropriately harsher than our rosy imaginations.

That is not my point - both of these personality traits Get Shit Done™. One, however, may be more profitable, if you win that particular lottery.

Grey

Cynicism aside, I don't see any one extreme of the spectrum as being more intelligent than the other. I view makers as being more stubborn by nature. I view adapters as being more easy-going by nature.

These are not a measure of intelligence, or even of perserverence. Instead it's a shift in focus in where you feel most comfortable putting effort.

And don't forget it's a spectrum.

An HTTP client, such as your browser, or perhaps jQuery's ajax method, can set an Accept header as part of an HTTP request.

Accept Header

This header is meant to tell the server what content types it is willing to accept. From the HTTP 1.1 spec, section 14.1:

The Accept request-header field can be used to specify certain media types which are acceptable for the response.

Such a header might look something like this:

Accept: application/json

In a typical request from Chrome, we see something more like this:

Accept:text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8

As you can see, the Accept header can get complex.

The above example lists a few media types the client (Chrome) is willing to accept, and even gives them a "quality" factor (the q rating, value 0-1). This is essentially telling the server the ordered preference of content types it wants back.

I won't get any more into that, but your can read more about it in the HTTP spec.

The Server

It's up to the server to follow the rules of HTTP. When a request comes to our application, it's pretty easy to ignore these rules, as our frameworks generally let us return whatever we want.

This is the "negotiation" part. The client says what content types it's willing to accept, and its preference. The server then can decide what it's willing/able to send back. Or ignore it, if it's a rebel. Not much of a negotiation, if you ask me.

For example, if a request comes into your Laravel app with an Accept header requesting JSON, you can totally ignore it without ever realizing the HTTP client wanted something else:

Route::get('/foo', function()
{
    // Accept header? Whatever, bruh
    return view('foo.bar');
});

Checking the Accept header:

If you want to check for that header, you can do some manual stuff:

Route::get('/foo', function()
{
    $accept = request()->header('accept'); // application/json

    if( $accept === 'application/json' )
    {
        return ['foo' => 'bar']; // Returns JSON, thanks to Laravel Magic™
    }

    return view('foo.bar');
});

It gets easier, however:

Laravel provides a nice, easy way to check if a request "wants json":

Route::get('/foo', function()
{

    // Look at this nice, friendly global helper function
    // Hello, global function! We sincerely love you with all our <3
    if( request()->wantsJson() )
    {
        return ['foo' => 'bar'];
    }

    return view('foo.bar');
});

If you check out the function linked above, you can see how Laravel is using the underlying Symfony HTTP classes, which handle the dirty work of knowing that HTTP requests might send down multiple accept content types:

public function wantsJson()
{
    $acceptable = $this->getAcceptableContentTypes();
    return isset($acceptable[0]) && Str::contains($acceptable[0], ['/json', '+json']);
}

Note that this is grabbing the first of the list of acceptable content types, ordered by preference, sent in the HTTP request and testing if it is a JSON content type. It's not simply saying "Yeah, I guess JSON was one of the content types you wanted".

If you want to see if the request will accept JSON, regardless of it's set preference, use the acceptsJson() method.

Other Content Types

I suggest taking a look at some of the shortcuts the Laravel Request class has - you can check for other content types with the accepts() method, for example, or use the prefers() method to see which is the most prefered content type.

Middleware, or something

This is ripe for some sort of middleware or other functionality which automatically can decide to return JSON or HTML (or hey, even XML!). If you use RESTful routing controllers, that might be a nice addition, similar to how Rails lets you set a return type.

Software

This article is primarily about the hardware I use, but you might as well know about the software too.

For editing screencasts, I started out recording using QuickTime (the one that comes with your Mac). I had a copy of Adobe Premiere from previous employment, and so tried that to edit the videos. That shit is complicated. You can figure it out, but it's harder to do simpler things than it should be. I never aspired to having "proficiency in Adobe Premiere" on any resume, so I looked for alternatives.

That lead me to Screenflow, which I can't recommend enough. It strikes a really good balance between control (aka complexity) and sensible defaults (aka ease of use). For example, exporting a video in Screenflow is super easy for sending right to Youtube or Vimeo. If you export an edited video to disk, you get a lot of options, but not so many that you feel like you need to be an expert to wade through them.

Editing sound and video is intuitive. Adding affects, adjusting volume, hiding the mouse, showing keys hit, adding annotations and similar popular uses cases are all fairly easy to add.

I haven't used it, but I'd also check out Camtasia if you're looking for alternatives.

Anyway, onward to audio stuff!

First Feeble Audio Steps

About a year ago, I made one or two videos using the ubiquitous white earbuds of Apple fame. As you might suspect, the quality was horrendous.

Cost: 1.5 Apple Pricing Units. Apple seems to have a baseline of $19.99 for even the cheapest item, so I've started calling $20 an "Apple Pricing Unit". It's a thing. I swear.

Next Attempt

Searching for better quality, I moved onto using the popular Blue Yeti (eventually also adding a pop filter). The quality is much improved, but varied a lot depending on how close I was to the microphone. Since it comes with a desk stand, this created a few issues:

• To get close enough, I had to lean forward uncomfortably while speaking into it, which was super awkward when also typing/mousing around while recording. Picture Quasimodo using a computer for the first time.
• As the Yeti sat on my desk, ALL desk vibrations were recorded as terrible, bassy thuds. This includes both the times I accidentally grazed the desk with my fingers, and any keyboard typing. Worse, it could even be heard after putting the Yeti on thick stacks of paper to help absorb vibrations.
• This is a condenser mic, and therefore picks up just about all stray noises. If you ever wondered how attractive your breathing and wet, sloppy mouthing noises were, grab yourself a condenser mic. Humans are seriously a pile of disgusting wet meat.

The fix for most of the vibration issues would be a desk-mount (or floor stand) and shockmount. However, the Yeti is really heavy, making it a poor choice to put on most stands, which have a hard enough time keeping normal microphones in position without drooping.

You can see the dismal reviews for Blue's Radius shockmount on Amazon, although that may mostly be fixed with the Radius II mount.

Cost: $130

Research

Researching better audio equipment should inform you of the trade-offs to decide between, rather than as a way to find "the one true mic". You can buy a better microphone, but in reality, you might just be buying a "different" microphone.

Although with a reasonable price increase, you likely are getting both different and better.

One of the larger trade-offs to make between microphone types is in deciding between condenser and dynamic microphones.

Condenser microphones, like the Yeti, pick up a LOT of background noise. This is actually "good", in that they have a good range/frequency of audio they can record. From what I read, this is great for things like music. In a studio. With lots of sound proofing.

This is not as good when you have a dog slopping from a water bowl (gross mouthing noises from across the room!!), or a community pool outside your window (don't these people have f&%^#@!$ jobs!?).

Dynamic microphones do a great job at filtering out extraneous sounds. In fact, they do such a good job, that you may find yourself needing an amp so people can actually hear you.

In any case, dynamic microphones are great for podcasting / screencasting. These need to pick up voice, but not necessarily require the frequency range desired for instruments and singing.

Plenty of people recommend dynamics for singing. This depends on the microphone brand, quality and your needs. Everything is a trade off between underlying technology and the quality/focus of a specific product.

What I needed was something to have great quality for voice, but not pick up every little bit of background noise.

Where I live, at any given hour, there's usually a gaggle of attractive, pool-side Abercrombie models just galavanting around like they don't have a care in the world.

Reducing background noise is really important to me.

Current Setup

I landed on the Heil PR40. This is based on personal recommendations and reviews, which emphasized the greatness of this mic for podcasts and voiceover - just what I do!

Of course, selecting the microphone is just to start. Then you realize you may need a lot more supporting equipment. Here's everything I got:

• Heil PR40 ~ $330
• Heil Overhead Broadcast Boom - Because I refuse to awkwardly sit forward and try to type while recording. ~ $130
• Heil shockmount because vibrations are a thing and they suck on any microphone. ~ $105
• Shure X2U Adapter - Most dynamics aren't USB, so this converts your XLR to a USB signal while providing extra "Phantom +48v" power, which I believe is for condenser mics instead of dynamics. This comes with a USB cable but not an XLR cable. It's nice to have to test various mics with, in case you own or test one that doesn't support USB. ~ $99
• Cloudlifter CL-1 - I picked this up to help boost sound output, as dynamic mics can come through a bit soft. I haven't tested the mic with vs without the Cloudlifter yet. ~ $150
• Mogami Studio Microphone Cable - I'm wary of cable prices, but went with good/expensive cables here since I was in the mindset of "investment for my business". Probably a bit of self-justification, but so far so good on quality! I guess. I don't really know. ~ $40 - $55
• Mogami Studio Microphone Cable - Listed here a second time because you need at least two XLR cables, one between the microphone and Cloudlifter, and another between the Cloudlifter to the Shure XLR to USB converter. Don't forget that like I did :D ~ $40 - $55

Here you can see just about everything. I had the Cloudlifter just hanging off the back until I got a longer cord. The Shure X2U is the black gizmo on the bottom right, under the monitor.

The Shure X2U is really nice. It lets you adjust volume and gain. To help you know what levels are good, it has a green light that blinks when sound is too low, is steady when it's just right, and turns orangey/yellowy when you're too loud.

What's fun is to learn that "S" sounds are something like 4 times as loud as any other sound we make. You can see that because anytime you make an S sound, the light on the X2U will turn orange. There are techniques to combat all this madness.

After rounding a bit, and because I ended up with 3 XLR cables (guessed wrong on the lengths I'd want), I spent ~ $940 on all of this. You can spend less or much much more depending on your heart's desire.

I love this microphone though. Having it over my head is very nice - it's out of the way. I can move it with me, so I can sit comfortably while casting. The shockmount and pop filter really helps with sound quality and in reducing vibration-based noise.

I still need to use my trackpad instead of my mouse when recording, as the scraping noise of my mouse comes through (I don't use a mouse pad, because I'm a rebel). Keyboard noises comes through as well, however just the tapping of the keys rather than the bassy vibrations caused by physically hitting the keyboard.

So that's it! One large blog post to brag about how I am writing off $940 dollars as a business expense this year.

There's a few ways to get general output from a CLI command using Console:

// Run the command (Laravelish)
public function fire()
{
    echo "This is sent to stdout";  // Just text

    $this->info('Some info'); // Regular Text
    $this->error('An Error'); // Red Text
}

// Run the command (Symonfyish)
public function execute(InputInterface $input, OutputInterface $output)
{
    echo "This is sent to stdout";  // Just text

    $output->writeln("<info>$string</info>");   // Regular Text
    $output->writeln("<error>$string</error>"); // Red Text
}

Console commands in Laravel use Symfony's Console component under the hood. While I'll write this (mostly) in context of Laravel, this is definitely applicable to Symfony users and those not using Laravel (collectively, "the haters") as well.

All of this, even the "error" output, writes to "Stdout". This isn't necessarily good. In fact, the default behavior can easily make for some surprises to other developers calling these commands over a CLI.

Convention

The following are well established *nix conventions to follow for any CLI tool.

• Only write pertinent, needed information to Stdout
• Write info messages (non-pertinent) to Stderr, even if it's not an error message
• Only attempt to detect if the command was successful based on the result the command returns (exit status code). That will be with 0 (success) or 1 (failure).
  • Do not attempt to guess if a command failed simply because output was sent to Stderr
• Usually if everything works, then nothing is output. Simply returning a 0 exit code is generally enough.

Writing important information to Stdout lets administrators send important data to log files. Writing non-important to Stderr lets administrators ignore it or send it to a log file specifically for errors or other information.

Perhaps more importantly is that Stdout output might get piped to another process to handle (think about anytime you do cat /some/file | grep 'search-term'). You don't want non-important output sent to Stdout in those cases. Sending those to Stderr makes the most sense then.

Lastly, because of these conventions, it's important that your commands return a 0 or 1 if they are successful or if the exit with an error. This is The Way™ that should be used to detect if there's truly an error or if the command operated successfully.

In Practice

Here's how I setup Laravel commands:

<?php namespace Foo\Bar;

use Illuminate\Console\Command;
use Symfony\Component\Console\Output\ConsoleOutputInterface;

class MyCommand extends Command {

    # Some boiler plate omitted

    public function run()
    {
        // Default $stdErr variable to output
        $stdErr = $this->getOutput();

        if( $this->getOutput() instanceof ConsoleOutputInterface )
        {
            // If it's available, get stdErr output
            $stdErr = $this->getOutput()->getErrorOutput();
        }

        try {
            // Some operations
             
            // Non-critical information message
            // Since we have the Symfony output object, use writeln function
            $stdErr->writeln('<info>Status: Working...</info>')
        } catch( \Exception $e )
        {
            // Since we have the Symfony output object, use writeln function
            $stdErr->writeln('<error>'.$e->getMessage().'</error>');
            return 1;
        }

        // Important output
        $this->info('Your new API key is: aaabbbcccddd');
        return 0;
    }
}

And the same in a Symfony command:

<?php namespace Foo\Bar;

use Symfony\Component\Console\Command\Command
use Symfony\Component\Console\Input\InputArgument;
use Symfony\Component\Console\Input\InputInterface;
use Symfony\Component\Console\Output\ConsoleOutputInterface;

class MyCommand extends Command {

    # Some boiler plate omitted

    public function execute(InputInterface $input, OutputInterface $output)
    {
        // Default $stdErr variable to output
        $stdErr = $output;

        if( $output instanceof ConsoleOutputInterface )
        {
            // If it's available, get stdErr output
            $stdErr = $output->getErrorOutput();
        }

        try {
            // Some operations
             
            // Non-critical information message
            $stdErr->writeln('<info>Status: Working...</info>')
        } catch( \Exception $e )
        {
            $stdErr->writeln('<error>'.$e->getMessage().'</error>');
            return 1;
        }

        // Important output
        $output->writeln('<info>Your new API key is: aaabbbcccddd</info>');
        return 0;
    }
}

These classes mirror each other. The Laravel version uses some of its syntactic sugar.

Let's go over what's going on.

First, I assign a variable $stdErr. This gets assigned a fallback of the Output object. I'm going to use this variable later for error output regardless of whether it's used for Stdout (the default) or Stderr.

If the Output object happens to be an instance of ConsoleOutputInterface, I'll know it has the getErrorOutput method available. Not all Output implementations do, so this check is important. Stderr can then be assigned the Error Output object, which will write to Stderr. I can then easily differentiate output between Stderr and Stdout.

The rest of this is implementation of the above conventions. I write non-essential information to Stderr, but use the "info" formatters, as they don't need the red error styling.

Actual errors are also output to Stderr, but with the red output styling.

Important information is output to Stdout, again with the "info" styling.

Note that I return 0 or 1 (0 for success). The return value is taken by the Symfony Console component and returned as the exit code of the command. If you define nothing, then 0 is returned, even if you have output an error!

If this command didn't need the resulting output (for example, the new API key), I would return nothing. If I had a "success" message, I would actually return that in Stderr, but with the "info" formatting.

• Video of Talk
• Slides for Talk

I found Hexagonal Architecture to be a good expression of how I think about code. In fact, when I wrote Implementing Laravel, I was actually espousing some ideals of Hexagonal Architecture without knowing it.

Hexagonal Architecture defines conceptual layers of code responsibility, and then points out ways to decouple code between those layers. It's helped clarify when, how and why we use interfaces (among other ideas).

Hexagonal Architecture is NOT a new way to think about programming within a framework. Instead, if's a way of describing "best practices" - practices that are both old and new. I use quotes because that's a bit of a loaded phrase. Best practices for me might not be best practices for you - it depends on what technical circles we engage in.

However, Hexagonal Architecture espouses common themes we'll always come across: decoupling of code form our framework, letting our application express itself, using a framework as a means to accomplish tasks in our application, instead of being our application itself.

Beginnings

The name for Hexagonal Architecture is brought to us (so far as I can tell) by Alistair Cockburn. He outlines the architecture very well on his website.

It's intent:

Allow an application to equally be driven by users, programs, automated test or batch scripts, and to be developed and tested in isolation from its eventual run-time devices and databases.

Why a Hexagon

The article takes on a shape of a hexagon. The number of sides is actually arbitrary. The point is that is has many sides. Each side represents a "port" into or out of our application.

A port can be thought of as a vector for accepting requests (or data) into an application. For example, an HTTP port (browser requests, API) can make requests on our application. Similarly, a queue worker or other messaging protocol (perhaps AMQP) can also make a request on our application. These are different ports into our application, but are also part of the "request port". Other ports could include those for data access, such as a database port.

Architecture

Why do we even talk about Architecture?

We talk about architecture because we want our applications to contain two attributes:

1. High Maintainability
2. Low Technical Debt

These are, in fact, the same thing. To word this succinctly: We want our applications to be easy to work with. We want to make future changes easy.

Maintainability

Maintainability is the absence (reduction) of technical debt. A maintainable application is one that increases technical debt at the slowest rate we can feasibly achieve.

Maintainability is a long-term concept. Applications in their early form are easy to work with - they haven't yet been formed and molded by the early decisions of the developers working on them. New features and libraries are added quickly and easily.

However, as time goes on, applications can get harder to work on. Adding features might conflict with current functionality. Bugs might hint at systemic issues, which may require large changes in code to fix (and to help clarify overly complex code).

A good architecture early on in a project can help prevent such issues.

What kinds of maintainability are we looking for? What are measures of a highly maintainable application?

1. Changes in one area of an application should affect as few other places as possible
2. Adding features should not require large code-base changes
3. Adding new ways to interact with the application should require as few changes as possible
4. Debugging should require as few work-arounds and "just this once" hacks as possible
5. Testing should be relatively easy

I use the word "should" because there's no perfectly coded application in existence. We want to make our applications easy to work with, but trying for "perfect" becomes a waste of time and an over-exertion of mental energy.

If you think you're spinning your wheels over "the right way" to do something, then just "get it done". Come back to the problem later, or keep your code in its "it just works" state. There's no perfectly coded application in existence.

Technical Debt

Technical debt is the debt we pay for our (bad) decisions, and it's paid back in time and frustration.

Applications all incur a base-line technical debt. We need to work within the confines and limitations of our chosen persistence mechanisms, language, frameworks, tooling, teams and organizations!

Bad architectural decisions made early on compound themselves to into larger and larger issues.

For every bad decision, we end up making work-arounds and hacks. Some of these bad decisions aren't blatantly obvious - we may simply make a class that "does too much" or mixes multiple concerns.

Smaller, yet equally bad decisions during development similarly also create issues. Luckily, these don't necessarily compound themselves like early architectural "mistakes" can. A solid basis reduces technical debt's rate of growth!

So, we want to reduce as many bad decisions as possible, especially early on in a project.

We make a discussion of architecture so that we can focus on increasing maintainability and decreasing technical debt.

How do we make maintainable applications?

We make them easy to change.

How do we make our applications easy to change? We...

We'll go back to this point quite a few times in the following explanations.

Interfaces and Implementations

Let's take some time to discuss something (seemingly) basic in the world of OOP: Interfaces.

Not all languages (notably: Python & Ruby) have explicit Interfaces, however conceptually the same goals can be accomplished in such languages.

You can think of an interface as contract, which defines an application need. If the application need can be or must be fulfilled by multiple implementations, than an interface can be used.

In other words, we use interfaces when we plan on having or needing multiple implementations of an interface.

For example, if our application sends notifications, we might define a notification interface. Then we can implement an SES notifier to use Amazon SES, a Mandrill notifier to use Mandrill and others implementations for other mail systems.

The interface ensures that particular methods are available for our application to use, no matter what implementation is decided upon.

For example, the notifier interface might look like this:

interface Notifier {

    public function notify(Message $message);
}

We know any implementation of this interface must have the notify method. This let's us define the interface as a dependency in other places of our application.

The application doesn't care which implementation it uses. It just cares that the notify method exists for it to use.

class SomeClass {

    public function __construct(Notifier $notifier)
    {
        $this->notifier = $notifier;
    }

    public function doStuff()
    {
        $to = 'some@email.com';
        $body = 'This is a message';
        $message = new Message($to, $body);

        $this->notifier->notify($message);
    }
}

See how our SomeClass class doesn't specify a specific implementation, but rather simply requires a subclass of Notifier. This means we can use our SES, Mandrill or any other implementation.

This highlights an important way interfaces can add maintainability to your application. Interfaces make changing our application notifier easier - we can simply add a new implementation and be done with it.

class SesNotifier implements Notifier {

    public function __construct(SesClient $client)
    {
        $this->client = $client;
    }

    public function notify(Message $message)
    {
        $this->client->send([
            'to' => $message->to,
            'body' => $message->body]);
    }
}

In the above example, we've used an implementation making use of Amazon's Simple Email Service (SES). However, what if we need to switch to Mandrill to send emails, or even switch to Twilio, to send SMS?

As we've seen, we can easily make additional implementations and switch between those implementations as needed.

// Using SES Notifier
$sesNotifier = new SesNotifier(...);
$someClass = new SomeClass($sesNotifier);

// Or we can use MandrillNotifier

$mandrillNotifier = new MandrillNotifier(...);
$someClass = new SomeClass($mandrillNotifier);

// This will work no matter which implementation we use
$someClass->doStuff();

Our frameworks make liberal use of interfaces in a similar fashion. In fact, frameworks are useful because they handle many possible implementations we developers may need - for example, different SQL servers, email systems, cache drivers and other services.

Frameworks use interfaces because they increases the maintainability of the framework - it becomes easier to add or modify features, and easier for us developers to extend the frameworks should we need to.

The use of an interface helps us properly encapsulate change here. We can simply make new implementations as needed!

Going Further

Now, what if we need to add some functionality around individual (or all) implementations? For example, we may need to add logging to our SEE implementation, perhaps to help debug an issue we're having.

The most obvious way, of course, is to add code directly to the implementations.

class SesNotifier implements Notifier {

    public function __construct(SesClient $client, Logger $logger)
    {
        $this->logger = $logger;
        $this->client = $client;
    }

    public function notify(Message $message)
    {
        $this->logger->logMessage($message);
        $this->client->send([...]);
    }
}

Adding the logger directly to the concrete implementation may be OK, but our implementation is now doing two things instead of one - we're mixing concerns. Furthermore, what if we need to add logging to all implementations? We'd end up with very similar code in each implementation, which is hardly DRY. A change in how we add logging means making changes in each implementation. Is there an easier way to add this functionality in a way that's more maintainable? Yes!

Do you recognize some of the SOLID principles being implicitly discussed here?

To clean this up, we can make use of one of my personal favorite design patterns - the Decorator Pattern. This makes clever use of interfaces in order to "wrap" a decorating class around an implementation in order to add in our desired functionality. Let's see an example.

// A class wrapping a Notifier with some Logging behavior
class NotifierLogger implements Notifier {

    public function __construct(Notifier $next, Logger $logger)
    {
        $this->next = $next;
        $this->logger = $logger;
    }

    public function notify(Message $message)
    {
        $this->logger->logMessage($message);
        return $this->next->notify($message);
    }
}

Similar to our other Notifier implementations, the NotifierLogger class also implements the Notifier interface. We can see, however, that it doesn't actually notify anything. Instead, it accepts another Notifier implementation in its constructor, calling it "$next". When run, NotifierLogger will log the Message data and then pass the message onto the real notifier implementation.

We can see that the logger decorator logs the message, and then passes the message off to the notifier to actually do the notifying! If you need to, you can reverse the order of these so the logging is done after the notification is actually sent, so you can also log the results of the sent notification, instead of simply logging the message being sent.

In this way, the NotifierLogger "decorates" the actual notifier implementation with the logging functionality.

The best part is that the consuming class (our SomeClass example above) doesn't care that we pass in a decorated object. The decorator also implements the expected interface, so the requirements set by SomeClass are still fulfilled!

We can chain together multiple decorators also. Perhaps, for example, we want wrap the email notifier with an SMS notifier that sends a text message in addition to sending an email. In that example, we're adding an additional notifier implementation (SMS) on top of an emailing implementation.

We aren't limited to adding additional concrete notifying implementations, as our logging example shows. A few additional examples can include updating the database, or adding in some metric gathering code. The possibilities are endless!

The ability to add additional behaviors, while keeping each class only doing one thing, and still giving us the freedom to add additional implementations, is very powerful - changing our code becomes much easier!

The Decorator Pattern is just one design pattern of many that make excellent use of interfaces to encapsulate change. In fact, almost all of the classic design patterns make use of interfaces.

Furthermore, almost all design patterns exist to make future changes easier. This is not a coincidence. Making a study of design patterns (and when to use them) is a critical step towards making good architectural decisions. I suggest the Head First Design Patterns book for further reading on design patterns.

I'll repeat: Interfaces are a central way of encapsulating change. We can add functionality by creating a new implementation, and we can add behaviors onto existing implementations - all without affecting other areas of our codebase!

Once properly encapsulated, functionality can more easily be changed. Easily changed codebases increase application maintainability (they're easier to change) by reducing technical debt (we've invested time in making changes easier to accomplish).

That was quite a lot on the topic of interfaces. Hopefully that helped clarify some of the important use cases of interfaces, and gave you an taste of how some design patterns make use of them to help make our applications more maintainable.

Ports and Adapters

Now, finally we can begin to discuss the meat of Hexagonal Architecture.

Hexagonal Architecture, a layered architecture, is also called the Ports and Adapters architecture. This is because it has the concept of different ports, which can be adapted for any given layer.

For example our framework will "adapt" a SQL "port" to any number of different SQL servers for our application to use. Similarly, we can create interfaces at key points in our application for other layers to implement. This lets us create multiple adaptations for those interfaces as needs change, and for testing. This is also the basis for decoupling our code between layers.

Creating interfaces for portions of our application that may change is a way to encapsulate change. We can create a new implementations or add more features around an existing implementation as needed with strategic use of interfaces.

Before returning to the concept of Ports and Adapters, let's go over the layers of the Hexagonal Architecture.

Layers: Code

The Hexagonal Architecture can describe an application in multiple layers.

The goal of describing the architecture in layers is to make conceptual divisions across functional areas of an application.

The code within the layers (and it their boundaries) should describe how the layers communicate with each other. Because layers act as ports and adapters for the other layers inside and surrounding them, describing the communication between them is important.

Layers communicate with each other using interfaces (ports) and implementations (adapters).

Each layer has two elements:

1. The Code
2. The Boundary

The code inside of a layer is just what it sounds like - actual code, doing things. Often times this code acts as adapters to ports defined in other layers, but it can also be any code we need (business logic or other services).

Each layer also has a boundary between itself and an outside layer. At the boundary we find our "ports". These ports are interfaces that the layer defines. These interfaces define how outside layers can communicate to the current layer. We'll go into this in more detail.

Domain Layer

The inner-most layer is the Domain Layer. This layer contains your business logic and defines how the layer outside of it can interact with it.

Business logic is central to your application. It can also be described as 'policy' - rules your code must follow.

The domain layer and its business logic define the behavior and constraints of your application. It's what makes your application different from others. It's what gives your application value.

If you have an application with a lot of behavior, your application can have a rich domain layer. If your application is more of a thin layer on top of a database (many are!), this layer might be "thinner".

In addition to business logic (the Core Domain), we often also find supporting domain logic within the Domain Layer, such as Domain Events (events fired at important points in the business logic) and use-cases (definitions of what actions an be taken on our applications).

What goes inside of Domain Layer is the subject of books by themselves - especially if you are interested in Domain Driven Design, which goes into much detail on how to create applications which closely match the real business processes you are codifying.

Some "Core" Domain Logic:

<?php  namespace Hex\Tickets;

class Ticket extends Model {

   public function assignStaffer(Staffer $staffer)
   {
       if( ! $staffer->categories->contains( $this->category ) )
       {
           throw new DomainException("Staffer can't be assigned to ".$this->category);
       }

       $this->staffer()->associate($staffer); // Set Relationship

       return $this;
   }

   public function setCategory(Category $category)
   {
       if( $this->staffer instanceof Staffer &&
           ! $this->staffer->categories->contains( $category ) )
       {
           // Unset staffer if can't be assigned to set category
           $this->staffer = null;
       }

       $this->category()->associate($category); // Set Relationship

       return $this;
   }
}

Above we can see a constraint within the assignStaffer method. If the provided Staffer is not assigned a Category under which this Ticket falls, we throw an exception.

We also see some behavior. If the Category of the Ticket is changed, and the current Staffer is unable to be assigned a Ticket of this Category, we unset the Staffer. We do not throw an exception - instead we allow the opportunity to set a new Staffer when the Category is changed.

These are both examples of business logic being enforced. In one scenario, we set a constraint by throwing an error when something is set incorrectly. In another scenario, we provide behavior - once a Category is changed, users must have the opportunity to re-assign a Staffer who is able to handle that Category of Ticket.

Inside of the Domain layer, we may also see some supporting Domain Logic:

class RegisterUserCommand {

	protected $email;
	
	protected $password;
	
	public function __construct($email, $password)
	{
		// Setter email/password
	}
	
	public function getEmail() { ... } // return $email
	
	public function getPassword() { ... }	// return $password
}

class UserCreatedEvent {
	
	public function __construct(User $user) { ... }
}

Above we have some supporting (but very important) domain logic. One is a Command (aka a Use Case) which defines a way in which our application can be used. It simply takes in the data needed to create a new user. We'll see how that's used later.

Another is an example Domain Event, which our application might dispatch after a user is created. These are important to the things that occur within a domain and so belong in the domain layer. They are not system events often found within the plumbing of our frameworks such as "pre-dispatch", often used for hooks in case framework behavior needs to be extended.

Application Layer

Just outside of the Domain Layer sits the Application layer. This layer orchestrates the use of the entities found in the Domain Layer. It also adapts requests from the Framework Layer to the Domain Layer by sitting between the two.

For example, it might have a handler class handle a use-case. This handler class in the Application Layer would accept input data brought in from the Framework Layer and perform the actions needed to accomplish the use-case.

It might also dispatch Domain Events raised in the Domain Layer.

This layer represents the outside layer of the code that makes up the application.

Of course, you can see that outside of the Application Layer sits the "Framework Layer". The Framework layer contains code that helps your application (perhaps by accepting an HTTP request or sending an email), but is not your application itself.

Framework Layer

The Framework Layer sits outside of the Application Layer. It contains code that your application uses but it is not actually your application. This is often literally your framework, but can also include any third-party libraries, SDKs or other code used. Think of all the libraries you bring in with Composer (assuming you use PHP). They are not your framework, but they do act in the same layer - performing tasks to handle application needs.

The Framework Layer implements services defined by the application layer. For example, it might implement a notification interface to send emails or SMS. Your application knows it needs to send notifications, but it may not need to care how they are sent (email vs SMS for example).

class SesEmailNotifier implements Notifier {

	public function __construct(SesClient $client) { ... }
	
	public function notify(Message $message)
	{
		$this->client->sendEmail([ ... ]); // Send email with SES particulars
	}
}

Another example is an event dispatcher. Inside the Framework Layer might be code to implement an event dispatcher interface defined in the application layer. Again, the application knows it has events to dispatch, but it doesn't necessarily need to have its own dispatcher - our framework likely already has one, or we might pull in a library to handle the implementation details of dispatch events.

The Framework Layer also adapts requests from the outside to our Application Layer. For example, it's responsible for accepting HTTP requests, gathering user input and routing this request/data to a controller. The Framework Layer can then call an application use-case, pass it the input data, and have the application handle the use case (rather than handling it itself inside of a controller).

In that way, the framework can sit between all requests made on the application externally (us, setting there using a browser) and the application itself (Application Layer and deeper). The Framework Layer is adapting raw requests into our application.

Communication Between Layers: Boundaries

Now that we've seen what code goes inside of each layer, let's talk about an interesting part of each layer: How they communication with each other.

As mentioned, each layer also defines how other layers can communicate with it. Specifically, each layer is responsible for defining how the next outside layer can communicate with it.

The tool for this is the interface. At each layer boundary, we find interfaces.. These interfaces are the ports for the next layer to create adapters for.

We saw this in the notifier and event dispatcher examples above.

The Application Layer will implement interfaces (make adapters of the ports) defined in the Domain Layer. It will also contain code for other concerns it may have.

Let's go through each layer's boundary and see how this works.

Domain Layer

At the boundary of our Domain Layer, we find definitions in how the outside layer (the Application Layer) can communicate with the domain objects/entities found in the Domain Layer.

For example, our Domain Layer might contain a command (use case). Above, we saw an example RegisterUserCommand. This command is pretty simple - you might call it a simple DTO (Data Transfer Object).

Our Domain Layer defines Use Cases, but it's job is just to say "This is how you can use me". Remember, the Application Layer is responsible for orchestrating the Domain Layer code in order to accomplish a task. So, we have communication across boundaries - the Domain Layer defines how it should be used, and the Application Layer uses those definitions (in part) to accomplish the defined use cases.

Our Application Layer, therefore, needs to know how to handle this command to register a user. Since we have communication across these layers, let's define an interface "at the boundary" of the Domain Layer:

interface CommandBus {

    public function execute($command);
}

So, we've told or Application Layer how to "execute" a command, using a Command Bus. The Command Bus is simple - it just needs to have a method execute available so that implementations can process a Command.

Our Domain Layer contains this CommandBus interface, so that our Application Layer can implement the CommandBus interface. The interface is the port, and the implementations of it are the adapters to that port.

Cognitively, we've done a few things:

1. Recognized that CommandBus may be processed in a few ways
2. Recognized that because of that, we may have multiple implementations of the CommandBus
3. Recognized that since our Application Layer orchestrates the use of the Domain Layer, it makes sense for the CommandBus implementation(s) to exist in the Application Layer (they'll orchestrate the use of the Commands defined in the Domain Layer).
4. We've decoupled the Domain Layer from the Application layer by using an interface. This also defines communication between layers. Command Buses have a defined way to execute commands.

Application Layer

So our Application Layer can implement a Command Bus. That's right in the middle of this layer - implementations (adapters) to other layers.

However the Application Layer has its own needs to communicate. The Application Layer might need to send a notification to a user. The Framework has the tools to do so - it can send emails, and we can pull in libraries to send SMS messages or other notification transports. The framework is a good place to implement our notification needs.

So, we have communication between layers. The Application Layer needs to send a notification, and we know it can use libraries in the Framework layer to do so. You know what's coming up: another interface!

interface Notifier {

	public function notify(Message $message);
}

The Application Layer is defining how it will be communicating to the Framework Layer. In fact, it's defining how it will use the Framework Layer, without actually coupling to it. Interfaces (ports) and implementations (adapters) give us the freedom to change the adapters. We don't tie our application to the Framework Layer in this way.

The interface defined "at the boundary" of the Application Layer is defining how the Application Layer will communicate with the Framework Layer.

This is very much conceptual and is not meant to be taken as concrete rules. If you find yourself asking "What if my Domain Layer needs a third party library found in the framework?", fear not!

If that's a need, then define an interface and implement it using that library! You have to make your code work after all - worrying about breaking the "rules" from some dude or dudette on the internet won't get you anywhere!

The key point is to make sure to decouple concerns (hint: You're doing so by defining an interface) so functionality is easy to switch/modify later. There's no need to live and die by what I write here. There's no "doing it wrong". To repeat: There's no doing it wrong. There's just varying levels of severity in how you shoot yourself in the foot.

A good starting place to read up on the topic of requiring third party libraries in your "domain layer" is in this linked thread.

Framework Layer

So far we've seen the boundary in our Domain Layer and in our Application Layer. These two layers both communicate with layers under our control. The Domain Layer communicates with the Application Layer. The Application Layer communicates with the Framework Layer. Who does the Framework Layer communicate with?

The outside world! That world is one filled with protocols - mostly TCP based protocols (such as HTTP/HTTPS). There is certainly lots of code in the framework layer (all the libraries we use), as well as some code we write ourselves, such as controller code and implementations of interfaces defined in the Application Layer.

What exactly is at the boundary of the Framework Layer and the "layer" outside of it, however? Well more interfaces (AND implementations of those interfaces) of course!

Most of our frameworks have code that takes care of talking to the outside work - HTTP implementations, various SQL implementations, various email implementations, and so on.

Luckily, for the most part, we don't have to care about the boundary between the framework layer and the outside world. That's the framework's concern; our benevolent framework creators have taken care of this for us.

This is, arguably, the whole point of a framework. Frameworks provide tools for us to communicate to the world outside of our application, so that we don't need to write that boiler plate code ourselves.

We don't usually need to add to the Framework Layer at its boundary, but of course this isn't always the case. If we're building an API, HTTP level concerns become an issue we need to work through. This usually means implementing CORS, HTTP caching, HATEOS and other specifics in how our application handles HTTP level requests - concerns that are important to our application, but aren't likely concerns of the Domain Layer or even the Application Layer.

Use-Cases/Commands

Earlier in this writing, I've made mention of "Use Cases" and "Commands." Let's go deeper into what these are.

Hexagonal Architecture isn't just about communication between layers on the micro level (interfaces, implementations for ports and adapters). There's also a concept of the Application Boundary, a macro-level concept.

This boundary separates our application as a whole from everything else (both framework and communication with the outside world).

We can strictly define how the outside world can communicate with our application. We do this explicitly by creating "Use Cases" (also called "Commands"). These essentially are classes which name actions that can be taken. For example, our RegisterUserCommand defines that our application can register a user. A UpdateBillingCommand might be the code path defined for us to update a user's billing information.

A Use Case (Command) is an explicitly defined way in which an application can be used.

Defining Use Cases has some useful side-affects. For example, we clearly and explicitly can see how our application "wants" to be interacted with. This can strictly follow the business logic that our application needs to perform. Use Cases are also useful for clarity amongst a team of developers. We can plan use cases ahead of time, or add them as needed, but we find it harder to create odd logic outside of use cases, which don't seem to fit business logic.

How do we define use cases?

We saw some examples already - What we can do is create objects representing an application use case. We’ll call such an object a "Command." These commands can then be processed by our application "Command Bus", which will call a command "Handler" to orchestrate the execution of the use case.

So, we have three actors in command processing:

1. A Command
2. The Command Bus
3. A Handler

A Command Bus accepts a Command in its execute method. It then does some logic to find and instantiate a Handler for that Command. Finally, the Handler's handle method is called, running the logic to fulfill the Command.

class SimpleCommandBus implements CommandBus {

	// Other methods removed for brevity
	
	public function execute($command)
	{
		return $this->resolveHandler($command)->handle($command);
	}
}

Notice that we are taking coordinating logic we often see within a controller and moving it into a Handler. This is good, as we want to decouple from our framework layer, giving us the benefit of protecting our application from changes in the framework as much as possible (another form if maintainability), and allowing us to run the same code in other contexts (CLI, API calls, etc).

The main benefit of use cases is that we create an avenue to re-use code run in multiple contexts (web, API, CLI, workers, etc).

For example, the code to create a new user in web, API and CLI can be almost exactly the same:

public function handleSomeRequest()
{
	try {
		$registerUserCommand = new RegisterUserCommand( 
			$this->request->username, $this->request->email, $this->request->password );
			
		$result = $this->commandBus->execute($registerUserCommand);
		
		return Redirect::to('/account')->with([ 'message' => 'success' ]);
	} catch( \Exception $e )
	{
		return Redirect::to('/user/add')->with( [ 'message' => $e->getMessage() ] );
	}
}

What might change between contexts is how we get user input and pass it into the command, as well as how we handle errors - but those are mostly framework-level concerns. Our application code doesn't need to care if its being used in an HTTP browser request, an HTTP api request or any other request type.

That's where we see the potential of Use Cases. We can re-use them in every context our application can be used (HTTP, CLI, API, AMQP or queue messaging, etc)! Additionally, we've firmly set up a boundary between a framework and our application. The application can, potentially, be used separately from our framework.

That being said, we still might use a framework to implement some application level needs, such as validation, event dispatching, database access, email drivers and many other things our frameworks can do for us! The Use-Case application boundary is just one aspect of Hexagonal Architecture.

Use Case/Command's main benefit is keeping code DRY - we can re-use the same use case code in multiple contexts (web, API, CLI, etc).

Use Cases also serve to further decouple your application from the framework. This gives some protection from framework changes (upgrades, etc) and also makes testing easier.

Taken to an extreme, you can potentially switch frameworks without re-coding our application. However, I consider this the edgiest edge case to ever case edges. It's neither a realistic nor a worthy goal. We want our applications to be easy to work with, not fulfill some arbitrary metric or rare use case.

Example Command, Command Bus and Handler

First, our application knows it needs Commands. It also knows it needs a Bus to execute the commands. Finally, we need a Handler to orchestrate the execution of the command.

Commands are, in a sense, arbitrary. Their purpose is simply to exist. Their mere existence fulfills the role of defining how an application should be used. The data they demand tells us what data is needed to fulfill the command. So, we don't really need to interface a Command. They are simply a name (a description) and a DTO (data transfer object).

class RegisterUserCommand {

    public function __construct(username, email, password)
    {
        // set data here
    }

    // define getters here
}

So, our Command used to register a new user is quite simple. All at once, we provide an explicit definition of one way our application can be used, and what data should accompany that command.

Our Handlers are a bit more complex. They are coupled to a Command in that they expect the data from a Command to be available. This is a spot of tight coupling. Changing some business logic may result in changing the Handler, which may result in changing the Command. As these are all concerns of the all-important business logic, this tight coupling within Domain concerns is deemed "OK".

While Commands are simple DTO's (containing various data), Handlers have behavior, which the Command Bus makes use of. The handlers, being in the Application Layer, orchestrate the use of Domain entities to fulfill a Command.

The CommandBus used to execute a command must be able to execute all Handlers, and so we'll define a Handler interface to ensure the Command Bus always has something it can work with.

interface Handler {

    public function handle($command);
}

Handlers then must have a handle method, but are free to handle the fulfillment of the Command in any way it needs. For our RegisterUserCommand, lets take a look at what it's Handler might look like:

class RegisterUserHandler {
	
	public function handle($command)
	{
		$user = new User;
		$user->username = $command->username;
		$user->email = $command->email;
		$user->password = $this->auth->hash($command->password);
		
		$user->save();
		
		$this->dispatcher->dispatch( $user->flushEvents() );
		
		// Consider also returning a DTO rather than a class with behavior
		// So our "view" layers (in whatever context) can't accidentally affect
		// our application - it can just read the results
		return $user->toArray(); 
	}
}

We can see that our handler orchestrates the use of some Domain Entities, including assigning data, saving it and dispatching any raised events (if our entities happen to raise events).

Similar to our interface example where we used a Decorator to add some extra behavior to our notifier, consider what behaviors might be useful to add to a CommandBus or Handler.

Lastly, we'll discuss the most interesting of our three actors - the Command Bus.

The Command Bus can have multiple implementations. For example, we can use a synchronous Command Bus (running commands as they are received) or perhaps we can create a queue Command Bus, which runs all queued commands only when the queue is flushed. Or perhaps we choose to create an asynchronous Command Bus, which fires jobs into a worker queue, to be worked on as the jobs are received, out of band of the current user's request.

Since we have multiple possible implementations, we'll interface the Command Bus:

interface CommandBus {

    public function execute($command);
}

We've seen a simple implementation of this already. Let's see it a bit more fleshed out:

class SimpleCommandBus implements CommandBus {

	public function __construct(Container $container, CommandInflector $inflector)
	{
		$this->container = $container;
		$this->inflector = $inflector;
	}
	
	public function execute($command)
	{
		return $this->resolveHandler($command)->handle($command);
	}
	
	public function resolveHandler($command)
	{
		return $this->container->make( $this->inflector->getHandlerClass($command) );
	}
}

The CommandInflector can use any strategy to get a Handler from a Command class. For example, return str_replace('Command', 'Handler', get_class($command)); is effective. It's simple, and only requires you keep a certain directory structure for Handlers and Commands (assuming PSR-style autoloading). How you accomplish this is up to your and your project needs.

What else might we use besides a "Simple" Command Bus? Well we might instead call it a "SynchronousCommandBus", as it's processing commands as they come - synchronously. This infers that we might also consider creating an AsynchronousCommandBus. Instead of processing the commands directly, it might pass them into a worker queue to get processed whenever the job is reached, out of band of the current request.

In addition to different implementations of the Command Bus, we can also add onto our existing ones with more Decorators. For example, I find it useful to wrap some validation around a Command Bus, so that it attempts to validate the Command data before processing it.

class ValidationCommandBus implements CommandBus {

	public function __construct(CommandBus $bus, Container $container, CommandInflector $inflector) { ... }

	public function execute($command)
    {
        $this->validate($command);
        return $this->bus->execute($command);
    }
    
    public function validate($command)
    {
    	$validator = $this->container->make($this->inflector->getValidatorClass($command));
    	$validator->validate($command); // Throws exception if invalid
    }
}

The ValidationCommandBus is a decorator - it does the validation, and then passes the Command off to another Command Bus to execute it. The next Command Bus might be another decorator (perhaps a logger?) or might be the actual Bus doing the processing.

So, combined with different types of Command Buses and possible behaviors we can add on top of our Command Buses, we have a pretty powerful way to handle our application Commands (Use Cases) being called!

And these ways are insulated as much as possible from layers outside of the Application. The Framework Layer (and beyond) does not dictate how the application is used - the application itself dictates its usage.

Dependencies

Not explicitly mentioned was the notion of dependencies. Hexagonal Architecture espouses a one-way flow of dependencies: From the outside, in. The Domain Layer (the inner-most layer) should not depend on layers outside of it. The Application Layer should depend on the Domain Layer, but not on the Framework Layer. The Framework Layer should depend on the Application Layer, but not on externalities.

We talked about interfaces as the primary means to encapsulate change. These let us define how communication between layers was accomplished within out application without coupling layers together. Thinking about dependencies is another way of saying the same thing. Let's see how.

Dependencies: Moving In

When our data/logic is flowing "in", dependencies are easier to visualize. If an HTTP request reaches our server, we need code to handle it, otherwise nothing happens. External HTTP requests require our Framework Layer to interpret a request to code. If our Framework interprets a request and routes it to a controller, the controller needs something to act upon. Without our Application Layer, it has nothing to do. The Framework Layer depends on the Application Layer. Our Application Layer needs the Domain Layer in order to have something to orchestrate - it depends on the Domain Layer in order to fulfill a request. The Domain Layer (for the most part) depends on the behavior and constraints found within itself.

When we talk about a request starting from the outside, and the flow of code for handling a request moving inward, dependencies are fairly easy to spot. The outside layers depend on the inside layers, but they can also be ignorant of what the inner layers are doing - they just need to know the methods to call and the data to pass. Implementation details are safely encapsulated away in their proper place. Our use of interfaces between layers has seen to that.

Dependencies: Going Out (IoC)

Dependencies going out are a little more complex. This describes what our application does in response to the request: Both in processing a request and responding to it when a request is processed. Let's look at some examples:

Our Domain Layer will likely need database access to create some domain entities. This means our application "depends" on some sort of data storage.

Our Application Layer may need to send a notification when it finishes a task. If this is implemented as an email notification (for example, if we're using SES), then our Application Layer can be said to depend on SES for sending a notification email.

We can see here that conceptually, our inner layers are depending on things found in layers outside of it! How do invert this?

Of course I used the term "invert" on purpose. This is the point of "Inversion of Control", the "I" in SOLID. Here again, our interfaces serve us well.

We Invert Control by using Interfaces. These allow our layers to inform other layers how they will be interacted with, and how they need to interact with other layers. It's up to the other layers to implement these interfaces. In this way, we are letting our inner layers dictate how they are used. This is Inversion of Control.

Our Domain Layer can define an interface for a repository class. This Repository Class will be implemented in another layer (Likely up in the Framework Layer with our Framework database classes). By using an interface, however, we are decoupling our Domain Layer from the specific persistence type used: We have the potential to change persistence models when we test, or if project needs dictate an actual technological change (lucky you, if you get to scale high).

It's a similar situation with the Notifier. Our Application Layer knows it needs to send out notifications. That's why we created the Notifier interface. Our Application doesn't need to know how the notifications are sent - it just knows it needs to define how it's to be interacted with. And so our Notifier interface, defined in the Application Layer, is implemented by an email (perhaps SES) in our Framework Layer. The Application Layer has inverted control by using an interface; it has told the outside layer how it's going to be used. The layers are decoupled as implementations can easily be switched.

So, when our logic is flowing from the "outside, in", we make use of our interfaces again. We employ the user of Inversion of Control so that dependencies keep flowing in one direction. We decouple our inner layers from outside layers, while still making using them!

Conclusion

This covers a lot of material! What you read here is the result of a lot of code architecture study. Instead of being very specific, it's a bit on the general side - we're dealing with concepts here, instead of concrete "do it this way" type rules.

Overall, Hexagonal Architecture is a description of "good" code practice. It's not a specific way to go about coding applications. Its concepts works for opinionated frameworks, as well as for the "no framework" crowd.

Hexagonal Architecture it another way to look at the same old rules we're reading about as we learn more about code architecture.

A New Repository

Vaprobash is moving onto Ubuntu 14.04 LTS, where it will stay until 16.04 LTS (if it's still relevant then).

Some people still need Ubuntu 12.04, and so they will still be able to use it! Going forward, there will be two development tracks (two repositories): 14.04 and 12.04. I'll be spending most of my time on 14.04. The 12.04 repository is now official in "maintenance". PR's and bug fixes are still welcome.

The fideloper/vaprobash repository is now at Ubuntu 14.04 LTS. You can, as always, grab the Vagrantfile by using http://bit.ly/vaprobash:

wget -O Vagrantfile http://bit.ly/vaprobash

For those who need to use Ubuntu 12.04 LTS (perhaps if you need to test against php 5.3 or php 5.4), there is the fideloper/vaprobash12 repository. The Vagrantfile for this repository is located at http://bit.ly/vaprobash12:

wget -O Vagrantfile http://bit.ly/vaprobash12

The Goal

Vaprobash is meant to get you a development server up and running, as quickly and painlessly as possible.

It's secondary goal (along with serversforhackers.com) is to have you learn how to set up servers yourself. To this end, Vaprobash uses bash scripts, which are reasonably clear in showing each step towards installing various pieces of software onto an Ubuntu server.

The hope is that by not obscuring (and complicating) the install process with a provisioner of some sort (Ansible, Chef, Puppet), people can learn by seeing, copying and tinkering on their own.

I want people to learn enough to grow out of Vaprobash.

Stability

Vaprobash gets a lot of pull requests. I love this - it's a sure sign that it's simple enough for people to grasp and modify.

However, this has a detriment. At any point in time, there's usually a bug somewhere as well as some new feature all intermingled into the develop or master branches. This makes it hard to pinpoint a release.

Furthermore, PR's regularly introduce bugs; They aren't always thoroughly tested. I can understand that - provisioning a server over and over to test all the various use cases is a pain. It's a slow process, and there are a ton of variations (depending on what people choose to provision).

As of yet, there isn't a great testing process. Even if we used the magic of Docker to make relatively quick, repeatable tests, it would still be hard to test. It's not enough to test that Nginx was installed. We also need to test that it's working (running), and it's virtual servers are configured, and probably some other factors as well. Furthermore, tests could be misleading due to other settings changed by other provisioners - they are not all self-contained. For example, Apache and Nginx need to be aware of if PHP was installed.

If anyone has a good solution for this, I'm all ears. I think some sort of Docker-based testing would be amazing.

Going Forward

Moving forward, I want to start using a tagging/versioning system for both repositories. This will lend some stability for those who need it.

Tagging versions will allow users to "freeze" Vaprobash, so they get the same sets of install scripts every time they provision a new server. This means that their installations are much, much less likely to be different, even as Vaprobash changes (new features, bug fixes).

Each repository will soon be marked as stable with a 1.0.0 release. Then bug fixes will be pulled in readily (increasing PATCH versions) while non-breaking features will increase MINOR versions. Finally new (breaking) features will increase MAJOR versions.

This will be similar to Laravel, where there will be branches for MAJOR.MINOR releases, and tags for all MAJOR.MINOR.PATCH releases.

PR's for new features will be added in more slowly. Perhaps use that extra time to keep testing your scripts :D

This is because there's no one way to accomplish a goal in coding. I find this fortunate, as novice programmers would be screwed by the mental leaps needed to understand a senior programmer's methodologies. It would also halt progress of discovering new and useful patterns.

Instead, everyone is somewhere in a spectrum between a "programming novice", and being an "experienced programmer". The difference between novice and experience is learning techniques in code maintenance.

We're all on our way to learning new techniques and ideas. What we write today, we'll throw out tomorrow. This article, and its expansion of ideas I touched on in my Laravel book, is one personal reflection of that.

So what's my point?

Instead of copying everything to the letter, take what you read and modify it to your needs. Try new and stupid things. Also, the depth of knowledge you'll find in a good book is better than anything you'll find in a blog article.