Use service provided healthcheck, with fallbacks (#870)

Two main issues with existing healthcheck setup:
- Only supports `wget` to `/health` out of the box. This requires some projects
  to overwrite the healthcheck line, which while minor, is still a custom
  deviation from the template they now have to carry and potentially deal with
  on updates.
- Requires an extra production runtime dependency (`wegt`) that projects may not
  need.

We could add some configurable options, as we've done in other places[1], but
rather than support a limited set of configurable healthcheck options, prefer a
defined `healthcheck` executable that the service image provides (since it will
best know what healthy is for itself), falling back through some common HTTP
options if a `healthcheck` executable is not provided.

And rather than (additionally) injecting `container_port` Terraform variable
into the healthcheck line, we can rely on the runtime `$PORT` env var, which is
part of the required service interface.

Longer term this also better supports folks trimming down their image size. As
it's quite likely if they are running an HTTP service, their existing stack will
support making an HTTP request, which they can use to provide the healthcheck
without any additional dependencies.

Misc. tweaks while here:
- Move `update-docker-digest` from `/template-only-bin/` to `/template-only-app/bin/`
- Fix the casing of `AS` on the `FROM` line in `/template-only-app/Dockerfile`

[1] navapbc/platform-test@6274c14#diff-2f885dc9fa1b819ce453d1f1e8f6a4c575cf19f7e7199135010961c69edf7340

Author: doshitan

.github/workflows/template-only-ci-app.yml

@@ -22,3 +22,11 @@ jobs:
     - uses: actions/checkout@v4
     - name: Run build
       run: make release-build
+
+  healthcheck-script-tests:
+    runs-on: ubuntu-latest
+
+    steps:
+    - uses: actions/checkout@v4
+    - name: Test healthcheck examples
+      run: ./bin/test-healthchecks

infra/modules/service/main.tf

@@ -103,8 +103,11 @@ resource "aws_ecs_task_definition" "app" {
         interval = 30,
         retries  = 3,
         timeout  = 5,
+        # If a `healthcheck` executable is available in the container's $PATH,
+        # use that, otherwise fall back the first available of: wget, curl, or
+        # bash.
         command = ["CMD-SHELL",
-          "wget --no-verbose --tries=1 --spider http://localhost:${var.container_port}/health || exit 1"
+          "([ -x \"$(command -v healthcheck)\" ] && healthcheck) || ([ -x \"$(command -v wget)\" ] && wget --quiet --output-document=/dev/null http://127.0.0.1:$PORT/health) || ([ -x \"$(command -v curl)\" ] && curl --fail --silent http://localhost:$PORT/health > /dev/null) || ([ -x \"$(command -v bash)\" ] && bash -c \"exec 3<>/dev/tcp/127.0.0.1/$PORT;echo -e 'GET /health HTTP/1.1\\r\\nHost: http://localhost\\r\\nConnection: close\\r\\n\\r\\n' >&3;grep -q '^HTTP/.* 200 OK' <&3\") || exit 1"
         ]
       },
       environment = local.environment_variables,

template-only-app/Dockerfile

@@ -1,5 +1,5 @@
 # Run `make update-docker-digest` to update the image
-FROM python:3-alpine@sha256:657dbdb20479a6523b46c06114c8fec7db448232f956a429d3cc0606d30c1b59 as release
+FROM python:3-alpine@sha256:657dbdb20479a6523b46c06114c8fec7db448232f956a429d3cc0606d30c1b59 AS release
 
 RUN adduser --system --disabled-password --no-create-home app
 
@@ -15,6 +15,7 @@ COPY requirements.txt ./
 RUN pip3 install --no-cache-dir -r requirements.txt
 
 COPY db-migrate /usr/bin/
+COPY bin/healthcheck-netcat /usr/bin/healthcheck
 COPY migrations.sql /app/
 COPY *.py /app/
 COPY /templates /app/templates
@@ -26,5 +27,7 @@ ENV HOST=0.0.0.0
 # Run as non-root user
 USER app
 
+HEALTHCHECK CMD /usr/bin/healthcheck
+
 # Create a basic webserver and run it until the container is stopped
 CMD ["python", "-m", "app"]

template-only-app/Makefile

@@ -9,4 +9,4 @@ release-build:
 		.
 
 update-docker-digest:
-	../template-only-bin/update-docker-digest Dockerfile
+	./bin/update-docker-digest Dockerfile

template-only-app/bin/healthcheck-bash-tcp

@@ -0,0 +1,25 @@
+#!/usr/bin/env bash
+# https://tldp.org/LDP/abs/html/devref1.html
+
+# open connection
+exec 3<>/dev/tcp/127.0.0.1/${PORT}
+
+# send request
+echo -e "GET /health HTTP/1.1\r\nHost: http://localhost\r\nConnection: close\r\n\r\n" >&3
+
+# read response
+#
+# one could use grep, like:
+#
+#   grep "HTTP/1.1 200 OK" <&3
+#
+# but we'll rely purely on bash builtins
+
+# read the first line
+read -u 3 status_line
+
+# be a good citizen, close the file descriptor as soon as we are done with it
+exec 3<&-
+
+# check for a 200 OK status response
+[[ "$status_line" =~ ^HTTP/.*\ 200\ OK ]]

template-only-app/bin/healthcheck-curl

@@ -0,0 +1,4 @@
+#!/usr/bin/env sh
+# https://curl.se/docs/manpage.html
+
+curl --fail --silent http://localhost:"${PORT}"/health > /dev/null

template-only-app/bin/healthcheck-netcat

@@ -0,0 +1,6 @@
+#!/usr/bin/env sh
+# https://busybox.net/downloads/BusyBox.html#nc
+# https://man.openbsd.org/nc.1
+# https://manpages.debian.org/unstable/netcat-traditional/nc.traditional.1.en.html
+
+printf 'GET /health HTTP/1.1\r\nHost: http://localhost\r\nConnection: close\r\n\r\n' | nc 127.0.0.1 "${PORT}" | grep -q '^HTTP/.* 200 OK'

template-only-app/bin/healthcheck-wget-busybox

@@ -0,0 +1,10 @@
+#!/usr/bin/env sh
+# https://busybox.net/downloads/BusyBox.html#wget
+
+# --spider causes a HEAD request instead of GET, which while not that different
+# and in most frameworks where you implement a GET handler it will automatically
+# handle HEAD requests as well, some checkers (like AWS ALBs) specifically send
+# GET requests, so we should match.
+#
+# So use --output-document instead, throwing away the response.
+wget --quiet --output-document=/dev/null http://127.0.0.1:"${PORT}"/health

template-only-app/bin/healthcheck-wget-gnu

@@ -0,0 +1,10 @@
+#!/usr/bin/env sh
+# https://www.gnu.org/software/wget/manual/wget.html
+
+# --spider causes a HEAD request instead of GET, which while not that different
+# and in most frameworks where you implement a GET handler it will automatically
+# handle HEAD requests as well, some checkers (like AWS ALBs) specifically send
+# GET requests, so we should match.
+#
+# So use --output-document instead, throwing away the response.
+wget --tries=1 --quiet --output-document=/dev/null http://127.0.0.1:"${PORT}"/health

template-only-app/bin/test-healthchecks

@@ -0,0 +1,86 @@
+#!/usr/bin/env bash
+#
+# Check that the healtcheck-* examples have consistent basic behavior
+
+set -euo pipefail
+
+SCRIPT_DIR=$(dirname "$0")
+
+PORT=3000
+export PORT
+
+trap_background_jobs() {
+    trap 'trap - SIGTERM && kill $(jobs -p)' SIGINT SIGTERM EXIT
+}
+
+start_server_ok_response() {
+    SERVER_DIR=$(mktemp -d)
+
+    pushd "${SERVER_DIR}" > /dev/null
+    # create the file so the python server will return a 200 response for
+    # requests to /health
+    touch health
+    python -m http.server ${PORT} &> /dev/null &
+    popd > /dev/null
+
+    # Give the server time to start
+    sleep 1
+}
+
+start_server_fail_response() {
+    pushd "${SERVER_DIR}" > /dev/null
+    # trigger a non-200 response for requests to /health
+    rm -f health
+    popd > /dev/null
+}
+
+run_healthchecks() {
+    local run_after_each_healthcheck=$1
+    local exit_code
+
+    for healthcheck in "${SCRIPT_DIR}"/healthcheck-*; do
+        echo "${healthcheck}"
+        "${healthcheck}" && exit_code=$? || exit_code=$? && :
+        echo "Exit code: ${exit_code}"
+        ${run_after_each_healthcheck} "${exit_code}"
+        echo ""
+    done
+}
+
+# shellcheck disable=SC2317
+fail_if_non_zero() {
+    local exit_code=$1
+    [ "${exit_code}" == "0" ] || { FAIL_TEST=true && echo "Failed"; }
+}
+
+# shellcheck disable=SC2317
+fail_if_zero() {
+    local exit_code=$1
+    [ "${exit_code}" != "0" ] || { FAIL_TEST=true && echo "Failed"; }
+}
+
+# Start tests
+
+trap_background_jobs
+
+FAIL_TEST=false
+
+# Healthy response
+start_server_ok_response
+
+echo "::group::Test handling of healthy responses"
+run_healthchecks fail_if_non_zero
+echo "::endgroup::"
+
+# Unhealthy response
+start_server_fail_response
+
+echo "::group::Test handling of unhealthy responses"
+run_healthchecks fail_if_zero
+echo "::endgroup::"
+
+if [ "${FAIL_TEST}" == "true" ]; then
+    exit 1
+else
+    exit 0
+fi

template-only-bin/update-docker-digest renamed to template-only-app/bin/update-docker-digest

@@ -5,14 +5,14 @@ In order to use the template infrastructure, you need an application that meets
 * The application's source code lives in a folder that lives in the project root folder e.g. `/app`.
 * The application folder needs to have a `Makefile` that has a build target `release-build` that takes in a Makefile variable `OPTS`, and passes those `OPTS` as options to the `docker build` command. The top level [Makefile](/Makefile) in this repo will call the application's `release-build` make target passing in release tags to tag the docker image with.
 * The web application needs to listen on the port defined by the environment variable `PORT`, rather than hardcode the `PORT`. This allows the infrastructure to configure the application to listen on a container port specified by the infrastructure. See [The Twelve-Factor App](https://12factor.net/) to learn more about designing applications to be portable to different infrastructure environments using environment variables.
-* The web application needs to have a health check endpoint at `/health` that returns an HTTP 200 OK response when the application is healthy and ready to accept requests.
-* The Docker image needs to have `wget` installed. This is used in the container task definition's healthcheck configuration in order to ping the application's `/health` endpoint. If you want to use a different healthcheck command (e.g. `curl`) then you'll need to modify the `healthCheck` configuration in the `aws_ecs_task_definition` resource in [modules/service/main.tf](/infra/modules/service/main.tf).
+* The web application needs to have a health check endpoint `GET /health` that returns an HTTP 200 OK response when the application is healthy and ready to accept requests.
+* Provide an executable named `healthcheck` in container's `$PATH` which exits with code `0` if your service is healthy, and non-zero if not healthy (some examples in `/template-only-app/bin/`). Or have `wget`, `curl`, or `bash+grep` available, in which case the application's `/health` endpoint will be pinged for container healthchecks.
 
 ## Database Requirements
 
 If your application needs a database, it must also:
 
-* Have a `db-migrate` command available in the container's PATH for running migrations. If you use a migration framework like [Alembic](https://alembic.sqlalchemy.org/) or [Flyway](https://flywaydb.org/) you can create a `db-migrate` script that then calls your framework's binary.
+* Have a `db-migrate` command available in the container's `$PATH` for running migrations. If you use a migration framework like [Alembic](https://alembic.sqlalchemy.org/) or [Flyway](https://flywaydb.org/) you can create a `db-migrate` script that then calls your framework's binary.
 * Both the application service container and the container running the `db-migrate` script will receive the following environment variables that are needed to [connect to the database using IAM authentication](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/UsingWithRDS.IAMDBAuth.Connecting.html):
   * `DB_HOST` - the hostname to connect to
   * `DB_PORT` - the port that the database is listening on