Hardening container images

Mike Cardwell set up a Flatcar Container Linux server and decided every container on it would be hardened from the start. The first one was PowerDNS Recursor. The post walks through what he changed compared to the official image. He says he isn't singling out PowerDNS and expects similar findings for most images.

Less in the image

The official image is Debian with pdns-recursor added: about 355 MB and 7,500 files. Cardwell's pdns-recursor-trimmed compiles the recursor statically against musl inside an Alpine build container, verifies the source tarball against PowerDNS's documented PGP release keys, and copies the executables onto a scratch image. That got it to 36 MB and two files at first, pdns_recursor and rec_control. A compromised process finds no shell and no system tools, which removes the pre-installed binaries a living-off-the-land attacker would reach for.

He also compiled out SNMP support and the url option of zoneToCache. Both pull in dependencies with a long CVE history. He expects few users to need them and admits he could be wrong. The reason he gives is less the attack surface itself than scanner noise, which comes back later.

Restricting the process

The container's entrypoint is a small C launcher. Before handing over to pdns it applies Landlock rules: read access only to the paths that need reading (essentially the config, not /proc), write access only where writes happen, and restrictions on which TCP ports may be connected to or bound, plus UDP bind ports on new enough kernels. That blocks writing malware somewhere and running it, and blocks connections to typical command-and-control ports. This is privilege-dropping done in a launcher rather than in the application, which is how it gets applied to a program Cardwell doesn't write.

System calls are restricted separately, by a seccomp.json allowlist that lives outside the image and is referenced from the compose file or Kubernetes manifest. He generated it with strace and backed it with extensive smoke tests across architectures and kernels. memfd_create is left off, since that is the usual fileless route around a shell-less, noexec image.

The documented runtime options add the rest: a read-only root filesystem (Landlock may be unsupported by the kernel), a tiny noexec,nosuid,nodev tmpfs on /tmp for the one socket rec_control needs, all capabilities dropped, IPC disabled, net.ipv4.ip_unprivileged_port_start set to 53 so a non-root user can bind the DNS port, and no-new-privileges. go-privilege-dropping covers the same Landlock and seccomp mechanisms from inside a Golang program.

Rebuilding every day

An official image captures whatever Debian shipped on the day it was built. Cardwell's GitLab CI follows the two newest stable trains, 5.4.x and 5.3.x at the time of writing, rebuilds the latest of each daily, and pushes only if something changed. File timestamps are pinned to the epoch so an unchanged build doesn't register as a change. When OpenSSL gets a critical fix, the next day's build picks it up from Alpine automatically. The tag 5.4.6 therefore moves between builds; 5.4.6_<hash> pins one, and 5.4 follows the train.

Each run smoke-tests four images (amd64 and arm64, current and previous train). GitLab's runners have kernels without Landlock, so a smaller test set also runs on five VMs per architecture with different kernels to confirm Landlock and seccomp still behave.

Making the image scannable

Trivy, Grype and OSV Scanner mostly read distro package indexes, and a scratch image has none, so they report nothing. The build writes an SBOM generated from the linker into the image as CycloneDX, and attaches an SPDX copy as an attestation along with SLSA provenance recording the git revision, Dockerfile and build arguments.

Cardwell notes that the official image is also invisible to scanners, in a different way: pdns-recursor isn't in its Debian package list, so a PowerDNS CVE would go undetected, as would CVEs in the Rust crates and vendored JavaScript it includes. His SBOM lists all of them. As a result, scanners started reporting critical and high CVEs in the bundled Handlebars.js. He judged they don't apply to how it's used and recorded that in an OpenVEX file, kept in git, fed to the scanners in CI and attached as another attestation. After each build the pipeline scans the newest images and fails on any finding not covered by the VEX, which is how he learns about new relevant CVEs.

Signing

GitLab issues a short-lived OIDC token during the build, which is exchanged with Fulcio for a signing certificate. cosign signs the multi-arch index digest, which also covers the SBOM, provenance and VEX attestations, and the signature is recorded in the Rekor transparency log. Anyone can check that an image came from the main-branch pipeline:

cosign verify \
  --certificate-identity \
    'https://gitlab.com/grepular/pdns-recursor-trimmed//.gitlab-ci.yml@refs/heads/main' \
  --certificate-oidc-issuer https://gitlab.com \
  grepular/pdns-recursor-trimmed:5.4

A compromised Docker Hub account pushing a different image then fails verification. A compromised GitLab account would not; he uses 2FA. This is the attestation half of supply-chain-security, applied by a downstream repackager with no stored signing key, the same shape as ephemeral-credentials.

Two small additions

With no dig or curl in the image, a health check has nothing to call, so /launcher --healthcheck sends a real DNS query and succeeds on any well-formed response. And in case the daily pipeline silently stops, for example when runner minutes run out, its last step pings healthchecks.io, which alerts if a day passes without one.

The outsider's limit

Cardwell's summary is fewer and newer dependencies, less room to escalate after a compromise, and faster detection of known vulnerabilities. He also names the cost of doing this from outside the project. A new release could add a dependency in a form his SBOM tooling can't see, or a feature that needs a syscall his allowlist blocks and his smoke tests don't cover. The only defence is reading every release, which is what the post itself is. Scanner noise pushing maintainers toward dependency churn is the pressure described in unmaintained-scanner-pressure, and the daily-rebuild choice sits on the opposite side from dependency-vendoring: it trades a frozen, audited dependency set for automatic security fixes.