Layered Testing for Ansible Hardening Code

Why Containers Stopped Being Enough

A Docker-Compose environment of SSH-reachable containers is a good substrate for service roles. It starts in seconds, runs on a laptop, and exercises real inventories, real group_vars trees and real SSH.

For OS hardening it falls short, because containers share the host’s kernel and boot chain. The things a hardening framework changes most are exactly the things a container cannot represent:

• sysctl values are either read-only or apply to the host, not the container.
• Kernel module blacklists have no effect; the container cannot load modules anyway.
• Boot arguments and GRUB passwords do not exist without a boot loader.
• Mount options for /tmp, /var/tmp or /dev/shm depend on the container runtime, not on /etc/fstab.
• FIPS mode, audit rules marked immutable, SELinux all need a real kernel.
• systemd either runs privileged (which undermines the isolation being tested) or is replaced by a mock (which tests the mock).

A green run in containers said little about whether a host would boot. The approach that replaced it splits testing into layers, each one testing what it can test well.

Layer 1: Static Gates

The cheapest layer runs on every commit, needs no infrastructure and no secrets:

lint: lint-yaml lint-roles     ## yamllint + ansible-lint

lint-yaml:
	yamllint -c .yamllint .

lint-roles:
	PYTHONPATH= ansible-lint $(LINT_SCOPE)

Two details matter more than the choice of linters:

• Pin everything. CI installs exact versions of ansible-core, ansible-lint, yamllint and every collection from requirements files. A linter upgrade should be a commit, not a surprise.
• CI never sees secrets. The test configuration uses no vault and no inventory with real hosts. Anything that needs a secret is not a CI test.

Layer 2: Controller-Local Logic Tests

Hardening frameworks contain real logic: merging profiles, computing the effective kernel command line, deciding what to restore during a rollback, validating /etc/fstab. That logic can be tested without any target host if it is written for it.

The prerequisite is structural. Separate computation from change. The profile engine computes all effective settings as facts in one task file that touches nothing on the host:

# roles/profile-manager/tasks/main.yml
- name: Read profile state of the host
  ansible.builtin.include_tasks: state.yml

- name: Compute effective profile configuration     # pure, no changes
  ansible.builtin.include_tasks: compute.yml

- name: Capture pre-change snapshots
  ansible.builtin.include_tasks: snapshot.yml

The test playbook runs on localhost, without root and without vault, and includes the same compute.yml once per case:

# tests/profile-engine/cases.yml
profile_cases:
  - name: baseline
    profiles: [baseline]
    that:
      - _hardening_profiles_normalized == ['baseline']
      - _boot_cmdline_effective == ['audit=1', 'kaslr']
      - _pam_effective.min_length == 14
      - _kernel_sysctl_overrides == {}

  - name: baseline-implicit
    profiles: [ospp]
    that:
      - _hardening_profiles_normalized == ['baseline', 'ospp']

  - name: ospp
    profiles: [baseline, ospp]
    that:
      - _boot_cmdline_effective == ['audit=1', 'kaslr', 'pti=on', 'vsyscall=none', 'page_poison=1', 'slub_debug=P']
      - _pam_effective.min_length == 12

  - name: unknown-profile
    profiles: [baseline, osppp]
    fails: true

Each case declares a profile stack and either a list of assertions on the computed facts or fails: true for stacks the engine must reject. Adding a regression test is adding five lines of YAML.

Test Doubles for System Tools

Some logic has to talk to system tools. The boot argument handling calls grubby; the test puts a small bash stub of it first in PATH:

#!/usr/bin/env bash
# Test double for grubby: one boot entry per line of $GRUBBY_STATE (its args).
set -euo pipefail
state="${GRUBBY_STATE:?GRUBBY_STATE not set}"
case "${1:-}" in
  --info=ALL)
    i=0
    while IFS= read -r line; do
      echo "index=${i}"; echo "args=\"${line}\""; i=$((i + 1))
    done < "${state}" ;;
  --update-kernel=ALL)
    # --args=... adds missing arguments, --remove-args=... removes them
    ... ;;
esac

The stub keeps boot entries in a plain file, so the test can assert on the result after adding and removing arguments. It implements only the behaviour the role relies on, including one subtle detail of the real tool: an argument already present is not added twice. The same approach covers grub2-mkconfig, and fstab parsing is tested against fixture files.

This layer runs in seconds and catches most logic regressions before anything touches a VM.

Layer 3: VM Acceptance with an Idempotency Gate

Everything that depends on a real kernel is tested on real VMs, provisioned from templates of the target distributions. The acceptance procedure is short and strict:

1. Deploy the host from a clean template.
2. Reboot. Boot arguments, FIPS mode, mount options and immutable audit rules only take effect after a reboot. A host that does not come back fails the test.
3. Verify from the outside against the compliance catalogues (see Verifying Hardening Against Several Catalogues at Once).
4. Deploy again. The second run must report changed=0.
5. Check the system state. systemctl is-system-running must report running, not degraded.

Step 5 was added after a real incident class: hardened hosts that came back after a reboot but reported degraded, because a vendor unit tried to load a kernel module that a newer kernel no longer shipped, or because a crash-dump service failed once a crash kernel could not be reserved. Nothing was broken functionally, but monitoring went red. The fix (masking the specific modules-load file, disabling the crash-dump service as part of the baseline, resetting failed units that the hardening disables) came with a new verification control so it cannot regress silently.

VM tests are slower than containers by an order of magnitude. They run per release and per significant change, not per commit, which is why layers 1 and 2 exist.

Layer 4: Provisioning Code with Mocked Providers

The VMs themselves are provisioned with OpenTofu. That code has logic too: size classes, per-VM overrides, NIC queue counts, VLAN validation, disk lifecycle. tofu test with a mocked provider tests it without a hypervisor:

mock_provider "hypervisor" {}

variables {
  name  = "unit-net-vm"
  size  = "m"
  cores = 2
}

run "single_core_gets_two_queues" {
  command = plan

  variables {
    cores = 1
  }

  assert {
    condition     = hypervisor_vm.this.network_device[0].queues == 2
    error_message = "A single-core VM must still get 2 NIC queues."
  }
}

Module outputs such as cores, memory and disk_size_gb exist partly so that tests can assert on them.

The OpenTofu gates mirror the Ansible ones: fmt-check, validate with -backend=false for every stack, a linter, a configuration security scan, and the tests. None of them needs credentials.

One gate is not a test but belongs here: drift detection.

drift:   ## fail if real infrastructure differs from the code
	tofu plan -lock=false -detailed-exitcode

Exit code 2 means the plan is not empty, and the target fails. Run on a schedule, it catches manual changes on the hypervisor before they surprise the next deployment.

One Contract for CI

Each stack has its own Makefile with the same three targets, and the repository root only delegates:

make ansible-lint ansible-test
make tofu-check
make check          # everything

CI runs exactly these targets. A developer runs exactly the same targets locally. There is no separate CI script that drifts from what people actually run.

What Each Layer Catches

Layer	Runs	Catches
Static gates	every commit, seconds	syntax, style, deprecated modules, risky patterns
Logic tests	every commit, seconds	merge rules, rollback plans, argument handling, rejected profile stacks
VM acceptance	per release, minutes per host	boot, kernel, mounts, FIPS, services, idempotency, degraded state
Provisioning tests	every commit, seconds	sizing, validation, resource shape; drift on a schedule

Lessons Learned

1. Test where the effect lives. Kernel and boot changes need a kernel and a boot loader.
2. Design for testability. A pure compute step makes most of the framework testable on a laptop.
3. Stub only what you rely on. A small, honest test double beats a full emulation.
4. Reboot is part of the test. Many hardening changes are invisible until then.
5. The second run is the acceptance test. changed=0 and a running system state are cheap, strict gates.