Verifying Hardening Against Several Catalogues at Once

SecurityComplianceHardeningLinuxAnsibleAudit

Applying Is Not Verifying

A hardening run that finishes green proves that Ansible executed its tasks. It does not prove that the host is in the state an assessor expects. Values can be overwritten by a package update, a vendor unit, an administrator, or by a later hardening profile that sets a different value on purpose.

Verification therefore runs separately from deployment, from the outside, against an explicit list of expectations: a catalogue.

This article describes how such catalogues are structured, how they are executed efficiently, and how to deal with the case that breaks most compliance tooling: hosts hardened against several baselines at once.

A Catalogue Is Data

A catalogue is a YAML file. It lists controls grouped by work package, each with a reference into the source guideline, a shell probe, and the expected output:

catalog:
  id: bsi
  name: "BSI IT-Grundschutz"
  version: "1.0"
  helpers: |
    svc_dis() { local r=$(systemctl is-enabled "$1" 2>/dev/null); [ "${r:-x}" = "enabled" ] && echo enabled || echo disabled; }
    mod_blk() { grep -q "install $1 /bin/true" /etc/modprobe.d/hardening-blacklist.conf 2>/dev/null && echo blocked || echo open; }
    pkg_abs() { rpm -q "$1" >/dev/null 2>&1 && echo installed || echo absent; }

work_packages:
  - id: "1.1"
    name: "Kernel Hardening"
    controls:
      - {ref: "2.3.1", name: "net.ipv4.ip_forward",    probe: "sysctl -n net.ipv4.ip_forward",    expect: "0"}
      - {ref: "2.3.1", name: "kernel.kptr_restrict",   probe: "sysctl -n kernel.kptr_restrict",   expect: "2"}
      - {ref: "2.3.1", name: "Module: usb-storage",    probe: "mod_blk usb-storage",             expect: "blocked"}
  - id: "1.5"
    name: "Service Control"
    controls:
      - {ref: "2.3.3", name: "kdump disabled",         probe: "svc_dis kdump",                   expect: "disabled"}
      - {ref: "2.3.3", name: "systemd state running",  probe: "systemctl is-system-running",     expect: "running"}

Design choices that matter:

  • Probes are plain shell. Anyone can run a probe by hand on the host and see the same value the tool saw. An assessor does not need to trust the tool.
  • Helpers normalize output. systemctl is-enabled has a dozen possible answers; svc_dis reduces them to the two the control cares about.
  • Every control carries its reference. A failed line points straight to the requirement it implements.
  • Catalogues are independent of the deployment code. They describe the target state, not how it is reached. The same catalogue can verify a host hardened by hand.

The framework ships catalogues for the BSI baseline, the Enterprise Linux security guideline (OSPP), directory-integrated hosts, and vendor best practices, with between roughly twenty and a hundred and forty controls each.

One Script, One Connection

Running a hundred controls as a hundred SSH commands is slow and noisy in the target’s auth log. Instead, the verifier generates one bash script per host: the helpers, followed by one line per control that prints a marker, the control’s index and the measured value:

# generated
svc_dis() { ... }
mod_blk() { ... }
echo "V|0|$(sysctl -n net.ipv4.ip_forward 2>/dev/null)"
echo "V|1|$(sysctl -n kernel.kptr_restrict 2>/dev/null)"
echo "V|2|$(mod_blk usb-storage)"
...

The script runs over a single SSH connection as the automation user; the few probes that need root (some sysctl keys are only readable with it) say so explicitly with sudo in the probe. The verifier parses the marker lines and compares each value with expect. Anything else on stdout (a login banner, a warning) is ignored because it does not carry the marker.

Output comes in three shapes: a full table, a per-work-package summary for reports, and JSON for pipelines.

The Problem: Several Baselines on One Host

Hosts are often hardened against more than one guideline. A national baseline forms the base layer; a protection profile such as OSPP is stacked on top for hosts assessed against it.

The two guidelines disagree in places:

ControlBaseline expectsOSPP expects
kernel.kptr_restrict21
minimum password length1412
GSSAPIAuthentication (directory-integrated hosts)–no; directory profile: yes

On a host running both, the later profile wins, by design (see Reversible OS Hardening). Verified naively against the baseline catalogue, that host fails kernel.kptr_restrict. The failure is false: the value was replaced on purpose. Teams then start ignoring failures, and the real ones get lost.

Multi-Catalogue Mode

The verifier accepts a chain of catalogues:

./platform verify bsi,ospp c_00000_acme 101---app-01

The order on the command line is the profile order. It mirrors the engine’s rule that the later profile wins.

Execution has four steps:

  1. Load every catalogue in the chain.
  2. Build the union of all probes, deduplicated by their exact shell string. The measured value depends only on the probe, not on any catalogue’s expectation, so shared controls run once.
  3. Run the deduplicated script once per host.
  4. Evaluate each measured value against every catalogue that defines the probe.

Status Semantics

StatusMeaning
PASSThe catalogue defines the control and the value matches.
FAILThe catalogue owns the control and the value does not match.
OVERRIDDENThe catalogue defines the control, but a later catalogue in the chain expects a different value.
ERRThe probe returned no value.
n/aThe catalogue does not define the control.

Ownership is derived from the chain alone:

  • The last catalogue that defines a probe owns it and is evaluated as PASS or FAIL.
  • An earlier catalogue with the same expectation is evaluated normally: they agree.
  • An earlier catalogue with a different expectation is marked OVERRIDDEN.

For kernel.kptr_restrict on a host measuring 1:

CatalogueExpectsOwnerResult
bsi2noOVERRIDDEN
ospp1yesPASS

If the host measured 0, the owning catalogue would report FAIL and the baseline would still report OVERRIDDEN. Every real failure is attributed to exactly one catalogue: the one that owns the control.

  Verify (multi): bsi -> ospp  ·  c_00000_acme/prod
  Unique probes: 193  ·  Catalogues: 2

  -- 101---app-01 --
  | Control                 | bsi        | ospp |
  |-------------------------|------------|------|
  | net.ipv4.ip_forward     | PASS       | PASS |
  | kernel.kptr_restrict    | OVERRIDDEN | PASS |
  | pam minlen              | OVERRIDDEN | PASS |
  | Module: usb-storage     | PASS       | n/a  |
  | kernel arg: pti=on      | n/a        | PASS |

Declared Deviations

Sometimes a later profile must deviate from an earlier one for functional reasons. Directory-integrated hosts need GSSAPIAuthentication yes for Kerberos logins; OSPP requires no.

The directory catalogue declares this by defining a control with a byte-identical probe and its own expectation. In the chain bsi,ospp,directory, the OSPP control is reported as OVERRIDDEN instead of FAIL. The deviation is visible in the catalogue file, reviewable, and tied to the profile that causes it, instead of being a line in an exception spreadsheet.

The profile engine enforces the matching order on the deployment side (the directory profile must come after OSPP), so verification and deployment cannot disagree about who wins.

Why Not Read the Override Map from the Host?

The profile engine records its overrides on the host. Using that file to decide OVERRIDDEN would be convenient, and wrong: the verifier would then trust the system it is supposed to check. Deriving ownership from the catalogue chain keeps verification independent. The host contributes measured values and nothing else.

Catalogues Evolve with Incidents

Catalogues are not written once. Two controls in the example above came from real incidents rather than from a guideline:

  • kdump disabled. A crash-dump service that failed to reserve memory left hosts in a failed state; disabling it is also good practice because dumps contain memory contents.
  • systemd state running. Hosts that booted fine but reported degraded after hardening. The control makes that state a verification failure instead of a monitoring surprise.

Every fix that changes the target state should come with a control that would have caught the problem.

Conclusion

Compliance verification earns trust when it is independent of deployment, transparent enough that an assessor can rerun any probe by hand, and precise about ownership when several baselines apply.

The OVERRIDDEN status is a small addition with a large effect: it removes false failures without hiding real ones, and it turns deliberate deviations between baselines into something the tool reports instead of something people have to remember.

Verifying Hardening Against Several Catalogues at Once - Patrick Paechnatz