One machine, several scores, and what a compliance percentage actually measures

August 11, 2026

I report compliance numbers for a living, so I built a small lab to see where those numbers come from. One Rocky Linux 9 VM, one STIG profile, two remediation sources, one scanner. The machine ended up with two very different scores, and neither one was wrong.

The lab

The target is a KVM virtual machine built from a kickstart file, with the STIG partition layout in place from the first boot: separate /home, /var, /var/log, /var/log/audit, /tmp and /var/tmp on LVM. A second small VM is the Ansible controller.

Before any remediation I took a snapshot of the target with it shut off, so the disk image is consistent and the snapshot stays small. The runbook is four commands: shut down, virsh snapshot-create-as, start, and virsh snapshot-revert when I want to go back. A revert restores the disk exactly, including the SSH host key, so nothing else needs fixing afterward. That made every experiment below cheap to repeat.

The baseline

I ran oscap xccdf eval with the stig profile against the SCAP Security Guide datastream for Rocky 9 (STIG V2R7 content):

• 260 fail
• 190 pass
• 26 not applicable
• 9 not checked

The score was 48.14%. Note that 190 passes out of 450 rules with a real verdict is about 42%, so the score is not a simple pass ratio. Severity weighting is in there. Even a clean install with a hardened partition layout starts under half.

Remediation one: the generated playbook

oscap xccdf generate fix --fix-type ansible produced a flat playbook of roughly 4,771 tasks. Applying it gave ok=3427 changed=488 failed=0 skipped=1116, and a rescan scored 96.31%. Enabling FIPS mode and rebooting moved it to 96.45%. That is only 0.14 points, but FIPS is the kind of finding that matters more than its weight in the score.

Applying real content also surfaced three problems, which were more useful than the number:

1. The generated tasks had non-boolean when: expressions that ansible-core 2.19 rejects as a hard error. I had to set ANSIBLE_ALLOW_BROKEN_CONDITIONALS.
2. The STIG limits accepted SSH public key algorithms to FIPS-approved types, which excludes ed25519. My controller's key was rejected on the next connection and I was locked out until I fixed it from the serial console with an RSA key.
3. After hardening, /home is noexec and fapolicyd is active, so Ansible's drop-a-module-and-run-it model fails with EPERM.

Remediation two: DISA's own content

DISA publishes an Ansible role for the same benchmark, at V2R9. It is about 371 tasks, roughly one per rule, against about 4,771 for the generated one. It ran clean on the same ansible-core: ok=347 changed=227 failed=0 skipped=23, no workaround needed.

I rescanned with the same oscap profile and datastream. Score: 73.85%.

Why they differ

The tempting read is that DISA's content is worse. It is not, and the reasons stack:

• Self-grading. The 96.31% playbook was generated from the same content the scanner checks. Of course it passes.
• Coverage choices. DISA left the crypto policy at DEFAULT. The generated playbook set FIPS:STIG. The scanner checks for the latter. This is also why the ed25519 lockout happened only on the first build.
• Interpretation. Where the two disagree on how a rule is met, the scanner marks a fail.
• Version drift. The scanner checks V2R7. DISA implemented V2R9.

I measured one authority's fixes with another authority's checker, and roughly 22 points of gap appeared. The unmeasured part is the fair test: scanning with DISA's own SCAP benchmark. I have not run it yet, and I expect a Rocky wrinkle there too, since that content targets RHEL and may mark rules not applicable on Rocky.

What this means for a reporter of numbers

A percentage is a property of the tool and the content as much as of the machine. The same host scored 48.14%, 73.85%, 96.31% and 96.45% depending on what was applied and what did the grading. Rule counts, not-applicable handling, manual checks, severity weights, and which revision of the STIG the content tracks all change the denominator or the numerator.

So when I hand someone a number, the number alone is not the finding. The tool, the content version, the profile, and the count of rules excluded or unchecked go with it. Otherwise two honest reports of the same system can disagree by twenty points, and nobody can tell which one to believe.