Entry 012 — "All steps passed" printed two lines after two suites announced they had proved nothing

Theory — the hypothesis held before the instruments answered

“Two test suites need cloud credentials the build server deliberately does not hold. They now skip with a plainly worded reason instead of erroring, so nobody mistakes a skip for a pass.”

Date — introduced and cured 2026-09-07, within about ninety minutes.

Symptom

Found by the other machine, reviewing the change. The build log, unedited:

withhold_1_controls: skipped … This suite asserted NOTHING.
publish_redact_controls: skipped … This suite asserted NOTHING.
── CI-1 · ALL STEPS PASSED ─────────────────────────────────────

Two lines apart. On every run of the build server — which holds no credentials by design, so every run — that is two suites proving nothing, reported as a pass.

Diagnosis

The skip exits with a success code. The build script read only the exit code. The sentence saying this suite asserted nothing went to the log for a person to read, and the script that writes the verdict never saw it.

The estate’s own rule on this is one line: a pass is a success code AND something actually scanned. It was written in the docstring of the very function that broke it.

The other machine’s sentence is the one that lands: the reader that matters is the gate, and it sees only the exit code.

Cure

A skip is now recorded to a file. The build script counts what is in that file and names each one, and the verdict is structurally two different sentences — it cannot say the clean one while anything skipped.

credentials present   ── CI-1 · ALL STEPS PASSED, 0 skipped
credentials absent    ── CI-1 · steps passed, 2 SUITE(S) ASSERTED NOTHING
                           SKIPPED withhold_1_controls · publish_redact_controls
                         A pass is rc=0 AND scanned > 0. rc=0 with 2 unscanned.

A skip that cannot be recorded refuses rather than skipping silently.

Measured

A second fault appeared while fixing the first. On a machine with credentials the build reported two unscanned — and both were the checking suite’s own test fixtures, writing into the real record. A test that writes into the thing it measures reports about itself. Its subprocesses now get their own file.

Permanent

Four new tests. The one worth naming asserts that a skip which cannot be recorded refuses, because the alternative — carry on quietly — is the original fault wearing a fix’s clothes.

The reusable part is the distinction between the two readers. A message written for a person is not a signal to a machine, and a log line is not an instrument. Every honest sentence in that output was true. None of it reached the thing that decided the verdict.

Sources

Estate record, private: technical-debt ledger TD-MM-176, opened and closed 2026-09-07; the related item TD-MM-160, closed the same day.

The warning was addressed to a reader who was not the one deciding.