Incident Log — Entries 001–005 · Charles Jones

Skip to content

Five production incidents, engraved as they ran. Each one starts as a theory — the tinfoil hypothesis you hold before you have an instrument — and ends where measurement either confirmed it or corrected it.

Nothing here is engraved that the record cannot back. Where the record carries no date, no count, or no cure, the entry says so rather than estimating. Two of these five are not closed, and are stamped that way.

Entry 001 · Closed Google Cloud Platform (GCP) billing

A compromised key, and a bill 1,000× the estimate

Theory — hypothesis held before the instruments answered

“A cloud bill cannot run away from you overnight; there is a ceiling somewhere, and somebody is watching it.”

Date: 2026-06
Symptom: $11,089.77 billed against $0.71 expected — a greater-than-1000% anomaly. Charges accumulated across roughly 48 hours, 7–8 June, before the account was suspended.
Diagnosis: A compromised firebase-adminsdk service-account key. The provider's suspension notice cited activity consistent with hijacked resources.
Cure: Moved to a keyless architecture on workload identity federation. Every user-managed key disabled; five plaintext key files shredded, including two that survived in cloud-synced folders and were found only because a previously killed sweep was re-run.
Permanent: Zero plaintext service-account keys anywhere under the home tree, marker-verified rather than assumed. A standing rule now blocks any credential-shaped artifact from entering a long-lived archive — scrub the exported copy, never the read-only source.
Measured: The theory resolved to fact in the wrong direction: there was no ceiling and nobody watching. Project reinstated 2026-06-17. Charges reversed by two banks independently as fraud; the full $11,089.77 was credited back.
Sources: The Register, 2026-07-03 — case covered, operator named. Google Cloud (official account) — public reply to the operator's comment on Google Cloud CEO Thomas Kurian's AI-security post.

The lesson that outlived the incident: a disabled key does not help if a copy is sitting in an archive nobody remembers restoring. The cure had to be the absence of the key, not the revocation of it.

Entry 002 · Closed Evidence integrity

A control was fabricated, then read as evidence

Theory — hypothesis held before the instruments answered

“An audit is trustworthy because it exists. If a file says it is the control, it is the control.”

Date: Cure ruled 2026-08-07. The record does not carry the date the fabrication was introduced, and none is estimated here.
Symptom: Four artifacts sat in the evidence base as controls. Their bytes did not come from the run they claimed to describe — two of the four differed from both the mirror and git, while the other two matched, so nothing about the set announced itself as wrong.
Diagnosis: The control was authored rather than captured. No runner, gate or hook caught it: a fabricated artifact passes every check that asks whether a file is present and well-formed, because it is both.
Cure: Defect header, no disposal. All four files now carry a FABRICATED-CONTROL header naming what they are, why they are not evidence, and where the real evidence base is. The first attempt at this cure failed by doing it in-band.
Permanent: A defective artifact is headered, never deleted — deletion destroys the evidence that the defect happened. The mirror holding the byte-exact pre-header originals is a ruled exclusion from any refresh: re-pushing it would overwrite the proof.
Measured: The first cure was measured two days later and had broken its own artifacts: a comment banner prepended to a CSV was read by csv.DictReader as the header row, yielding 205 bogus rows instead of 192. The JSON half stayed valid — a top-level key parses fine — and was still wrong for the same reason. A version-drift sentinel was commissioned off the back of it; it remains unbuilt, and is recorded as a commission rather than a build.
Sources: Estate record, private: technical-debt ledger TD-122 (the fabricated control) and TD-110; closing rulings filed 2026-08-07.

The theory resolved to its opposite: the existence of an audit says nothing about whether it measured anything. A control has to be captured from the run, and a file that cannot prove its provenance is decoration.

Entry 003 · Closed Serving layer · TLS

The www host presented a certificate for a different domain

Theory — hypothesis held before the instruments answered

“If the certificate were broken, something would have said so. Silence from the monitoring is the same as health.”

Date: Cured 2026-08-04. The record does not establish how long the condition was live, so no duration is claimed.
Symptom: The www host resolved, 301'd to itself over HTTPS, and presented a certificate for firebaseapp.com. Visitors met NET::ERR_CERT_COMMON_NAME_INVALID — a full-page security interstitial, not a 404. On a licensed-insurance surface, a certificate warning costs more trust than a missing page would.
Diagnosis: The host had never been provisioned as a custom domain, so no certificate naming it was ever minted. Measured directly: no alternative certificate subject name matches the www host. Two classic stall causes were checked and cleared — certificate-authority authorization records: none; DNSSEC delegation signer: none — so issuance could not have been blocked by either.
Cure: Provision the host properly as a redirect domain, bracketed by an identical probe before and after, touching no DNS.
Permanent: A hardening step that removes the user's escape hatch must never ship before the thing it hardens is correct. Strict-transport enforcement here would have converted a dismissible warning into a hard lockout cached in the visitor's own browser for up to a year, with no override. Fix the certificate first, or never fix it at all — the strengthening was deliberately excluded from this cure and left as a separate decision.
Measured: Post-cure: valid certificate, clean 301 to the apex, 12/12 path fidelity, apex control unmoved. The theory was refuted — nothing had said so, for as long as it had been true. Separately recorded, and the reason this entry names an instrument rather than a promise: the apex certificate carried an expiry 82 days out and the census did not observe a renewal. Auto-renewal is documented; it was not measured.
Sources: Estate record, private: technical-debt ledger TD-104.

Verified at the edge afterwards, not in a console: this site's own apex and www hosts each carry a dedicated certificate with one subject name, and the expiry is a calendar item rather than an assumption.

Entry 004 · Open · routed around Instrumentation

Analytics flatlined, and the flatline read as quiet

Theory — hypothesis held before the instruments answered

“Monitoring dashboards are mood rings. A flat line means a slow week, and traffic will come back on its own.”

Date: Collection stopped approximately 2026-05; diagnosed and filed 2026-08-06.
Symptom: Analytics reported a flat line for roughly three months. It was read as low traffic rather than as a dead instrument.
Diagnosis: The tag was dead on production: the measurement library was never fetched, so no beacon was possible. A worker-thread script loader scans for its own script type at DOMContentLoaded; the analytics component injected those scripts 1,000–1,500 ms after load, so the scan found nothing. The flat line and the outstanding tag defect were one defect, not two.
Cure: None on the client. The obvious fix — move the tag to the main thread — was withdrawn, because it bought data back by spending the performance standard the site is held to. It stands recorded as rejected so no later session resurrects it: the right diagnosis attached to the wrong cure. Collection was instead re-established server-side, around the defect rather than through it.
Permanent: An instrument that reports zero and an instrument that is dead look identical from the dashboard. A reading of zero is only evidence once a second, independent instrument agrees — here, server-side request logs and search-console data, which do not depend on a browser executing anything.
Measured: The theory was half right and dangerously so: the line really was flat, and the cause was not the audience. Corroborated independently — the reporting pipeline was found to carry no session metric at all, meaning the dashboard had never carried this data in the first place, so no collection path existed that the fire test could have missed. This entry is stamped open: the client tag remains dead, and the server-side path routes around that defect rather than curing it.
Sources: Estate record, private: governing ruling of 2026-08-06 and its follow-on wiring rulings; technical-debt ledger TD-113.

Filed open on purpose. Closing it would require the client tag to fire, and it does not.

Entry 005 · Open · cure staged Gate honesty

One hardcoded state literal, across 14,877 measured pages

Theory — hypothesis held before the instruments answered

“Twelve gates came back green, so the content is correct. That is what the gates are for.”

Date: Cure staged in source 2026-08-09; still held at build.
Symptom: A component hardcoded the literal , FL. On a corpus spanning multiple states, that told a non-Florida consumer their own city was in Florida — a live, consumer-facing misstatement of jurisdiction on an insurance surface.
Diagnosis: One literal in one component, rendered across every page that component touched. The build gates all passed, because none of them read the sentence: they audited structure, links, schema and payload, and a grammatically perfect sentence naming the wrong state is structurally identical to the right one.
Cure: Replace the literal with the page's own resolved state, and make a missing value break the build rather than fall back to a plausible default. Staged in source; it has not shipped, and is held pending a build word.
Permanent: A gate that reports green while never having read the thing it claims to cover is not enforcement — it is a green light with no lamp behind it. This defect class is carried in the record under its own severity, gate honesty: a check that overstates its own coverage is itself the defect. The companion rule: a pass requires a positive census, not merely the absence of an error. Silence is not a pass state.
Measured: 14,877 pages measured with the wrong state label — 82.1% of the set examined. The corpus-wide total is larger. It is not measured here and it is not estimated: a number the measurement did not produce does not appear in this entry. That restraint is a filed rule of the estate, not a stylistic choice.
Sources: Estate record, private: technical-debt ledger TD-128, severity gate honesty.

The theory is the one worth keeping as a warning: green gates measure what they were built to measure, and nothing else. This one is still open, and stamped open.

Reading this log

Entries 001–003 are closed. Entries 004 and 005 are open and say so — 004 is routed around with the underlying defect still live, and 005 has a cure staged but unshipped. Entries are appended, never rewritten; a correction becomes the next entry rather than an edit to an old one.

Entry 001 is public and carries external sources. Entries 002–005 come from a private client estate: the mechanism, the measurement and the doctrine are engraved here, and the client's data is not. Where that record carries no date, no duration or no count, this log says so rather than supplying one.

← The governance architecture these entries run under