AI Platform Engineer building Model Context Protocol servers and agentic AI systems. 1.35M pages in production, quad-100 Core Web Vitals, every number sourced.https://callcharlie.ioen-ushttps://callcharlie.io/incident-log/incident-006-minted-review-datehttps://callcharlie.io/incident-log/incident-006-minted-review-dateOne timestamp file fed both dateModified and lastReviewed across ~61,000 pages, and a pre-build gate refused any build that did not bump it. The estate could not ship without minting the claim.Thu, 27 Aug 2026 00:00:00 GMT<blockquote> <p><strong>Theory</strong> — the hypothesis held before the instruments answered</p> <p>“A ‘last reviewed’ date on a page means a person reviewed it. The pipeline would not put it there otherwise.”</p> </blockquote> <p><strong>Date</strong> — filed 2026-08-11 (containment); ruled at the root 2026-08-12; cure staged in source 2026-08-12; shipped in the wave promoted 2026-08-23; measured on production and closed 2026-08-27.</p> <h2 id="symptom">Symptom</h2> <!-- literal: dated-fact 2026-08-27 --> <p>One timestamp file fed both <code>dateModified</code> and <code>lastReviewed</code> on ~61,000 <code>WebPage</code> nodes. A pre-build gate refused any build whose timestamp was older than 60 minutes, with the instruction “Every build MUST bump isoTimestamp.” The estate could not ship without minting the claim.</p> <h2 id="diagnosis">Diagnosis</h2> <p>The gate demanded a human fact — a review — from a build fact — a timestamp. The file also carried a licensed producer’s name and license number as <code>lastReviewedBy</code>; that personal form was never published, because no build completed after the 2026-08-11 bump was reverted. The impersonal form was live: served JSON-LD carried <code>lastReviewed</code> = a build timestamp, <code>lastReviewedBy</code> 0, <code>reviewedBy</code> 0.</p> <h2 id="cure">Cure</h2> <p>Variant (b): <code>lastReviewed</code> removed from every generated page’s graph; <code>dateModified</code> retained; review fields opt-in and human-set only; the dormant <code>lastReviewedBy</code> deleted from the source file. Grep across source for a default <code>lastReviewed</code> emitter: zero.</p> <h2 id="permanent">Permanent</h2> <p>No build may mint a per-page human-review claim that no human performed. <code>dateModified</code> belongs to the build; <code>lastReviewed</code> belongs to a human. A gate that demands the second will be fed it by a script.</p> <h2 id="measured">Measured</h2> <!-- literal: dated-fact 2026-08-27 --> <p>On the cure build, the schema gate reported 61,574 pages missing <code>lastReviewed</code> — read first as a regression, then recognised as the cure landing. The corpus-wide count of pages that had carried</p> <!-- literal: dated-fact 2026-08-27 --> <p>the claim is recorded as ~61,000, declared-diff estimate only; the measured figure is 61,574. Production, 2026-08-27, 8 pages sampled (3 named + the first 5 of the deploy-verification sample): <code>lastReviewed</code> 0/8, <code>lastReviewedBy</code> 0/8, <code>reviewedBy</code> 0/8, <code>dateModified</code> 8/8. The theory was refuted in the plainest way available: the date was there, the review was not.</p> <h2 id="sources">Sources</h2> <p>Estate record, private: technical-debt ledger TD-133 (gate honesty), closing annotation 2026-08-27; rulings of 2026-08-11 and 2026-08-12.</p> <hr> <p>Demand a claim from the pipeline and the pipeline will mint one.</p> Gate honestyStructured dataBuild gatesIncidentcharlie@callcharlie.io (Charles Jones)https://callcharlie.io/incident-log/incident-005-hardcoded-statehttps://callcharlie.io/incident-log/incident-005-hardcoded-stateTwelve gates came back green while a component told non-Florida consumers their own city was in Florida. A grammatically perfect sentence naming the wrong state is structurally identical to the right one.Sun, 09 Aug 2026 00:00:00 GMT<blockquote> <p><strong>Theory</strong> — the hypothesis held before the instruments answered</p> <p>“Twelve gates came back green, so the content is correct. That is what the gates are for.”</p> </blockquote> <p><strong>Date</strong> — cure staged in source 2026-08-09; still held at build.</p> <h2 id="symptom">Symptom</h2> <p>A component hardcoded the literal <code>FL</code>. On a corpus spanning multiple states, that told a non-Florida consumer their own city was in Florida — a live, consumer-facing misstatement of jurisdiction on an insurance surface.</p> <h2 id="diagnosis">Diagnosis</h2> <p>One literal in one component, rendered across every page that component touched. The build gates all passed, because none of them read the sentence: they audited structure, links, schema and payload, and a grammatically perfect sentence naming the wrong state is structurally identical to the right one.</p> <h2 id="cure">Cure</h2> <p>Replace the literal with the page’s own resolved state, and make a missing value break the build rather than fall back to a plausible default. Staged in source; it has not shipped, and is held pending a build word.</p> <h2 id="permanent">Permanent</h2> <p>A gate that reports green while never having read the thing it claims to cover is not enforcement — it is a green light with no lamp behind it. This defect class is carried in the record under its own severity, <strong>gate honesty</strong>: a check that overstates its own coverage is itself the defect. The companion rule: a pass requires a positive census, not merely the absence of an error. Silence is not a pass state.</p> <h2 id="measured">Measured</h2> <!-- literal: dated-fact 2026-08-09 --> <p>14,877 pages measured with the wrong state label — 82.1% of the set examined. The corpus-wide total is larger. It is not measured here and it is not estimated: a number the measurement did not produce does not appear in this entry. That restraint is a filed rule of the estate, not a stylistic choice.</p> <h2 id="sources">Sources</h2> <p>Estate record, private: technical-debt ledger TD-128, severity <strong>gate honesty</strong>.</p> <hr> <p>The theory is the one worth keeping as a warning: green gates measure what they were built to measure, and nothing else. This one is still open, and stamped open.</p> Gate honestyBuild gatesIncidentcharlie@callcharlie.io (Charles Jones)https://callcharlie.io/incident-log/incident-002-fabricated-controlhttps://callcharlie.io/incident-log/incident-002-fabricated-controlFour artifacts sat in the evidence base as controls. Their bytes never came from the run they claimed to describe, and nothing about the set announced itself as wrong.Fri, 07 Aug 2026 00:00:00 GMT<blockquote> <p><strong>Theory</strong> — the hypothesis held before the instruments answered</p> <p>“An audit is trustworthy because it exists. If a file says it is the control, it is the control.”</p> </blockquote> <p><strong>Date</strong> — cure ruled 2026-08-07. The record does not carry the date the fabrication was introduced, and none is estimated here.</p> <h2 id="symptom">Symptom</h2> <p>Four artifacts sat in the evidence base as controls. Their bytes did not come from the run they claimed to describe — two of the four differed from both the mirror and git, while the other two matched, so nothing about the set announced itself as wrong.</p> <h2 id="diagnosis">Diagnosis</h2> <p>The control was authored rather than captured. No runner, gate or hook caught it: a fabricated artifact passes every check that asks whether a file is present and well-formed, because it is both.</p> <h2 id="cure">Cure</h2> <p>Defect header, no disposal. All four files now carry a <code>FABRICATED-CONTROL</code> header naming what they are, why they are not evidence, and where the real evidence base is. The first attempt at this cure failed by doing it in-band.</p> <h2 id="permanent">Permanent</h2> <p>A defective artifact is headered, never deleted — deletion destroys the evidence that the defect happened. The mirror holding the byte-exact pre-header originals is a ruled exclusion from any refresh: re-pushing it would overwrite the proof.</p> <h2 id="measured">Measured</h2> <p>The first cure was measured two days later and had broken its own artifacts: a comment banner prepended to a CSV was read by <code>csv.DictReader</code> as the header row, yielding 205 bogus rows instead of 192. The JSON half stayed valid — a top-level key parses fine — and was still wrong for the same reason. A version-drift sentinel was commissioned off the back of it; it remains unbuilt, and is recorded as a commission rather than a build.</p> <h2 id="sources">Sources</h2> <p>Estate record, private: technical-debt ledger TD-122 (the fabricated control) and TD-110; closing rulings filed 2026-08-07.</p> <hr> <p>The theory resolved to its opposite: the existence of an audit says nothing about whether it measured anything. A control has to be captured from the run, and a file that cannot prove its provenance is decoration.</p> EvidenceGovernanceIncidentcharlie@callcharlie.io (Charles Jones)https://callcharlie.io/incident-log/incident-004-analytics-flatlinehttps://callcharlie.io/incident-log/incident-004-analytics-flatlineThree months of flat line read as low traffic rather than as a dead instrument. The obvious cure was withdrawn because it bought data back by spending the performance standard.Thu, 06 Aug 2026 00:00:00 GMT<blockquote> <p><strong>Theory</strong> — the hypothesis held before the instruments answered</p> <p>“Monitoring dashboards are mood rings. A flat line means a slow week, and traffic will come back on its own.”</p> </blockquote> <p><strong>Date</strong> — collection stopped approximately 2026-05; diagnosed and filed 2026-08-06.</p> <h2 id="symptom">Symptom</h2> <p>Analytics reported a flat line for roughly three months. It was read as low traffic rather than as a dead instrument.</p> <h2 id="diagnosis">Diagnosis</h2> <p>The tag was dead on production: the measurement library was never fetched, so no beacon was possible. A worker-thread script loader scans for its own script type at <code>DOMContentLoaded</code>; the</p> <!-- literal: dated-fact 2026-08-06 --> <p>analytics component injected those scripts 1,000–1,500 ms after load, so the scan found nothing. The flat line and the outstanding tag defect were one defect, not two.</p> <h2 id="cure">Cure</h2> <p>None on the client. The obvious fix — move the tag to the main thread — was <strong>withdrawn</strong>, because it bought data back by spending the performance standard the site is held to. It stands recorded as rejected so no later session resurrects it: the right diagnosis attached to the wrong cure. Collection was instead re-established server-side, around the defect rather than through it.</p> <h2 id="permanent">Permanent</h2> <p>An instrument that reports zero and an instrument that is dead look identical from the dashboard. A reading of zero is only evidence once a second, independent instrument agrees — here, server-side request logs and search-console data, which do not depend on a browser executing anything.</p> <h2 id="measured">Measured</h2> <p>The theory was half right and dangerously so: the line really was flat, and the cause was not the audience. Corroborated independently — the reporting pipeline was found to carry no session metric at all, meaning the dashboard had never carried this data in the first place, so no collection path existed that the fire test could have missed. This entry is stamped open: the client tag remains dead, and the server-side path routes around that defect rather than curing it.</p> <h2 id="sources">Sources</h2> <p>Estate record, private: governing ruling of 2026-08-06 and its follow-on wiring rulings; technical-debt ledger TD-113.</p> <hr> <p>Filed open on purpose. Closing it would require the client tag to fire, and it does not.</p> InstrumentationAnalyticsIncidentcharlie@callcharlie.io (Charles Jones)https://callcharlie.io/incident-log/incident-003-www-certificatehttps://callcharlie.io/incident-log/incident-003-www-certificateThe www host resolved, redirected to itself over HTTPS, and served a certificate for firebaseapp.com. Visitors met a full-page security interstitial, not a 404.Tue, 04 Aug 2026 00:00:00 GMT<blockquote> <p><strong>Theory</strong> — the hypothesis held before the instruments answered</p> <p>“If the certificate were broken, something would have said so. Silence from the monitoring is the same as health.”</p> </blockquote> <p><strong>Date</strong> — cured 2026-08-04. The record does not establish how long the condition was live, so no duration is claimed.</p> <h2 id="symptom">Symptom</h2> <p>The www host resolved, 301’d to itself over HTTPS, and presented a certificate for <code>firebaseapp.com</code>. Visitors met <code>NET::ERR_CERT_COMMON_NAME_INVALID</code> — a full-page security interstitial, not a 404. On a licensed-insurance surface, a certificate warning costs more trust than a missing page would.</p> <h2 id="diagnosis">Diagnosis</h2> <p>The host had never been provisioned as a custom domain, so no certificate naming it was ever minted. Measured directly: no alternative certificate subject name matches the www host. Two classic stall causes were checked and cleared — certificate-authority authorization records: none; DNSSEC delegation signer: none — so issuance could not have been blocked by either.</p> <h2 id="cure">Cure</h2> <p>Provision the host properly as a redirect domain, bracketed by an identical probe before and after, touching no DNS.</p> <h2 id="permanent">Permanent</h2> <p>A hardening step that removes the user’s escape hatch must never ship before the thing it hardens is correct. Strict-transport enforcement here would have converted a dismissible warning into a hard lockout cached in the visitor’s own browser for up to a year, with no override. Fix the certificate first, or never fix it at all — the strengthening was deliberately excluded from this cure and left as a separate decision.</p> <h2 id="measured">Measured</h2> <p>Post-cure: valid certificate, clean 301 to the apex, 12/12 path fidelity, apex control unmoved. The theory was refuted — nothing had said so, for as long as it had been true. Separately recorded, and the reason this entry names an instrument rather than a promise: the apex certificate carried an expiry 82 days out and the census did not observe a renewal. Auto-renewal is documented; it was not measured.</p> <h2 id="sources">Sources</h2> <p>Estate record, private: technical-debt ledger TD-104.</p> <hr> <p>Verified at the edge afterwards, not in a console: this site’s own apex and www hosts each carry a dedicated certificate with one subject name, and the expiry is a calendar item rather than an assumption.</p> TLSServing layerIncidentcharlie@callcharlie.io (Charles Jones)https://callcharlie.io/incident-log/incident-001-compromised-keyhttps://callcharlie.io/incident-log/incident-001-compromised-key$11,089.77 billed against $0.71 expected. A compromised service-account key, two banks, and a cure that had to be the absence of the key rather than its revocation.Wed, 17 Jun 2026 00:00:00 GMT<blockquote> <p><strong>Theory</strong> — the hypothesis held before the instruments answered</p> <p>“A cloud bill cannot run away from you overnight; there is a ceiling somewhere, and somebody is watching it.”</p> </blockquote> <p><strong>Date</strong> — 2026-06</p> <h2 id="symptom">Symptom</h2> <!-- literal: dated-fact 2026-06-17 --> <p>$11,089.77 billed against $0.71 expected — a greater-than-1000% anomaly. Charges accumulated across roughly 48 hours, 7–8 June, before the account was suspended.</p> <h2 id="diagnosis">Diagnosis</h2> <p>A compromised <code>firebase-adminsdk</code> service-account key. The provider’s suspension notice cited activity consistent with hijacked resources.</p> <h2 id="cure">Cure</h2> <p>Moved to a keyless architecture on workload identity federation. Every user-managed key disabled; five plaintext key files shredded, including two that survived in cloud-synced folders and were found only because a previously killed sweep was re-run.</p> <h2 id="permanent">Permanent</h2> <p>Zero plaintext service-account keys anywhere under the home tree, marker-verified rather than assumed. A standing rule now blocks any credential-shaped artifact from entering a long-lived archive — scrub the exported copy, never the read-only source.</p> <h2 id="measured">Measured</h2> <p>The theory resolved to fact in the wrong direction: there was no ceiling and nobody watching. Project reinstated 2026-06-17. Charges reversed by two banks independently as fraud; the full</p> <!-- literal: dated-fact 2026-06-17 --> <p>$11,089.77 was credited back.</p> <h2 id="sources">Sources</h2> <p>The Register, 2026-07-03 — case covered, operator named. Google Cloud (official account) — public reply to the operator’s comment on Google Cloud CEO Thomas Kurian’s AI-security post.</p> <hr> <p>The lesson that outlived the incident: a disabled key does not help if a copy is sitting in an archive nobody remembers restoring. The cure had to be the absence of the key, not the revocation of it.</p> Google Cloud Platform (GCP)CredentialsIncidentcharlie@callcharlie.io (Charles Jones)