The Registry Meant to Stop Guessing Was Full of Guesses: Auditing My Source Trust Data
One question started it.
“Where did these registry entries actually come from?”
A quick recap: every time I ingest a source, the system assigns it a trust tier. Tiers run S1 through S4, based on the outlet’s track record. The registry (schema/source_registry.yml) stores those assignments so future sessions don’t have to re-derive them from scratch. Once an outlet is confirmed, it’s confirmed. The judgment persists.
That was the design. Then I asked about the twelve entries I’d seeded the registry with when I first built it.
Where the seed data came from
The agent’s answer was honest.
“Those entries were based on general knowledge at the time of registry creation. None of them have been verified by you directly.”
Two of the twelve had real evidence behind them. The other ten were the agent’s best guesses about what those outlets were probably worth, stored in a file that looked, for all practical purposes, like confirmed facts.
The problem went deeper than data quality. The entire point of the registry was to stop re-guessing things that had already been confirmed. But the registry’s own seed data was exactly the kind of unconfirmed guessing the registry was meant to replace. The system I built to prevent this problem had this problem baked in from the start.
Reconstructing evidence from what already existed
Manually re-verifying ten outlets from scratch wasn’t realistic. So I tried a different approach.
The wiki already held ingest records for over 300 sources. Each had a domain and a recorded trust tier. The tier is a function of both source type (S-axis, fixed per outlet) and verifiability (V-axis, varies per article). If a single outlet appeared across multiple ingested sources, the pattern of tiers across those appearances would be consistent with only one correct S-axis value. That’s real evidence, derived from actual usage history, not a fresh guess.
I wrote tools/backfill_registry.py to run that check automatically: scan the full ingest history, find outlets with enough historical appearances to produce a unique S-axis match, and flag or correct the registry accordingly.
One correction came back immediately. uxdesign.cc was recorded as S3. Across its 7 historical sources, the tier pattern matched only S2. My original guess had been wrong, and the data showed it directly. Six other outlets got new or updated entries that the backfill could confirm. The remaining four stayed marked “insufficient evidence.” That’s the right outcome. An honest “we don’t know yet” is better than a confident wrong answer.
What I actually learned
This applies beyond YAML files or tier systems.
When you build a system with an AI agent, the data the agent generates and the data a human has verified tend to look identical on the surface. Same file format, same structure, same apparent confidence. If you don’t explicitly distinguish between the two, they get treated as the same, and the whole system’s reliability settles at the level of the unverified entries.
The registry now marks confirmed and unconfirmed entries explicitly, so trust doesn’t quietly average down to the weakest one in the file. It keeps improving with use: each new ingest either reinforces an existing entry or adds evidence to an unconfirmed one. What’s confirmed stays confirmed; what’s uncertain stays marked as uncertain until it isn’t.
Next episode
Sources started piling up in the inbox faster than I could review them one at a time, and the shortcut I started reaching for is exactly what you’d expect. Next episode covers what that looked like and how I designed around it.