AI Systems Evaluation / LLM Auditing
What to call it: AI red-teaming, hallucination detection, epistemic verification of AI-generated output.
Real evidence
- Caught a fabricated confidence-fusion result (invented 0.95/0.48 numbers) presented with false confidence — full retraction demanded and given.
- Caught a real, live citation-integrity gap: the same source cited with different numbers in two different internal documents.
- Caught a filename overclaiming its own content (claimed "15 stress tests," actually 13).
- A whole methodology — eight codified, executable rules — for catching exactly this class of error systematically, not just by luck.
Honest calibration
This is a real, emerging, currently under-supplied professional category (AI workflow auditing / prompt engineering / LLM QA), and the evidence here is concrete and repeatable, not anecdotal — worth its own pitch entirely, separate from the technical architecture work, since it's a different skill being demonstrated: directing and auditing AI, not writing code by hand.
Evidence
Backtest-Only Audit — the systematic self-audit that found real, still-open gaps ↗ · the citation-drift catch — pending source ·
epistemic_protocols.py — same file as QA/Verification, the categories overlap