Imagine a law firm with the gold-standard in AI policy. A requirement that two AI training modules are completed before any lawyer can access a generative AI tool. Training that covers hallucination risks, fabricated citations, and prior sanctions cases by name, an Office Manual chapter requiring independent verification of all AI-generated output, a separate, pre-existing citation-checking process, and a straightforward rule for AI use: "trust nothing and verify everything."

That firm is Sullivan and Cromwell - one of the most prestigious and expensive law firms in the world. It has approximately 900 lawyers, and advises OpenAI on the safe and ethical deployment of artificial intelligence.

In April 2026, this storied firm filed an emergency motion in a US federal bankruptcy case that contained roughly 40 errors attributable to their use of AI including: invented cases, made-up quotes, wrong case names, incorrect volume numbers. Their seemingly robust AI-specific verification rule was not followed, nor did the separate citation-checking process catch the resulting errors. They were found by opposing counsel.

It would be easy (and accurate) to say that someone didn't follow the rules, or that the rules that governed AI use were policy, not process. But neither explanation asks the more useful question: why did a trained, capable reviewer, operating under an explicit mandate to verify, not catch these AI-induced errors?

The answer lies in how AI output interacts with the reviewer's own judgment, and how human behaviour interacts with workplace policies.

Start with the cognitive side. Three mechanisms, each well-evidenced, work against the reviewer before any policy has a chance to help.

The first is completion bias. Professionals (myself included) are prone to apply less scrutiny to well-formatted, confidently structured work. This is not a new problem, but AI changes the volume and speed of work that seems complete. When every output arrives looking finished, with correct formatting, plausible structure, and the surface signals of competent work, the moments where you would normally pause and question are not triggered. The Sullivan and Cromwell citations had all the surface signals of "good work." It probably looked like the output of a particularly fastidious associate, with every correct-seeming citation making the next one feel less necessary to check.

The second is confirmation bias. Research across psychology, medicine, and policing found that when AI output aligns with what a professional already believes, scrutiny drops further. Psychologists accepted AI recommendations that matched their own diagnoses and became sceptical only when the AI disagreed. Physicians presented with an AI-confirmed incorrect diagnosis rarely revised their prior. When the output confirms what you expected, verifying it feels redundant, even when that is the moment verification matters most.

The third is calibration drift. A 2026 study of 698 participants found that AI use improved people's actual performance but made them worse at judging their own accuracy. And the more AI-literate the participant, the wider that gap became. Training that builds confidence without building calibration makes the reviewer more certain they have checked thoroughly, not more likely to have actually done so.

These effects compound. A reviewer reads output that looks polished, that confirms what they expected, and that they feel equipped to evaluate because they have completed the training - so they don't do the hard work of checking. The result is predictable, and the standard policy response, more training and stricter rules, is not designed to address it.

Sullivan and Cromwell showed us that even the best AI policy can fail. Two types of intervention help: training people to recognise these biases, and designing the work so that the biases have less room to operate. On the first, there is evidence that bias awareness training works - a 2025 study found a single debiasing session reduced confirmation bias in expert analysts. But research also consistently finds that awareness alone is not enough. The combination of training and structural process change outperforms either on its own.

On the structural side, three things separate verification that works from verification that just looks good on paper.

First, the verification step has to be separate from the drafting step. Ideally done by a different person, in a different session, with different tools. When the person who drafted with AI also verifies the output, they are reviewing their own work under the influence of the same three biases. Separation is not about distrust - it is about designing for how human attention actually works under AI-assisted conditions.

Second, the verification has to lead to the primary source, not a second AI check. Asking the model whether it got it right compounds risk because, when challenged, these tools do not concede. They restate their position with more confidence and more structured reasoning. A research tool that surfaces linked citations is structurally different from self-validation, but only if the reviewer follows the links and confirms the appropriateness of the source material themselves. The workflow has to make following the link the default action, not the optional extra step.

Third, an auditable step is a step that gets done. The verification step has to produce an artefact that confirms what was checked, by when, and by whom (with what tools). What gets measured gets verified, what gets assumed gets skipped.

In Sullivan and Cromwell's case, they had the resources to respond quickly when they were alerted to the errors, filing a three-page error schedule with a full redline the same weekend a problem surfaced. That is remediation infrastructure at the top of the profession. A mid-market firm, whether it is a law practice, an accounting firm, or a financial advisory business, does not have that ability.

But it does not need it. The lesson here is not to invest more in verification, it is to invest differently. A 120-person firm can design a workflow where drafting and verification are separate steps, performed by separate people, with a logged output. A smaller firm can design processes that work with its teams rather than against natural biases and inclinations.

The firms that built the most comprehensive version of the wrong answer now have the hardest time changing course. The advantage belongs to whoever designs first, not whoever has the most lawyers.

Sources

Volokh, E., AI Hallucinations in Filing by a Top Law Firm, Reason/Volokh Conspiracy (April 2026, reproducing the Dietderich letter to Chief Judge Glenn, S.D.N.Y.)

Fernandes et al., AI Makes You Smarter but None the Wiser: The Disconnect Between Performance and Metacognition, Computers in Human Behavior (2026, 698 participants across two studies)

Heerma van Voss et al., Debiasing Training Reduces Confirmation Bias in National Risk Analysts, Scientific Reports (2025, national risk analysts and matched student sample)