In 2023, two lawyers submitted a federal court filing citing six judicial decisions. None of them existed. ChatGPT had invented the cases, complete with quotes and citations, and the lawyers hadn't checked. They were sanctioned. It became the cautionary tale of legal AI — but the real lesson is broader than "check your citations."
Hallucination is a feature, not a glitch
Language models generate the most plausible next words. Plausibility and truth usually overlap — but not always, and the model has no way to tell the difference. In legal work, where a fabricated citation can collapse a case, that property is uniquely dangerous.
The guardrails that actually matter
- A mandatory verification protocol for anything AI-generated that leaves the building
- Tools built on legal databases (Westlaw, Lexis) rather than open chatbots for research
- Clear rules on what client data can enter which tools
- Disclosure norms: when the team tells a client or court that AI was used
The firms pulling ahead aren't the ones avoiding AI. They're the ones using it aggressively for drafting and review — with the discipline to never trust it blindly.