A real citation is only the beginning: measuring legal AI by the work that survives review

An AI-generated legal memorandum can look complete long before the research behind it is complete. Its headings are orderly, its language is confident, and its footnotes open real documents. For a buyer choosing legal software, that presentation creates a measurement problem: the easiest qualities to demonstrate are not necessarily the qualities that make the answer usable.

Open Market Notes' analysis is that the relevant unit of value is a reviewed conclusion tied to a defined question. Generating a longer memorandum or finding more cases can be useful, but only if those steps reduce the work needed to reach that conclusion. If the reviewer must reconstruct every connection between the question, the source and the answer, some of the apparent productivity gain has merely moved into a less visible stage.

This is an analytical assessment as of October 22, 2026, not a report of a new product test. Its research anchors are explicitly historical. A preregistered study submitted May 30, 2024 found errors remained in the legal-research systems it tested despite retrieval augmentation. That historical result cannot establish current product performance.

What a citation leaves unresolved

Consider a hypothetical assignment asking whether a particular contractual notice was sufficient. A system locates a genuine decision containing similar language and attaches its citation to a categorical answer. The document exists, and the passage may be quoted accurately. The remaining work is to decide whether the passage supports the proposition in the memorandum and whether the differences between the two situations matter.

The reviewer might discover that the passage describes a party's argument rather than the court's reasoning. Or the decision may concern a different procedural question, a different contractual term or an issue that the memorandum silently assumed away. These are hypothetical failure modes, not allegations about any named product. They illustrate why checking the existence of a document and checking an answer should be treated as separate tasks.

An effective interface would expose the bridge between them. Alongside an important proposition, it could show the relevant passage, the role that passage plays in the document, the facts assumed by the generated answer and the unresolved difference from the present question. That makes the interpretation inspectable. A single confidence score, by comparison, can conceal disagreement about what is uncertain: finding the document, interpreting it or applying it.

Authority adds another dimension. A buyer should be able to test whether the system distinguishes a source offered as binding authority from one offered as a persuasive analogy or background explanation. Which treatment is appropriate depends on the forum and issue; no universal badge can settle that question. The useful output is an explicit explanation of the source's proposed role, open to correction by the qualified reviewer.

The version belongs with the conclusion

Even an accurate reading needs a recoverable reference point. The U.S. Supreme Court's official publication page explains that opinion PDFs are updated as publication proceeds from slip opinions toward United States Reports editions.

For product design, this suggests retaining enough information to reproduce what was reviewed: the document version, the passage and the date of retrieval. It does not imply that every publication update changes the legal result. The point is narrower. If a colleague later finds different text or pagination, the team should be able to determine whether it is looking at the same material rather than debating two unidentified copies.

Temporal scope also belongs in the assignment itself. Research into a current question and research into the rules applicable to a historical event can require different cutoffs. A system that silently uses “latest” for both may answer a different question from the one intended. A defensible workflow therefore starts by recording the relevant date and identifying which assumptions still need professional resolution.

The complete review object is consequently larger than a URL. It includes the proposition, supporting passage, source role, temporal scope and reviewer disposition. Preserving that object can also make a correction useful beyond one memorandum: the team can find which conclusions depended on the changed assumption without re-reading every document from scratch.

Human review needs a measurable job

The ABA's Formal Opinion 512, issued July 29, 2024, addresses lawyers' ethical responsibilities when using generative AI, including competence and supervision. It interprets ABA Model Rules, rather than creating a single nationwide rule.

Our operational proposal goes beyond attaching a “human reviewed” label. Define what the reviewer has actually resolved. Has the reviewer located the support, accepted the interpretation, checked the answer against the assignment and recorded remaining uncertainty? These decisions may be made together, but storing them as one undifferentiated approval makes it difficult to learn from errors.

A practical evaluation would use assignments with a known scope and have qualified reviewers assess outputs without knowing which workflow produced them. The comparison should include the ordinary research process, not only another AI system. Record elapsed time through acceptance or rejection, material corrections, unresolved questions and the reasons for returning an answer to research. These are proposed evaluation measures, not reported experimental findings.

The denominator matters. Counting only accepted memoranda can hide assignments that the tool failed to complete. Counting only generation time ignores the review burden. A useful record includes abstentions and abandoned attempts, then explains whether they helped the team identify an unresolved issue or simply transferred the whole assignment back to a person.

Error costs are uneven

Suppose two hypothetical systems save similar drafting time. One regularly adds an irrelevant citation that a reviewer can remove immediately. The other rarely makes a mistake, but occasionally changes the scope of the question without making that change visible. An average correctness score alone may not tell the buyer which workflow is cheaper or safer to use. The second kind of error could require substantially more investigation to detect and repair.

The proposed business measure is therefore reviewed output per unit of total effort, with consequential errors examined separately rather than averaged away. That effort includes finding evidence, checking interpretation, resolving uncertainty and repairing work already shared. It should also include the time spent learning the tool. A demonstration conducted by an expert operator cannot by itself establish how a wider team will perform.

There is a legitimate competing interpretation: a demanding review protocol may erase the savings that made AI attractive. That concern deserves testing. For exploratory research, a tool that produces candidate directions could be valuable even when its prose is not ready for use. The team can evaluate that narrower role rather than forcing the product to pass as an autonomous memorandum writer. Different purposes require different acceptance criteria.

Buying an answerable workflow

The strongest procurement exercise would ask the supplier to work through representative assignments and make failed attempts visible. The buyer should inspect how readily a reviewer can challenge an interpretation, recover the underlying text and change an assumption without losing the rest of the work. A system that returns fewer conclusions but makes their boundaries clear may serve that workflow better than one that always produces a polished answer.

None of the sources cited here proves that a particular product now meets or fails that standard. The historical study, professional guidance and court publication process establish limited factual reference points. The evaluation design and commercial implications are our analysis, and should be tested against the buyer's own tasks.

The investment question for legal AI is how much accepted research it can produce without making verification disproportionately harder. Real citations are a necessary starting point for many assignments. The more discriminating test is whether a qualified person can understand, challenge and ultimately take responsibility for the conclusion with less total work. This article concerns software evaluation, not advice on a legal matter.