How do you measure privilege review quality?

Privilege is the one area of document review where a single miss can be unrecoverable. Quality therefore has to be measured, not assumed.

Two error types matter, and they pull in opposite directions:

• Under-designation: a privileged document is produced. This is the disclosure risk everyone worries about.

• Over-designation: a non-privileged document is withheld. This inflates the privilege log, invites challenges, and costs credibility.

The usual measurements:

Elusion testing. Sample the documents the review coded as not privileged and re-review them carefully. The proportion that turn out to be privileged is your elusion rate, and it is the closest thing to a direct measure of what you are about to produce by mistake.

Recall and precision. Against a carefully reviewed control sample, recall tells you what share of truly privileged documents were caught, and precision tells you what share of withheld documents were genuinely privileged.

Reviewer consistency. Route the same documents to more than one reviewer and compare. Wide divergence usually means the privilege criteria are ambiguous, not that the reviewers are careless.

Log quality. Descriptions should support the claim without waiving it, and should characterise similar documents consistently.

Whatever the method, write the protocol down before you start. A documented, sampled, reproducible process is what makes the result defensible when it is challenged.

Related reading: Privilege review in the age of generative AI.

One structural advantage of AI-assisted review is consistency: the same criteria applied to every document, with the reasoning recorded for QC. You can see how Claira handles it.