Claira Stories
Running a Large-Scale AI Review
This guide walks through the strategy, sequencing, and judgment calls involved in running a large-scale AI-assisted document review in Claira. It is written for legal teams who are about to AI-review tens or hundreds of thousands of documents and want to do it defensibly, efficiently, and without surprises.
The core idea: an AI review is not a bigger, faster keyword search. It is closer to delegating discretionary work to an informed junior colleague - except that colleague can read 10,000 documents an hour, never gets tired, and keeps a perfect record of what it did and why. Your job shifts from reviewing documents to instructing well, validating output, and exercising the final judgment that gets written to the production field.
1. The mindset shift
Before anything else, get your team out of the Boolean / STF mindset.
Traditional database querying is algorithmic and faceted: think about every spelling, every variant of a name, every way a product or person could be referenced, weighted by metadata like recipient counts or date ranges. That work is no longer necessary. The LLM handles linguistic variability natively - spelling, translations, misspellings, naming variations, synonyms. You do not need to construct a list of every way "the contract" might appear in 200,000 documents.
What replaces that work is plain-language instruction. You tell the model what the case is about and what you want to know, the same way you would brief an articling student over coffee. The quality of the review is now driven by the quality of that briefing, not by the cleverness of your search syntax.
This is the single biggest adjustment for experienced eDiscovery practitioners. Trust the model on linguistic variability. Spend your effort on context and instruction.
2. Two layers of scoping
Every AI review has two layers of scoping. Keep them separate in your head.
Layer 1 - the task. What is this review trying to determine? What are the issues? Who are the relevant people? What is the theory of the case? This must come from the legal team. No one else can write it.
Layer 2 - the output. What format should the answers come back in - yes/no flags per issue, free-text rationale, structured fields, summaries? This is mechanical and Claira handles it. The goal is output that is easy to validate, easy to filter, and easy to bulk-code to production fields without re-running anything through AI.
You worry about Layer 1. Claira worries about Layer 2.
3. Pre-engagement preparation
Before you turn on the subscription and start building prompts, prepare three inputs.
3.1 The case context
This is a written summary of what the matter is about. It does not need to be polished or follow a template. Think of it as the briefing memo you would give a new lawyer joining the file. Include:
A short factual narrative of the dispute or investigation
The relevant time period
The main allegations or issues in play
Theory of the claim or defense, where relevant
Key entities, products, projects, or systems referenced in the documents
You can pull this from pleadings, but pleadings tend to be poor source material - too procedural, too stripped down. A briefing memo, an internal case summary, or a strategy document is usually better. If you have one already written for the file, that is your starting point.
Claira will run this raw context through a structured extraction step to produce a case context object - a clean breakdown of issues and people that the model uses on every subsequent prompt. You will see this structured output and confirm it before any review runs. This is the first sanity check.
3.2 The questions
List every question you want answered about the documents. Do not pre-filter, do not try to optimize, do not collapse questions you think are similar. If you would assign 30 or 50 questions to a team of articling students, give us 30 or 50 questions.
Write them in plain language, in whatever language is natural for your team. The model is language-agnostic on input and output.
Claira reviews the question list and recommends how to structure them - which to combine, which to tier, which to ask only on documents that pass an earlier filter. That recommendation depends on the question shape, so giving us the full list matters more than giving us a tidy one.
3.3 The exclusion list
Identify any document binders, custodians, source collections, or document types that the team is comfortable excluding from review entirely. Be conservative - the throughput is high enough that broad inclusion is usually the right default - but there are almost always pockets of clearly out-of-scope material that do not need to be touched.
This list comes alongside the context and questions when you hand off for prompt construction.
4. Building the prompt
A Claira prompt is the case context plus the question. The case context is shared across the matter; the question is what changes per task. Together they tell the model: here is what the matter is about, here is what I want you to determine on this document.
Claira builds and refines the prompt initially, especially on a team's first review. You see every prompt, every context object, and every output. Nothing is hidden.
The prompt produces structured output that gets written to dedicated AI fields on each document - never to the production relevance field directly. This separation is deliberate: the AI's determination is one input to your decision, not the decision itself. You bulk-code from the AI fields to production fields after QC, on your own judgment.
Each output also includes a short rationale - one sentence on why the model answered the way it did. This is what makes skim-review possible at scale, and it is what gives you defensibility.
5. Quality assurance before the full run
Do not run the prompt against the full population the first time you build it. Run it against small batches and read the output carefully.
The pattern:
Pull a sample of documents you already know should hit each issue - the obvious ones, the low-hanging fruit you could find with a basic keyword search.
Run the prompt against 50 of them.
Read the output.
Run another 50.
Read the output.
You are not looking for statistical representativeness here. You are looking at whether the model is interpreting the instruction the way you meant it. If it is over-coding because the instruction was too broad ("everyone with that surname") or under-coding because the instruction was too narrow, you refine the prompt and run again. This is the "junior associate comes back with the first batch" moment. Most prompts need at least one round of refinement; some need several.
For document-type heterogeneity (emails, contracts, technical specs, spreadsheets), consider running QA batches per type to confirm the prompt holds up across formats. You may decide to split the run into separate batches per document type.
6. The full run
Once QA is clean, the full review runs. Claira processes around 10,000 documents per hour. A 200,000-document set finishes in under a day of clock time.
For most matters, Claira recommends a tiered workflow rather than asking every question of every document. The pattern:
A first-pass prompt makes a coarse determination - is this document related to the matter at all, and to which broad bucket of issues.
Documents that fail the first pass do not get the full battery of detailed questions.
Documents that pass go through the tier-two prompts, which are more granular.
Tiering does two things: it concentrates token spend on documents that have potential, and it keeps the output cleaner because irrelevant documents are not being asked questions they have no answer to. Claira recommends the tiering structure based on the question list.
7. Quality control after the run
QC is fundamentally different in an AI review than in a human review.
Human reviewers make random errors - one reviewer codes inconsistently with another, an attention lapse miscodes a document, fatigue drives drift over a week of work. Random errors require random sampling to find. Traditional discovery QC samples around 350 documents from a population to bound the error rate at 5% or 2.5%, regardless of the size of the population (up to about 10 million documents).
AI reviewers do not make random errors. They make consistent errors - the prompt was ambiguous in one direction, so the model bent every borderline document the same way. You do not need 350 random documents to find that. You need stratified sampling: pull the documents that hit each issue and look at whether the instruction was interpreted as intended.
In practice, QC looks like this:
Pull all documents the AI flagged for issue 1. Skim 20-30 with their rationales. Are these all reasonably hits? If yes, move on. If something is consistently off, the prompt was wrong - refine and re-run that issue.
Repeat per issue.
Spot-check non-hits to confirm the prompt is not systematically missing a category.
Two important asymmetries to internalize:
False negatives are the real risk. A document that should have been flagged and wasn't is a problem you may never discover. Bias QC effort toward catching these.
False positives are tolerable. An over-inclusive prompt that flags some documents that turn out not to be relevant is fine - you catch those at human review, and the cost is reading time, not lost evidence.
QC can be deferred. The AI determinations are persistent fields on each document. They sit in the database. You can run the prompts in May and do final QC in November and the output is still there, queryable, with no additional token spend. This matters when review capacity is the bottleneck rather than processing capacity. You can run the AI work as fast as the prompts allow, then bring on human reviewers later to validate and bulk-code without re-running anything.
8. Bulk coding to production
After QC, the lawyer makes the call on what gets coded to the production relevance field. Because the AI output sits in dedicated fields with rationale, this is usually a bulk operation: filter on AI fields where the team is confident, bulk-code to production fields. Where the team disagrees with the AI, those documents go to manual review or get re-prompted.
The defensibility narrative is straightforward: yes, AI was used. The lawyer made the determination. The audit trail shows what context was loaded, what prompt was run, what the model returned, and what the human decided. This avoids the hallucination concern in the same way human-supervised junior work avoids it - the senior lawyer is responsible for the call.
9. Defensibility and audit trail
Everything is recorded:
The case context object, including the structured issues and people extracted from the briefing
Every prompt, including all revisions
The link between each prompt and the responses it produced
Per-document outputs with rationale
This means the team can demonstrate, for any given coding decision, the exact instruction that was running when the AI made its determination. The breadcrumbs go all the way back to the original briefing. If a coding decision is challenged later, the chain of reasoning is reconstructable.
10. Working with question lists - some refinements
A few practical notes on the question list:
Do not write Boolean logic. "Documents that mention X but not Y" is a query pattern, not a question. Ask the underlying question - "Is this document discussing X in the context of A?" - and let the model handle the logic.
Be specific about what makes a yes a yes. "Is this document relevant to the dispute" is too vague. "Does this document discuss negotiations between Party A and Party B regarding the termination clause" is workable.
Ask for the rationale to be useful. Specify what you want in the one-sentence explanation - the actor, the action, the date, whatever lets you skim-validate fastest.
Group related questions. If three questions all only matter for documents that hit a parent issue, flag that. Claira will tier them.
11. Timing and scale
Some practical numbers to plan against:
Subscription needs to be active before prompt-building. Storage of context and prompts is per-case.
Around 10,000 documents per hour throughput.
A typical engagement spends most of its calendar time on prompt construction and QA, not on the full run.
Realistic timeline for a 200,000-document set, from kickoff to handoff to human reviewers: roughly four weeks, mostly front-loaded on context, questions, and QA iteration.
12. Common adjustments to expect
Almost every review involves at least one of the following adjustments during QA. None are problems - they are the normal shape of refining instruction.
The prompt was too broad on a specific name or entity, picking up unrelated namesakes. Tighten with disambiguating context.
The prompt missed a category of relevant documents because the team had a tacit assumption that wasn't written into the context. Add it to the context, re-run.
The output rationale is consistent but not informative for skim-review. Adjust the output format.
A document type (a chat export, a particular spreadsheet structure) confuses the model in a consistent way. Split that type into its own batch with a tailored prompt.
13. The summary picture
A well-run AI review on Claira looks like this:
Brief the matter in plain language. Give us your context, your full question list, and what to exclude.
Confirm the structured case context that Claira extracts from your briefing.
Run small QA batches against known-good documents. Refine the prompt until the output is what you would have written.
Run the full population, tiered as recommended.
QC by issue, not by random sample. Look for consistent errors. Bias toward catching false negatives.
Defer final QC if reviewer capacity is the bottleneck - the output persists.
Bulk-code to production fields based on the lawyer's judgment, with the AI determination as input.
Keep the audit trail for defensibility.
The total effort shifts from "reviewing 200,000 documents" to "instructing well, validating output on a few hundred, and exercising final judgment on the bulk." That is the productivity story, and it is also why getting the early steps right matters more than anything else in the workflow.
