You Don't Want Magic in Your eDiscovery Process

Arthur C. Clarke once wrote that any sufficiently advanced technology is indistinguishable from magic. He was writing about how hard it is to predict the future, but a good deal of AI software seems to have read the line as a design brief.

You have probably met the type. There is an empty box, you type a question into it, a row of dots pulses for a moment, and an answer arrives from somewhere offstage, fluent and assured and with no visible means of support. It has the charm of a parlour trick, the coin drawn from behind a child's ear, and like any parlour trick it invites you to enjoy the effect without inquiring into the cause.

An audience at a magic show has agreed to be fooled, which is why everyone goes home happy. A lawyer certifying an eDiscovery production has agreed to something far more serious. Their name goes on the result, the client is paying for their judgment, and the court will rely on their account of how the documents were reviewed. We have yet to meet the lawyer who would tell a case management judge that the review was carried out by something wonderful they could not describe.

A review team needs something plainer from its AI, which also happens to be something harder to build. It needs a tool that is fast, effective and easy to use, and whose workings can be followed and explained at each step along the way.

The trouble with a trick

Every conjuror knows that the method has to stay out of sight, because once the audience glimpses the false bottom in the box the wonder drains out of the room. The craft depends on concealment and on drawing the eye elsewhere at the right moment.

eDiscovery runs the other way. When opposing counsel asks why a document was withheld, or a client wants to know how you can be sure nothing was missed, the method is precisely what you have to produce, and you have to produce it in words, often months after the work was done.

A tool that works like magic lets you down here in two ways, and they correspond to the two kinds of error litigators already lose sleep over.

The first is the error of commission, a wrong answer delivered in the same even voice as a right one. When you cannot see how an answer was reached, you have no way of telling the two apart by looking at them. The best-known Canadian example is the one we wrote about in Lessons from Zhang v. Chen, where authorities invented by a chatbot found their way into a British Columbia filing, perfectly plausible and entirely without a source.

The second is the error of omission, and in document review it is the more dangerous of the two because it leaves so little trace. A wrong answer at least sits on the page where someone might notice it, while a missing document makes no sound at all, and a ranked list or a summary offers you no account of what fell below the line or out of the frame. This is why we argued in Every Document Matters that any document which survives culling should be reviewed, and the review recorded.

A good review leaves nothing missed and nothing invented, and the difficulty has always lain in demonstrating that it did. A tool that works like magic can offer you its confidence and little else. Claira was built so that you could see for yourself.

A method you could sketch on a napkin

The way Claira works can be told in a paragraph. You write the question you want answered in plain words, much as you would brief a junior reviewer on the first morning of a matter, and you try it on a handful of documents to see what comes back. You adjust the wording until the answers match what you meant. Then you run the same question across the review set with Bulk Scan, and Claira writes each answer into a field on its document in Nuix Discover, where your team can search, sample and correct it like any other coding.

Because the question is written down in your own words, the documents are the ones you chose, and the answers sit in your own fields beside the documents they describe, each stage of the work leaves something behind that you can inspect later.

The speed comes from asking one well-made question many thousands of times over, which is the kind of patient repetition machines are good at, and the same simplicity is what makes the process easy to explain. You could describe it to a judge in under a minute. Nobody on the team needs to be an AI specialist to follow it, since it is conducted in the ordinary vocabulary of document review.

The ledger Claira keeps

Bookkeepers have their ledgers and scholars their marginalia. Claira keeps a running record of the review as it happens, and several of its most recent features exist mainly to make that record fuller and easier to read.

Prompt History holds every version of every prompt used on a case, along with its author, how often it has been used and any comments the team left on it. When the team settles on a wording, marking it final records that choice where everyone on the case can see it, and the whole history can now be exported as a spreadsheet, one revision per row, with each comment attached to the revision it was written about. That export is the document you reach for when someone asks, a year on, how the review question took its final shape. If you would like the prompt to travel with the evidence itself, a bulk scan can also write it into each document's Prompt Log field.

Claira can draft prompts too, when you ask it to. The Agent and the Prompt Generator can write one from scratch and Revise with Agent can suggest changes to yours, and any prompt produced that way carries a small wand icon in Prompt History; hover over it and Claira tells you which tool wrote it. We rather like the irony that the only wand in Claira is there to show you where the machine has been.

Insights build chronologies, tables and memos from coded documents, and they are designed to be checked line by line. An Insight states how many documents it read. Where it cites a document, the citation opens that document's entry among its sources, and in the exported PDF the links run in both directions, so a colleague who has only the file can travel from a sentence to its source and back without opening Claira or Nuix Discover. Hovering over a Document ID in the pane shows the AI answer that brought the document into the Insight, and if the model ever names a document the Insight did not read, Claira removes that citation before it reaches you. You can even watch an Insight being written as Claira ticks off its steps, from reading the results to checking the references.

Agent Mode is just as open about what it is doing. A chip above the message box shows the document you have open and whether Claira will receive its text alone or the text together with the file, and a single click keeps it out of the conversation. Each chat can be set to ask before it starts any work or to set the work up and start it for you, within limits your administrator chooses, and the largest steps, such as scanning the whole case or creating a field, always wait for a person to press the button.

Nothing missed, nothing invented

To return to the two kinds of error, a tool built to be understood should let you show that a review avoided both, and Claira's record was designed with exactly that in mind.

On omission, the record accounts for every document you selected. As a bulk scan runs it counts each outcome, whether a document was coded, skipped because it had no readable content, or failed, and the task history keeps those counts afterwards. A run interrupted partway through is marked Incomplete and shows exactly how many documents remain, so documents that were never scanned can never pass for finished ones. When you resume the task, it keeps the prompt it started with, which means the last thousand documents are asked the same question as the first.

On invention, the answers stay beside their evidence. Each answer is written to a field on the document it describes, so checking it is a matter of opening that document. An Insight reads your fields as they stand at the moment you click Generate, so a reviewer's correction is the version the chronology works from.

Checking all of this takes very little time. Sampling the documents coded as not responsive remains good practice, and with the answers sitting in fields it is an ordinary search in Discover. A review you can see all the way through is much easier to stand behind.

The same question, asked twice

Give three capable associates the same contract and ask each of them for a summary, and you will receive three accurate summaries that read nothing alike. Ask the same three whether the contract contains an indemnity, and to quote it if so, and you will hear the same answer three times over, because a narrow question leaves so little room to wander.

Our help centre makes the point in a phrase worth pinning above the review desk: summaries may vary, but decisions shouldn't. The article Why you might not get the same answer twice walks through the reasons in plain terms, beginning with the way a language model chooses its words.

Consistency of that kind is what makes a process defensible. When a well-built question reaches the same decision on the same document whoever happens to run it, the decision can be traced back to the question, and the question belongs to you.

Explainable to anyone who asks

As we wrote in The Competence Obligation, the duty of competence asks lawyers to be able to explain what their tools did, in plain terms, much as they would account for the work of a junior reviewer. That duty is difficult to meet with a tool that works like magic, and comfortably within reach with one that keeps its method in view.

Breaking the staff

Near the end of The Tempest, Prospero renounces his art. "This rough magic I here abjure," he says, promising to break his staff and drown his book, and in the epilogue he comes forward without his charms to speak to the audience plainly and ask for their indulgence. It is one of the loveliest moments in Shakespeare, and it is the spirit in which we have tried to build Claira.

There is a great deal of careful engineering underneath, and the speed is real, but at the surface where your judgment meets the work we have tried to keep everything in plain sight, so that the path from your question to the answers in your fields can be followed by anyone who needs to follow it. The result is fast, effective and easy to use, and it can be explained at each step to the people who need to understand it.

Where to start

The surest way to judge any of this is to watch it work on documents you already know. Take a closed matter, write the question your team once answered by hand, ask Claira to give its reasons alongside its answers, and compare where it agrees with your reviewers and where it parts company with them.

If you would like company for that first hour, book a short session with our team, bring along a document set you know well, and we will walk you through how it all works.