Agentic search in due diligence: finding the right document
On FinanceBench, the same model goes from 27% to 86% correct answers depending on the search software. In due diligence, the hard part is finding the file.
In August 2026, Mistral published the FinanceBench results of its agentic search: 150 financial analyst questions asked over 368 filings of listed companies. With the same model, GLM-5.2, accuracy goes from 26.7% to 86% depending on how the software searches the filings. The model does not change, nor do the documents: only the way of fetching the information does.
This confirms what the benchmark already showed in 2023: a language model interprets a financial report well when the right page is put in front of it. What drags accuracy down is having to find that page. The model is no longer the point: what matters is the software that fetches the information and drives the agent.
For a due diligence team, the consequence is direct. A data room is hundreds of look-alike documents: three years of accounts, several versions of a contract, one set of statements per subsidiary. The hard part is not interpreting the documents, it is finding the right one, and that is where AI tools part ways.
Interpreting a document vs. finding the right one: two different tasks
FinanceBench is a question set published by Patronus AI in November 2023 to measure how well an AI answers analyst questions from real filings: annual reports (10-K), quarterly reports (10-Q), earnings releases. Each question has a single, checkable answer: an amount, a ratio, or an explanation drawn from the filing.
The authors asked GPT-4 Turbo the 150 public questions in five settings, changing only what the model is given. Answers were graded by hand. The results in table 2 draw a staircase:
Bar chart: GPT-4 Turbo answers 9% of questions correctly with no document, 19% when it searches the whole filings base, 50% with the right filing in chunks, 79% with the whole right filing, 85% with the right page
With no document at all, the model gets 9% right. With the right page, it reaches 85%. In between, everything depends on how much searching it is left to do: 19% when it must find the passage in a store shared by all filings, 50% when the store holds only the right filing, 79% when the whole filing fits in its context window. From 19% to 85% without changing the model: the gap comes entirely from what has to be found.
The paper's abstract puts it bluntly: GPT-4 Turbo used with a retrieval system "incorrectly answered or refused to answer 81% of questions". A year later, the vendor Ragie found the same gap with its own engine: 27% in the shared store, 51% in a store limited to the right document. Narrowing the search scope doubles accuracy.
The model is no longer the point: same model, the software triples accuracy
Agentic search (or agentic RAG) is software that does not settle for a single search: it chains searches, opens the documents that look relevant, reads through them, checks what it found and decides when it knows enough to answer. The language model writes and reasons; the software around it runs the investigation.
Mistral's post of August 20, 2026 isolates the effect of that software. Same 150 questions, same 368 filings, same model, three ways of searching: a "one-shot" RAG that "retrieves a fixed set of text chunks and asks a model to answer in a single pass", a loop of chained searches, then the same loop with tools to navigate inside the documents.
Two-part chart. Left, same model and different software: Mistral Medium 3.5 goes from 22.7% to 70% then 78.7%, GLM-5.2 from 26.7% to 79.3% then 86%. Right, same software and different model at Dewey: 62.9% for GPT-5.4, 83.7% for Claude Opus 4.6
The search loop adds 47.3 points for Mistral Medium 3.5 and 52.6 points for GLM-5.2; navigation adds another 8.7 and 6.7. All in all, the same model triples its accuracy without anyone touching its weights. Mistral also notes that navigation cuts token use by a quarter to a third: searching better costs less than searching more.
The model still matters. Dewey ran the opposite experiment: same ingestion, same agentic search engine, only the model changes. Claude Opus 4.6 reaches 83.7%, GPT-5.4 62.9%, a 21-point gap. But the same post shows that Opus makes 21.2 calls to the search tool per question on average, against 9.4 for GPT-5.4: part of the gap comes from how each model uses the software. The model matters; the software matters more, with 56 to 59 points of difference at Mistral against 21 at Dewey.
Our own experience points the same way. On FinanceBench, replacing Qwen 3.8 27B, the model that writes ATG's answers, with Gemini 3.8 Flash does not change the score significantly [À COMPLÉTER : score exact obtenu avec Gemini 3.8 Flash]. We also ran the test with the same model at two different hosting providers, one global, one European: 138 and 139 correct answers out of 150. Only the response time changes: 10.5 seconds on average on one side, 30.9 on the other.
What this software does: search, open, choose, stop
The software that drives the search makes four decisions the model alone does not make. It narrows the scope before searching: the company, the fiscal year, the document type. That is what lets a RAG hold up on tens of thousands of documents, as we explain in our article on how many documents a RAG can handle. It chains searches when the first one is not enough. It opens and reads through a filing when the information sits in a table that chunks split badly. Finally, it stops: it decides it has enough evidence, or that it has not found anything and must say so.
An example from our FinanceBench reports shows these decisions at work. Question: "Was there any drop in Cash & Cash equivalents between FY 2023 and Q2 of FY2024?", for Best Buy. The answer needs two documents from two different periods, among the same retailer's annual reports, quarterly reports and earnings releases. ATG cited the 2023 annual report and the Q2 2024 earnings release: $1,874 million at the end of fiscal 2023, $1,093 million at the end of the second quarter of the following fiscal year, a 41.7% drop. The expected answer was "about 42%". Along the way, ATG noted that Best Buy's fiscal year ends in late January, which shifts every label by a year. One search, 7.2 seconds.
The look-alike document trap
A base of financial filings is a hard case for a simple reason: the documents look alike. The 2022 and 2023 annual reports of the same company share the same structure, the same headings, often the same sentences. A semantic similarity search puts them on the same footing. If the software does not filter on the fiscal year, the model gets the right table from the wrong year, and interprets it perfectly.
FinanceBench offers a clue. Eleven of the original questions name neither the fiscal year nor the company ("this 10-K report"): they were written for a single document, while the base holds several per company. Asked as they are, they have no single answer. In our protocol, we added one fixed sentence giving the period or the company, leaving the rest of the question untouched.
A due diligence data room takes this trap further. It holds three years of accounts, statements per subsidiary, several versions of the same contract, spreadsheets that repeat the figures of the PDFs. A correct answer from the wrong fiscal year is still a wrong answer, and it is up to the software, not the model, to pick the right document. The ways to give an AI access to these documents (API, MCP, export) are covered in our article on data rooms and AI.
What this changes for due diligence
Each FinanceBench setting matches a due diligence situation.
| FinanceBench setting | Due diligence situation | GPT-4 Turbo (2023) |
|---|---|---|
| no document | asking a consumer assistant what it knows about the target | 9% |
| the whole filings base | querying the whole data room | 19% |
| the right filing, in chunks | uploading the annual report to an assistant | 50% |
| the right page | pasting a table into the assistant | 85% |
The row that matters is the second one. It is the only one that looks like real work: hundreds of documents, one question, and nobody to point at where the answer is. It is also where the gap between software is widest, and the one a single-file demo does not test.
How to read an AI tool demo
When a demo runs on a PDF uploaded in front of the client, it proves the model interprets a filing well. It says nothing about its ability to find that filing among five hundred others, which is what makes a data room hard.
Three questions separate tools better than the model's name. Does the demo run on a file chosen in advance, or on the whole base? What does the software around the model do: a single search, or an investigation that opens documents and filters by company and fiscal year? And when the information does not exist, does the tool say so? A fourth question completes them: does the vendor publish an accuracy measure on a public benchmark, with its conditions? Choosing a tool for its model means looking at the wrong variable.
For M&A, transaction services and fund teams, we describe our approach to financial due diligence: data room, accounts, spreadsheets, with a citation for every answer.
Our FinanceBench results
We ran FinanceBench in the conditions of the table's second row: 368 filings in a standard ATG knowledge base, each question asked on its own, without pointing at the right document. ATG answers 138 questions out of 150 correctly (92%), with an average response time of 10.5 seconds and 1.5 calls to the search tool per question on average. The protocol, every question, every answer and every verdict are published on our FinanceBench page. Grading conditions differ from one source to another (human grading at Patronus AI, automated judges elsewhere): these figures place the approaches, they do not rank them.
Frequently asked questions about agentic search in due diligence
What is agentic search?
Agentic search is software that drives information retrieval for a language model. Instead of running a single search and answering from the chunks it gets back, it chains searches, filters by company or period, opens and reads through documents, checks what it found and decides when to stop. On FinanceBench, this approach takes the same model from 26.7% to 86% correct answers, according to Mistral.
Does the choice of AI model still matter for analyzing financial documents?
Yes, but less than the search software. At Dewey, with the same software, Claude Opus 4.6 and GPT-5.4 are 21 points apart on FinanceBench. At Mistral, with the same model, changing the search method adds 56 to 59 points. The model remains decisive for other tasks, such as writing a long analysis or making sense of a poorly worded question.
Why does an AI make more mistakes on a data room than on a single PDF?
Because it first has to find the right document. On FinanceBench, GPT-4 Turbo answers 50% to 79% of questions correctly when given the right filing, and 19% when it has to find it in the whole base. Look-alike documents (successive fiscal years, versions, subsidiaries) widen the gap: a semantic similarity search easily confuses them.
How do you evaluate an AI tool for due diligence?
The most telling test asks real questions over a complete set of documents, without telling the tool where the answer is, and includes questions whose answer is not in the documents. You then look at answer accuracy, source citations, response time and the tool's ability to say it found nothing. A vendor that publishes results on a public benchmark such as FinanceBench makes comparison possible before any trial.
Switching models takes one line of configuration. The software that drives the model takes much longer to build: filters, navigation, picking the right fiscal year, refusing to answer without a source. In due diligence, that software makes the difference, and you see it at work on a real data room, not on a PDF picked in advance.
