RAG audit: fix or replace your internal AI chatbot

We see the same story in many of the IT departments we meet. A technical team built its own RAG in 2024 or 2025: an internal chatbot plugged into SharePoint or Confluence, a vector database, a frontier model's API. The demo was convincing, the tool went to production, and it disappoints: it misses the point, or answers too slowly for anyone to bother waiting. Gartner predicted that at least 30% of generative AI projects would be abandoned after proof of concept by the end of 2025. For the ones that survived, the next question is the RAG audit: where do these flaws come from, and what should you do about them?

The reflex is to switch models. That is rarely where the problem lies. We saw it on our own RAG: on the public FinanceBench benchmark, replacing Qwen3.8 27B, the model chosen for our benchmark, with Gemini 3.8 Flash made no significant difference, and neither did raising the model's thinking budget. A RAG hits its ceiling because of its architecture, not its LLM.

This article gives you a seven-layer audit grid with tests you can run this week on your own tool, a way to measure, and criteria to decide whether to improve it or replace it. The tests come from what we had to build to keep our own RAG accurate on more than 100,000 documents at several of our customers.

Which symptoms show that an internal RAG needs an audit?

Users never say "retrieval recall is poor". They say "it can't find it". Translating each complaint into a technical layer is half the audit. Scott Barnett and his co-authors identified seven failure points of a RAG system from three deployments. The table below builds on them and adds the ones we run into in companies: synchronization, access rights, operations.

The same authors draw a lesson that applies to any enterprise RAG: validation is only feasible during operation, and robustness evolves with use rather than being designed in at the start. A RAG shipped without measurement in production degrades without anyone noticing.

Why in-house RAG hits a ceiling: single-pass retrieval

Most RAG systems built from the 2023 and 2024 tutorials follow the same pattern. The question becomes a vector, the database returns the 5 or 10 closest passages, the model writes its answer from those passages. That works well for "what is our remote work policy?". It fails as soon as the question requires comparing documents, listing, counting or reading a long document.

FinanceBench puts numbers on it. This set of financial analyst questions, published by Patronus AI in late 2023, forces the system to find answers in hundreds of annual and quarterly reports. Results change completely depending on how the system searches:

Mistral's and ATG's figures are published by each vendor about its own product. We publish the protocol and the verdict for every question so anyone can check.

Switching LLMs does not move you from one row of the table to another. With the same model, Mistral goes from 22.7% to 78.7% by changing only how it searches. Our experience points the same way: on FinanceBench, Qwen3.8 27B and Gemini 3.8 Flash score with no significant gap, and giving the model a bigger thinking budget changes nothing either. For finding the right passages, the model carries little weight.

It does make a big difference elsewhere: drawing up a complex, multi-step search plan, writing a long and structured answer, or interpreting a poorly worded question. In an internal chatbot, where questions are rarely as precise as in a benchmark, those gaps show. If your RAG is built on single-pass retrieval, the audit will most likely conclude that you need a different architecture, not a different model. We cover the move from linear to agentic RAG in our guide.

Why a slow internal chatbot ends up abandoned

Your employees do not compare your assistant with the old intranet search engine. They compare it with ChatGPT, whose answer starts appearing almost at once, and with the AI Overviews Google shows at the top of its results. Next to them, an internal chatbot that shows a spinner for twenty seconds loses, even when its answer is better. Nobody comes to complain: users go back to searching SharePoint or ask the colleague next to them, and the usage curve goes down. An accurate but slow assistant ends up ignored. An audit that only measures answer accuracy misses it.

The thresholds have been known in usability for a long time. Jakob Nielsen sums them up: one second is the limit for the user's flow of thought to stay uninterrupted, ten seconds the limit for keeping their attention. Beyond that, you need to show progress. A RAG that chains query embedding, retrieval, reranking and writing without showing anything easily goes past these limits.

Agentic search, which adds 56 points on FinanceBench, makes the problem sharper: several searches take longer than one. That does not mean choosing between accuracy and speed. Three levers give you both:

• keep long reasoning for the questions that need it, and answer simple questions right away;
• show the work as it happens: searches launched, documents consulted, then the text appearing as it is generated;
• run steps in parallel and give each one a time limit, so that one slow component does not hold back the whole answer.

At ATG, a small model classifies each question in 300 milliseconds to route it to a fast model or a reasoning model. The question's embedding is requested from two providers at once, and the faster one wins. Searches run in parallel with an 8-second budget, reranking gets 2, and a model provider that has not started answering after 5 seconds is replaced by another. Meanwhile, the user sees each tool call and its duration, then the answer appearing word by word. On FinanceBench, where an analyst question often needs several searches across hundreds of reports, the full answer arrives in 10.5 seconds on average.

How to audit a RAG: the seven-layer grid

Each layer below lists what to check and concrete tests. Most of them need no access to the code: a user account and a few documents whose content you know are enough. For each layer, we describe what ATG does, as a point of comparison.

1. Ingestion: what your RAG actually read

An answer can never be better than what was indexed. Scanned PDFs, tables spanning several pages, Excel files, diagrams: many errors start here, well before the model.

• Take a price table that runs over three pages and ask about a row on the last page. If the answer mixes up the columns, the chunk holding page 3 has lost the table header.
• Ask for a specific figure from an Excel file. Check that it comes from the cell, not from a summary a model wrote at indexing time.
• Upload a scanned PDF, then find it with a sentence it contains.
• Ask a question whose answer only appears in a diagram or a screenshot.

At ATG, when a table is split, each continuation repeats the header and caption: without that, the second half of a table is just a list of numbers without columns. Spreadsheets are read from the source at answer time, hidden sheets and computed formulas included. Image descriptions produced by a vision model are flagged as to-be-checked, because they are often wrong, and the agent can verify them against the original image.

2. Connectors and synchronization: what happens after six months

Connecting a source takes a day. Keeping it in sync for months, without losing or duplicating anything, is the hard part, and the least visible one in a demo.

• Rename or move a document in your document management system. How long does the assistant keep citing it under its old name or folder?
• Ask the team what happens if the source API returns an empty list because of an incident. If the answer is "we delete whatever is no longer listed", a one-minute incident can wipe out your knowledge base.
• Same question for a connector's service account whose password expires.
• Search the index for a "Please sign in" page or a captcha indexed instead of a website's real content.

At ATG, a drop of more than 30% in a source's document count blocks every deletion until a second full scan confirms it. An authentication failure stops the refresh instead of being read as "content deleted". The web crawler recognizes captchas, login pages and "access denied" pages, in French too.

3. Retrieval: finding the right passage among hundreds of thousands

• Search for a document by an exact reference: "NF-2024-117", a product code, a contract number. Vector search alone often fails here, because a reference carries no meaning that embeddings can capture. That is what hybrid search is for.
• Restrict the search to a small folder and compare the results with what you know is in it.
• Ask "how many procedures mention X?" or "list every contract signed in 2025". A RAG that returns 10 passages cannot count: it extrapolates from what it saw.
• Ask a question that requires reading a whole 400-page specification, for instance "list every security requirement".

At ATG, retrieval combines a lexical index (BM25), a vector index and a reranker that reorders the results. The second test took us a while to understand: on 341,000 chunks, a naive filtered search took 13.5 seconds, and forcing the approximate vector index (HNSW) found none of the 8 expected results when the filter covered 2.5% of the collection. The system now picks its method based on the filter's size: 369 milliseconds at the 90th percentile, with 99.8% recall. Approximate vector indexes lose results without telling you. Inventory questions go through deterministic SQL queries instead, with a check that every page of results was read before the list is declared complete.

4. Generation: what the model does with the passages it gets

• Ask a question you know the documents do not answer. The right behavior is "I can't find this information", not a plausible answer.
• Submit a document longer than the model's context window. Does the assistant say it did not read everything, or does it silently drop the end?
• For an assistant connected to a database, ask a two-part question. Does it flag that it only answered one part?
• Check that every answer cites its sources, with a link to the exact passage.

Professional tools get it wrong too. Stanford researchers found that Lexis+ AI and Thomson Reuters' tools hallucinate between 17% and 33% of the time on legal questions. If specialized vendors cannot avoid it, an unmeasured internal RAG has no reason to. At ATG, reading a whole document is capped, and beyond the cap the tool refuses and steers the agent towards passage-by-passage exploration instead of truncating silently. An answer like "I will go through the remaining pages" when nothing was done is detected and retried, then reported as a failure if the retry is not enough.

5. Access rights, security and sovereignty

A RAG puts the whole company's knowledge behind a single input box. It is also an attack surface.

• Remove a user's access to a document. Can they still find it in a past conversation with the assistant?
• Ask where the vectors are stored, and which providers receive document content at indexing time (image descriptions, tagging, translation), not only at answer time.
• Read the application logs: do they contain document excerpts or employees' questions?
• For an HR or legal assistant: who in IT can read the questions people ask?

At ATG, access rights are checked on every read, including for a table already extracted into a conversation. The provider policy (Europe or worldwide) is set for the whole organization and also applies to document processing, because a "sovereign" mode limited to the final answer misses most of the data flow. An agent can be marked private: its conversations become unreadable to administrators while statistics stay available. For more on hosting and applicable law, see our sovereign AI page.

6. Operations: speed, outages and load

• Time about ten real questions, simple and complex: how long before the first sign of life on screen, then how long before the full answer. Ask ChatGPT the same ones, since that is the comparison your users make.
• Say "hello" or ask a trivial question. If the answer takes as long as an analysis, every question goes through the same heavy pipeline.
• What do your users see when your only model provider is down or rate-limits you? We covered this risk in the mistakes that come after the POC.
• Upload 50 files at once and watch the processing times.
• On a phone, ask a long question, lock the screen for 30 seconds, come back. Is the answer there?
• If an indexing job crashes halfway, do you know which documents are up to date, half-processed or missing?

The 50-file test comes from a real incident on our side: 56 files uploaded at the same time had triggered 56 full recomputations that queued for two hours. Those jobs are now batched. At ATG, each task has several model providers with automatic failover, and an answer keeps generating even if the browser connection drops.

7. Measurement and continuous improvement

• After your last model or prompt change, how did you know the assistant was not answering worse than before?
• What happens to users' negative feedback? Is it read, sorted by cause, followed through to a fix?
• If two versions of a procedure coexist in your sources, which one does the assistant cite?
• When a business answer is wrong, who fixes the instructions: the domain expert, or an IT ticket?

At ATG, every negative rating becomes a task sorted by cause: agent instructions, documentation, retrieval, source, bug. Business owners edit and test their own agent's instructions before publishing. The platform also detects contradictions between documents, a frequent cause of wrong answers while the RAG works technically. Without this loop, quality drops as documents change.

How to evaluate a RAG: test set and LLM judge

Fixing without measuring just moves the errors around. Before touching the system, you need a way to know whether it answers better or worse than yesterday.

A test set built from real questions

Collect 100 to 300 questions from usage logs, each with its correct answer and the document that contains it. Mix the types: simple questions, exact references, comparisons across documents, counts, old documents, and out-of-scope questions whose correct answer is "I don't know". Barnett and his co-authors note that such application-specific question-answer pairs are often missing: building them is the first deliverable of an audit, and an asset you keep whatever happens next. Replay the set after every change of model, prompt or document processing.

The LLM judge

Grading hundreds of answers by hand every week is not realistic. An LLM judge compares each answer with the expected one and scores it. The method holds up: in the study by Zheng and his co-authors, GPT-4 agreed with human preferences more than 80% of the time, as often as two human raters agree with each other. The same paper also describes its biases: a preference for the first answer shown, for longer answers, for its own answers. Have a sample reviewed by hand. On FinanceBench, our verdicts are graded by an LLM judge, then reviewed by a human.

What a score does not show

A test set only sees the questions it contains. A document that was never indexed, a source wiped by a synchronization incident, a badly inherited access right or a provider outage go unnoticed as long as no question touches them, and inventory questions are often missing from it. A test set evaluates the answer chain, an audit evaluates the whole system.

Improve or replace your RAG: how to decide

An audit ends in one of these situations:

The last case is real. When the RAG is your product, building it is your job. In the other cases, the prototype is cheap. What costs money is everything around it, and that never stops. Models change with every release, source APIs evolve, documents contradict each other. MIT's report on generative AI in business gives an order of magnitude: solutions bought from specialized vendors succeed about 67% of the time, internal builds only one-third as often. The report's methodology is debated, but the gap matches what we see in the field.

What to keep if you replace it

The engine can be replaced. Your knowledge of how people use it cannot. Before switching, save:

• the question logs, which show what users are really looking for;
• the test set and its expected answers;
• the business vocabulary: acronyms, product names, legacy names;
• the business instructions and the list of authoritative sources.

You can also replace the engine and keep the rest. A solution like ATG can run behind your existing interface, through an API or a widget embedded in the portal your teams already use.

Counterpoints to keep in mind

A vendor creates a dependency. Check reversibility (export of data, conversations and settings), where the service is hosted and which law applies to the parent company. Benchmarks a vendor publishes about itself, ours included, are claims: ask to test on your own documents, with your own test set.

What ATG offers for a RAG that disappoints

Two ways in, depending on where you are.

Have your RAG audited, even if we did not build it. Our audit combines interviews with the designers and users, a review of the design, tests on your documents with questions whose answers are known, and analysis of real usage logs when they exist. Allow a few days to a few weeks depending on scope. You get a written diagnosis and recommendations ranked by impact, from quick settings to a change of approach. If a third-party tool fits your need better than our platform, we tell you.

Replace the engine with a solution that already runs. The mechanisms described in this grid exist in ATG's RAG: connectors to your tools, inherited access rights, hybrid and agentic retrieval, routing between fast and reasoning models, answers displayed as they are written, citations, evaluation campaigns, feedback tracking. Hosted in Europe or on your own servers. On the projects we have delivered, total cost of ownership drops by about 80% compared with an internal build.

Frequently asked questions about auditing a RAG

How do you audit an enterprise RAG?

By translating each user complaint into a technical layer, then testing seven layers: document ingestion, connectors and synchronization, retrieval, generation, access rights and security, operations, measurement. Most tests only need a user account: a three-page table, an exact reference, a counting question, an out-of-scope question. You then add a set of questions with known answers, replayed and graded.

How do you improve a RAG that gives bad answers?

Measure first, with a set of 100 to 300 real questions, then locate the fault. If the right passage is not retrieved, work on ingestion or retrieval (hybrid search, reranker, multiple searches). If the passage is retrieved but misused, work on generation. Switching models rarely helps find the right passages: in our FinanceBench tests, moving from Qwen3.8 27B to Gemini 3.8 Flash did not change the score significantly, while Mistral goes from 22.7% to 78.7% correct answers just by changing how it searches.

How do you evaluate the quality of a RAG?

With a test set drawn from real questions, each with its correct answer and source document, graded by an LLM judge and partly reviewed by a human. That score does not replace an audit, which also checks indexing, source synchronization and access rights.

Why can't my RAG find a document that exists?

Three causes come up most often. The document was badly extracted or never indexed (scanned PDF, silent failure). The right passage is retrieved but ranked below the 5 or 10 results passed to the model. Or the question contains an exact reference, a code or a proper name that vector search alone handles poorly: you then need hybrid search that combines exact words and meaning.

What response time should an internal chatbot aim for?

A first sign of life on screen in under a second, and the start of the answer within a few seconds for a simple question: that is what ChatGPT and Google's AI Overviews have taught users to expect. A question that needs several searches can take longer, as long as the tool shows what it is doing. In usability, ten seconds without feedback is the limit beyond which you lose the user's attention. For comparison, ATG answers FinanceBench's financial analyst questions in 10.5 seconds on average.

Should you switch LLMs to improve a RAG?

Rarely as a first step. Most disappointments come from what the model receives, not from what it does with it. On FinanceBench, GPT-4 Turbo answers 19% of questions correctly with a simple search and 85% when it is given the right page. A stronger model helps after that, mostly for planning complex searches, writing long answers and understanding poorly worded questions.

How long does a RAG audit take?

From a few days to a few weeks depending on scope: number of sources, access to technical documentation and usage logs, whether a test set already exists. Building that test set often takes a good share of the time, and it stays useful whatever you decide next.

Should you replace an internally developed chatbot?

Not necessarily. If the audit shows localized causes, fix them. If the architecture runs a single search per question, if connectors break regularly or if nobody measures quality, replacing the engine often costs less than rebuilding it. You then keep the logs, the test set and the business vocabulary, and you can keep the existing interface.

---

An internal RAG that disappoints was most often built for the demo, then shipped without the layers that keep it accurate and fast in production. The audit tells you which ones are missing. What remains is an economic decision: build them yourself, or start from a base that already has them. To talk about your case, book a call.