RAG chatbot fix: 20/30 correct answers before, 29/30 after
A support chatbot over 12 synthetic company documents, tested on 30 questions with known answers. Same documents, same questions, same LLM. Only the pipeline changed. Each version was run twice, and both runs gave the same scores: 10 wrong answers became right, and 1 right answer became a miss.
Your RAG bot gives wrong answers? Send me 10 questions it gets wrong.
What was wrong, and the fix
The cause of each miss was found by reading the chunks the old bot got. All fixes went in together; the next section measures each one.
- Tables and sections cut in the middle (4 questions missed before, in both runs).
Fix: heading- and table-aware chunking: one chunk per section, tables never split, each chunk tagged with its document and section title. After the full fix: 4 of 4 right, in both runs. - Outdated policy retrieved next to the current one, with no dates (3 questions missed before, in both runs).
Fix: document metadata (effective date, current or superseded): outdated documents rank below current ones and are labelled in the prompt. After the full fix: 3 of 3 right, in both runs.
Where the dates come from: each of the 12 test documents has its effective date and status (current or replaced) in its header, written when the documents were made. On real documents this comes from file dates, version names, or a one-time review that I do in the first step. - The right document was not in the top 2 results (3 questions missed before, in both runs).
Fix: hybrid search (keywords + embeddings), a reranker, and 6 chunks instead of 2. After the full fix: 3 of 3 right, in both runs. - No sources and no “I don't know” rule in the prompt.
Fix: every fact must cite its document, and the bot must say “The documents don't say.” when the answer is missing. After: 29 of 30 answers cite a source (before: 0). To be fair, the old bot already declined all 5 questions the documents don't answer, so that score did not move.
Which fix mattered (measured)
A fair question: did the fixes matter, or did the bot just get more text? To check, I changed one thing at a time and ran each version twice on the same 30 questions.
| Version | Correct of 30 (run 1 / run 2) | Outdated or wrong facts given as valid | Answers citing a source |
|---|---|---|---|
| Old bot | 20 / 20 | 3 / 3 | 0 / 0 |
| Old bot, 6 chunks instead of 2 | 25 / 25 | 2 / 2 | 0 / 0 |
| Fixed bot | 29 / 29 | 0 / 0 | 29 / 29 |
| Fixed bot, but the old 500-character chunks | 27 / 27 | 0 / 0 | 28 / 28 |
| Fixed bot, but no dates or status | 29 / 27 | 0 / 2 | 29 / 28 |
| Fixed bot, but no reranker | 28 / 28 | 0 / 0 | 29 / 30 |
| Fixed bot, but the old prompt | 30 / 30 | 0 / 0 | 0 / 0 |
- More text helps a lot. With 6 chunks instead of 2, the old bot went from 20 to 25. But it still gave the outdated refund rule as valid (2 answers per run).
- Whole tables matter for math. With the old chunks, the fixed bot fell to 27: the price row it needed was not in its chunks.
- Dates and status keep the old policy out. Without them, the fixed bot scored 29 and 27, and in run 2 it gave the outdated refund rule as valid 2 times.
- The reranker added 1 question: 28 without it.
- The strict prompt did not add accuracy here. With the old prompt on the same chunks, the fixed bot scored 30/30, but cited no source. The strict prompt puts a source on every answer, so a person can check it. It also cost one question (below).
I did not change the fixed bot after these runs. Picking the best version from the test answers would be tuning on the test.
Three real questions, before and after
Table lookupWhat is the price per user of the Team plan when billed annually?
Based on the provided context, the pricing details for the Team plan are missing/cut off from the table, so the exact price cannot be determined from the text.
The Team plan costs $20 per user per month when billed annually [pricing-plans]. Annual billing is charged once a year for the whole term [pricing-plans]. Calculation: $20 × 12 months = $240 per user per year [pricing-plans].
Why it changed: The 500-character split cut the pricing table inside its first row (Starter), so the Team row never reached the model. Keeping tables whole put the full table in the context. Measured: the old bot with 6 chunks instead of 2 also got this one right, in both runs, so more chunks alone fix it too.
Current vs outdated policyIs there a fee when a refund is paid back for a bank transfer payment?
Based on the provided text, there are conflicting policies:
• One section states: There is no fee for refunds, including refunds of payments made by bank transfer.
• Another section states: Refunds of payments made by bank transfer carry a $25 processing fee.
No, there is no fee for refunds, including refunds of payments made by bank transfer [refund-policy]. Approved refunds are sent to the original payment method within 5 business days [refund-policy].
Why it changed: The old 2024 refund policy and the current one were both retrieved, with no dates, so the bot offered both rules. With document dates and status, the outdated policy ranks below the current one and is labelled as superseded. Measured: without the dates and status, the fixed bot gave the old $25 fee as a second valid rule in 1 of 2 runs.
Multi-hop (two documents)A Business customer with 10 users on monthly billing had 98.5% uptime last month. How much is the service credit in dollars?
Based on the provided context, the exact monthly price for the Business plan is missing (the plan comparison table cuts off after the Starter plan). Therefore, the exact dollar amount of the service credit cannot be determined.
However, based on the uptime table, an uptime of 98.5% qualifies for a 25% service credit (since it is below 99.0% but at least 95.0%) applied to that month's subscription fee.
The Business plan on monthly billing costs $39 per user per month [pricing-plans], making the monthly fee $390 for 10 users ($39 × 10 = $390). Because the Business uptime commitment is 99.9% [sla] and the monthly uptime of 98.5% is below 99.0% but at least 95.0%, the customer is eligible for a 25% service credit [sla].
Calculation: $390 × 0.25 = $97.50.
The service credit is $97.50, which is added to the account balance [sla].
Why it changed: The answer needs two documents: the price table and the SLA credit table. Before, the price table was cut off; after, both whole tables were retrieved and the bot showed the calculation. Measured: with the old 500-character chunks, the fixed bot still missed it in both runs.
Scores by question type
10 questions fixed, 1 got worse (M2, below). Net: 20 → 29 in both runs.
| Before, run 1 | Before, run 2 | After, run 1 | After, run 2 | |
|---|---|---|---|---|
| Correct answers | 20/30 | 20/30 | 29/30 | 29/30 |
| Numbers inside tables | 5/7 | 5/7 | 7/7 | 7/7 |
| Current vs outdated policy | 3/6 | 3/6 | 6/6 | 6/6 |
| Answers needing two documents | 2/6 | 2/6 | 5/6 | 5/6 |
| Exact numbers in text | 5/6 | 5/6 | 6/6 | 6/6 |
| Not in the documents (should decline) | 5/5 | 5/5 | 5/5 | 5/5 |
| Outdated or wrong facts given as valid | 3 | 3 | 0 | 0 |
| Said “not in the documents” when it was | 4 | 5 | 1 | 0 |
| Right document(s) found (25 answerable questions) | 20/25 | 20/25 | 25/25 | 25/25 |
| Answers citing a source | 0/30 | 0/30 | 29/30 | 29/30 |
| Median time per answer | 2.0 s | 2.0 s | 2.1 s | 2.0 s |
| LLM cost per answer (mean) | $0.0005 | $0.0005 | $0.0009 | $0.0009 |
One question got worse
“Which plans have an uptime SLA, and what is the API rate limit per minute on the cheapest of them?” The old bot got it right in both runs: Business, 1,000 requests per minute. The fixed bot missed it in both runs.
Both bots had the same two tables (uptime and API limits) and no prices. The fixed bot had no price table in its 6 chunks, and its rule is “never guess”, so it would not say which plan is cheaper. When the price table is in its chunks (question T7), it does answer. The old prompt guessed and happened to be right: with the old prompt on the same chunks, the fixed bot was also right in both runs.
No document says Business costs less than Enterprise (Enterprise is a custom quote), so the refusal is defensible. I still count it as a miss: 10 old misses fixed, 1 old pass lost, 20 → 29.
How it was tested
- 30 questions with a known answer and source, written and saved in git before the first run: numbers inside tables, current vs outdated policy, answers needing two documents, exact numbers, and 5 questions the documents do not answer.
- Grading: exact checks on key numbers and phrases first, then an LLM judge with a written rubric for the mixed cases. Then a second AI review, separate from the grader, checked 21 graded answers one by one against the documents: the grade (right or wrong) held for 21 of 21. For the one-change versions it checked 30 more: 30 of 30 held.
- Before = a typical first build: fixed 500-character chunks, embedding search with the top 2 chunks, no dates, a plain prompt. After = section-based chunks with whole tables, keyword + embedding search, a reranker, document dates, and a prompt that must cite and may say “The documents don't say.”
- Both use the same LLM (Gemini 3.8 Flash) and the same local embedding model. The whole demo, including the 4 main test runs, the 10 one-change runs and grading, cost $0.76 in LLM calls.
Your RAG bot gives wrong answers? Send me 10 questions it gets wrong.
Synthetic data: the company (Orbitask) and its 12 documents are made up for this demo, with the usual traps (tables, an outdated policy, similar wording, facts split across documents). Real documents will change the numbers. Python, BM25, sentence-transformers (all-MiniLM-L6-v2 and an ms-marco cross-encoder reranker), OpenRouter.