Bad RAG answers usually start with bad parsing
When a RAG system answers badly, most teams rewrite the prompt. The cause is usually earlier: the document never parsed correctly, and no prompt recovers from that.
AI Tools Hub Editorial · published · updated
First, read the parser output
Before touching retrieval or prompts, dump what the parser produced and read it. This finds half the problems immediately. The common four:
- Multi-column layouts interleaved line by line into nonsense
- Tables flattened to a single string, losing the column relationships
- Headers and footers repeated on every page, polluting retrieval
- Scanned PDFs that yielded no text at all
The last one is the most dangerous because nothing errors. The document was ingested, the pipeline succeeded, and the content is simply absent — so retrieval quietly returns a different, plausible document instead.
Chunking by document type
| Document | Split by | Why |
|---|---|---|
| Manuals and policies | Heading | One procedure lands in one chunk |
| Transcripts | Fixed length with overlap | No structure to split on |
| FAQs | One question and answer | Already the retrieval unit |
| Contracts | Clause | Clauses are self-contained |
| Table-heavy documents | Whole table plus heading | Splitting a table destroys the columns |
Attach metadata now, not later
Document name, section, last updated and visibility scope should go on every chunk at ingest. Filtering to current policy only, or to the reader's own department, then becomes a query parameter rather than a reindex.
Visibility especially. Retrofitting permissions means rebuilding the index, and until you do, the system can surface things people should not see.
The answers go stale
The most common operational failure is that the source document changed and the index did not. Six months later the assistant is confidently wrong. No product solves this; someone has to own re-ingestion as part of their job, or at minimum a quarterly review has to be in the calendar.
Order of operations
- Read the parser output
- Change or tune the parser
- Match chunking to document structure
- Add metadata for filtering
- Only then tune retrieval and prompts
Teams usually start at the last step. With problems in the first four, nothing there produces a measurable improvement, and the effort disappears.
FAQ
What chunk size should I use?
There is no universal answer; it follows the document structure. If you must split by length, start around 400–800 characters for Japanese or 1,000–1,500 for English and adjust by looking at retrieval results.
Should I change the embedding model?
Fix preprocessing first. Changing the model forces a reindex, which is expensive to do before you know whether it helps.