Clause
Ask a plain-English question about an Indian health insurance policy and get a sentence-level answer with the exact source clause highlighted in the original PDF.
- Role
- Design and implementation
- When
- 2026
- Stack
- Python
- FastAPI
- pgvector
- Docling
- Next.js
Insurance policies are answerable documents that are hostile to read. A benefit sits in one section, its waiting period in another, its sub-limit in a table, and an exclusion three sections further on that quietly overrides both. A rider can replace any of it.
Producing an answer is the easy half. Producing an answer somebody can check, against the clause it came from, is the half that matters, and knowing when the policy simply does not say is harder still.
A policy PDF is parsed with Docling, chunked structure-aware, embedded, searched with hybrid dense and lexical retrieval, reranked, and then either answered with its source clause or refused.
Parse before you chunk
Docling parses each policy into structure rather than a flat string, and chunking is structure-aware, so a sub-limit table survives as a table instead of being sliced through the middle of a row.
The corpus is 19 policy PDFs from 16 insurers, about 460 pages.
Hybrid retrieval, then rerank
Dense search with BAAI/bge-large-en-v1.5 and lexical search run side by side and are fused with reciprocal rank fusion, then reranked with bge-reranker-v2-m3. Dense retrieval alone misses exact policy wording; lexical alone misses paraphrase. Fusing them and reranking recovers both.
Decompose the multi-hop questions
A cheap model splits a question that spans several clauses into parts that can each be retrieved for, before an expensive model is allowed to answer. Most real questions about a policy are multi-hop whether or not they look it.
Answer with the clause attached
The answer is returned at sentence level with its source clause highlighted in the original PDF, so the reader checks the policy rather than trusting the model. When the policy does not address the question, the correct output is a refusal.
Measured, not asserted
A 120-question hand-labelled test set, an ablation across six configurations showing what each pipeline stage actually contributes, and explicit refusal accounting. Recall@k, MRR, answer correctness, citation precision, hallucination rate and refusal accuracy are all tracked, with the failure cases written up rather than hidden.
The rule that shaped the whole system: never state a benefit without its waiting period, sub-limit or exclusion. A technically correct answer that omits the waiting period is worse than no answer, because the reader acts on it. That turns a retrieval problem into a completeness problem, and completeness is much harder to score.
Exclusions that point at other sections defeat naive chunking. The clause you retrieve is correct and the answer built from it is still wrong, because the thing that overrides it lives somewhere the retriever never looked.
Refusal is a feature that fights every instinct in the stack. A model asked a question will answer it, and an evaluation that only measures correctness on answerable questions rewards exactly that. Refusal accuracy had to be measured separately or it would have been optimised away.
The lexical arm uses Postgres ts_rank_cd, which is not BM25, and it is labelled lexical search in the write-up for that reason. Swapping in real BM25, through ParadeDB's pg_search or the bm25s library, is roughly a day of work and would earn an extra row in the ablation. Mislabelling it would have been easier and dishonest.
The ablation exists to show what each stage contributes, which means it also shows which stages are not paying for their latency. The next version should cut on that evidence rather than keep every stage because it is already built.