Stop Paying For Retrieval Latency On Chunks You Never Use In The Prompt

Stop Paying For Retrieval Latency On Chunks You Never Use In The Prompt



Tópico: Stop Paying For Retrieval Latency On Chunks You Never Use In The Prompt
Categoria: Tutoriais | Programação & Tecnologia
Idioma Principal: Português (Conteúdo de Tecnologia)

Descrição do Conteúdo / Informações:
-------------------------------------------------------------------------


Your pipeline fetched 10 chunks. Your LLM saw 3.


You set TOP_K=10 on your vector store. Ten candidate chunks means more signal for the model — that's the logic. Then you run npx ragscope and the audit prints:

WARN   51/100  █████░░░░░  my-rag-service
│  "what are the pricing tiers?"
│
│  ✗  precision     30  ███░░░░░░░  3/10 chunks used
│  ✗  efficiency    30  ███░░░░░░░  70% tokens wasted
│  ✓  uniqueness   100  ██████████  chunks are distinct
│  ✓  coverage     100  ██████████  all chunks scored
│
│  → Reduce TOP_K 10→3 (only 3 chunks reached LLM)
│  → 70% of retrieved tokens never reached the LLM
│

Three of ten chunks made it into the final prompt. You paid for ten round-trips to the vector store and seven went nowhere. The other seven were fetched, scored, and discarded somewhere between your retrieval step and your prompt assembly code.

This is the gap ragscope (npm) was built to surface.



Where the Chunks Disappear


Retrieval and prompt assembly are separate steps, but most teams treat them as one. The gap is where the waste lives.

A typical RAG pipeline:

• Embed the query

• Fetch TOP_K chunks from the vector store

•
(Optional) Rerank

• Assemble the prompt — filter by score threshold, truncate to fit the context window

• Send to the LLM

Step 4 is where chunks disappear. Post-retrieval filtering — a score threshold, a hard token budget, a deduplication pass — silently drops chunks you already paid to retrieve. If your prompt assembly filters out anything below a certain confidence score and your vector store returns six chunks that don't clear it, those six fetches were wasted. The network round-trip still happened. The latency still accumulated.

The problem compounds over time. TOP_K=10 gets set as a safe default, the pipeline ships, and the setting never gets revisited. LLM eval scores look fine because the three chunks that do reach the prompt are the right ones. The waste is invisible in your evals — it only shows up in latency and cost.

Vector stores typically scale retrieval latency with TOP_K. Fetching ten results takes measurably longer than fetching three, especially at tail latencies. When seven of those ten are discarded before the prompt, you're paying that latency premium on every query for nothing.



How ragscope Measures precision


ragscope runs as a local server (default port 4321) and receives spans from your pipeline via OTLP at http://localhost:4321/v1/traces. No changes to your RAG code needed — configure your OpenTelemetry exporter to point there.

When both a retrieval span and an LLM span arrive for the same trace, ragscope compares them. The key field is inContext on each RagChunk. It inspects the full text of the LLM span's assembled prompt and checks whether each retrieved chunk's content appears in it — positionally, by string match. A chunk either appears in the prompt or it doesn't. No LLM, no heuristics, no sampling.

precision starts as:

base = (chunks where inContext is true / total retrieved chunks) × 100

There's one additional penalty: if high-retrieval-rank chunks land in the middle of a long context window — the zone where LLM attention typically falls off — ragscope subtracts 12 points per buried chunk, capped at 36. This is the lost-in-the-middle detection. A pipeline where 3/10 chunks are used but two of those three are buried mid-context will score lower than one where the same 3 chunks sit at the prompt edges.

When the base falls below 60, ragscope generates a specific recommendation with the exact numbers: Reduce TOP_K 10→3 (only 3 chunks reached LLM). It prints this directly below the score bars, in the terminal, at the time the trace arrives — not in a cloud dashboard after a deploy.

efficiency is a related metric: the fraction of retrieved tokens that reached the LLM. If precision is 30 and your chunks are roughly uniform in size, efficiency will also be around 30 — meaning 70% of the tokens you transferred from the vector store never appeared in a prompt. That shows up in retrieval latency and, depending on your pipeline, in processing time downstream.



Tuning TOP_K Based on the Audit


Once you have precision data, tuning is mechanical.

Follow the recommendation. ragscope prints Reduce TOP_K 10→3 because Math.max(used_chunks, 3) gives you the minimum viable retrieval count with a floor of 3. Start there and re-run the audit against a few real queries.

Check efficiency alongside. Low efficiency paired with low precision means your unused chunks are large — a chunking strategy problem as much as a TOP_K problem. If efficiency is high but precision is low, your unused chunks are small and the token cost is modest; fixing TOP_K will mostly recover the latency.

Check uniqueness too. If the chunks that do reach your prompt include near-duplicates, the uniqueness metric will flag them. Two near-duplicate chunks in the prompt means one is redundant regardless of your TOP_K setting — that's a deduplication-at-ingest problem, not a retrieval count problem. ragscope computes overlap between chunk pairs and surfaces high-overlap counts in the audit output.

The typical path after a low-precision audit:

• Audit shows 3/10 chunks used — follow the Reduce TOP_K 10→3 recommendation

• Re-run against real queries — precision should move above 75 (PASS)

• If uniqueness also flagged duplicates, fix chunking and re-run

• If efficiency is still low after fixing TOP_K, chunks may be too large for the token budget

Going from TOP_K=10 to TOP_K=3 on a pipeline that was only ever using 3 chunks means 70% fewer vector store round-trips on every query. No model changes, no prompt rewriting, no reranker added.



The Number You Should Check First


The assumption that more retrieval means better answers is almost never validated against what the model actually sees. TOP_K defaults get set once and forgotten. Eval scores stay flat because the chunks that do reach the LLM are the right ones — the waste doesn't affect quality, just cost and latency.

npx ragscope gives you precision, efficiency, and uniqueness in the time it takes to run a dev command. If you haven't checked what fraction of your retrieved chunks survive to the prompt, that's the number to look at first.

GitHub · npm



Key Takeaways


•
precision measures what fraction of retrieved chunks appear in the final LLM prompt — anything below 60/100 triggers a Reduce TOP_K recommendation with exact numbers

• Vector store round-trips typically scale with TOP_K; fetching chunks you discard is pure overhead with no quality benefit

• ragscope detects in-prompt chunk presence by matching content against the OTLP LLM span's prompt text — deterministic, no LLM needed

• Low efficiency paired with low precision points to a chunking strategy problem, not just a TOP_K problem

• Low uniqueness means near-duplicate chunks in the prompt — fix at ingest, not by adjusting retrieval count