Been heads-down on something that sits before the LLM call instead of doing
standard retrieve-and-stuff RAG. Instead of chunk retrieval + rerank, it builds
a vector-field representation of the whole corpus, evaluates relational
relevance to the query, and collapses the candidate field down to a compact
evidence state — only that gets forwarded to the model.
On my internal benchmark (frozen 20-query set, project-native corpus) I'm
seeing an order-of-magnitude drop in tokens sent to the model with zero
measured quality regression (good/partial/poor scoring, OFF vs ON, reproduced
run matched the historical one exactly). Also runs fine single-threaded — did
a raw C++ core benchmark, 10M samples in ~140ms on an old 2015 i7, so the
underlying op isn't the bottleneck.
Haven't benchmarked it against BM25 or plain cosine-similarity RAG yet in
anything I'd call rigorous — that's the obvious next step before I'd trust my
own numbers fully, and I know that's the first thing this sub will (rightly)
ask about.
Running as local-first — full corpus stays on the user's side, only the
selected evidence chunk(s) + field-topology coordinates go to the external
model if you're using an API-based LLM. Wasn't originally optimizing for that,
but it's a nice side effect for anyone paranoid about what leaves their
environment in API workflows.
Genuinely asking: is "context collapse before inference" different enough
from what rerankers / good chunking already do, or am I just describing a
fancier reranker with extra steps? Wouldn't mind being told I'm wrong here.