How a page becomes a vector

Asking a knowledge vault a question means finding the handful of passages that answer it among several hundred pages which is not a trivial problem. The retrieval layer LibrisVault uses carries two different encodings of the same text, chosen because they fail in opposite directions. The retrieval design belongs to claude-obsidian, the MIT-licensed vault system by AgriciDaniel. What LibrisVault added is a measurement of which encoding its own vault actually needs, and the answer was not the textbook one.

Sparse: the page as a bag of words

BM25 represents a page as a vector over the vocabulary, one dimension per term in the collection. That translates to nearly all of them being zero for any single page which is what sparse means here. A dimension’s value grows with how often that term occurs in the page while it is damped for terms that are common everywhere. It is also normalized for page length so a long page cannot win by size alone.

Nothing about meaning is modelled which is its main strength. A query containing “ALC-0315” or a specific error code will surface the pages that literally contain those tokens, however rare they are. However, there is a downside with regard to synonyms: a query for “shelf life” will score nothing against a page that only ever writes “stability period” even though both terms describe the same concept.

Dense: the page as a point in meaning space

An embedding model maps a passage to a few hundred or few thousand numbers. Those are positioned in such a fashion that texts with similar meaning sit near one another. Similarity is the cosine of the angle between two such vectors.

While this finds the synonym the keyword search missed, it is weak precisely where BM25 is strong. An unusual identifier carries little meaning for the model, so it gets placed near superficially similar strings and an exact rare token can be smeared away.

A closer look at the hybrid approach

To get the best of both worlds, the textbook answer is to use BM25 to gather candidates followed by an embedding-based cosine re-rank to order them. Each mechanism covers the other’s blind spot, and on Anthropic’s benchmark the combination removes a large share of the failures that either method leaves behind on its own.

claude-obsidian ships exactly that pipeline: BM25 over contextualized chunks, then an optional cosine re-rank on local embeddings. What LibrisVault contributed was the measurement, 35 labelled questions against its own vault, 21 of them curated and 14 drawn blind at random so that the curated ones could not flatter the result. BM25 alone put the right page in the top five 97 percent of the time; with the dense re-rank on top it was 94 percent, and the top-1 hit rate fell from 69 to 54 percent. LibrisVault therefore runs the pipeline with the re-rank switched off by default, one setting away for a corpus that turns out to need it. That is not a refutation of the theory. A technical vault asks mostly for identifiers, error codes and named things, which is the case sparse retrieval was built for, and the dense model smeared exactly those. Thirty-five questions is a small sample, which is a reason to keep measuring rather than a reason to trust the textbook over the measurement.

We should also consider that chunking matters as much as the encodings do. A short passage lifted out of a page loses the context that made it interpretable, let’s look at an example and think of “it” and “the method”. Taken by themselves, both no longer refer to anything so each chunk is also stored with a short statement of where it came from. The underlying idea is set out in Anthropic’s contextual retrieval work which reports that two moves together, adding that context and combining both encodings, remove a large share of retrieval failures.

Historically, such an approach is well established, think of a library. The index at the back of a book finds an exact word, a librarian who deeply understands the question will know which books to check in the first place. Neither replaces the other, but how much each is needed depends on who walks in. A reading room whose visitors arrive with a book title keeps the index busy and the desk quiet, which is what the measurement above said about this vault.

How do we put the choices in the proper order? First work out what a miss costs, then pick the encoding. Exact identifiers and paraphrases are different failure modes, a system that has to survive both needs both representations, and a system that mostly meets one of them is allowed to notice.

Across a formulation, a molecule and a page, the same pattern holds true: a thing becomes a vector, and the choice of vector decides what “similar” is allowed to mean.