The video could not be loaded. Press Play to try again, or read the transcript below.
- With the player focused:
- Space play or pause
- C captions
- JL back or forward 10 seconds
- F full screen
The Search Retriever is not the LLM
The model gets the blame, but the retriever is the cause.
Transcript
There’s been a lot of talk about context engineering lately, giving AI access to your documents and information. But what does it actually mean when an AI system finds a document? To judge whether one AI tool finds documents better than another, you first need to understand how it works.
When an AI assistant searches your SharePoint or your knowledge base, the large language model is not doing the searching. It can’t. An LLM generates text from the context it’s given; it doesn’t reach into other systems and pull documents.
That work is done by a retriever. It’s a separate piece of software, a separate service with its own logic, and its whole job is to take a query and return the most relevant pieces of content. The LLM only enters the picture once the retriever has done its job.
So when two AI tools give different answers from the same data, your first instinct might be to blame the model. Almost always, it’s the retriever.
There are two main ways a retriever searches. The oldest is keyword search. It builds an index of every word in every document, a bit like the index at the back of a book, and returns the documents that contain the words in your query.
The limitation follows from the mechanism. Ask how we handle major incidents, and if the document says P1 escalation procedure, keyword search finds nothing useful. The concept is the same, but the words are different.
Semantic search solves that. A separate model, called an embedding model, turns text into a list of numbers, a vector, so that text with similar meaning ends up close together. Your query is turned into a vector too, and the retriever returns whatever sits nearest. And again, the embedding model is not the LLM.
Before any of that can happen, the documents have to be indexed. They’re broken into smaller pieces called chunks, each chunk is turned into a vector, and the vectors are stored in a vector database.
Each chunk only gets one vector, so that vector has to represent everything in the chunk. A chunk focused on one procedure or one policy sits cleanly in one place. A chunk that cuts a table in half, or spans two topics, gets pulled in two directions, so it’s missed for some of the right queries and returned for the wrong ones.
In other words, the chunking decision shapes what the LLM receives, and so the quality of every answer.
The strongest retrievers run keyword and semantic search together, which is called hybrid search. Keyword search catches exact matches like product codes and policy references, and semantic search catches the meaning.
Then a reranker, another separate component, scores the combined results for how well they answer the question, and only the best of them go to the LLM.
So when you’re evaluating an AI knowledge tool, the question is not which LLM it uses. It’s what the retrieval layer looks like, and how much control you have over it.
Ask what chunking strategy is used. Ask whether search is hybrid. Ask whether there’s a reranker. And if you’re connecting through a pre-built connector, you’re accepting someone else’s decisions on all of those.
Two tools can sit on the same SharePoint site, use models from the same provider, and give very different answers. The model gets the blame, but the retriever is the cause.
