WEBVTT

1
00:00:08.570 --> 00:00:12.238
There's been a lot of talk about
context engineering lately,

2
00:00:12.238 --> 00:00:15.753
giving AI access to your documents
and information.

3
00:00:16.110 --> 00:00:20.561
But what does it actually mean
when an AI system finds a document?

4
00:00:20.918 --> 00:00:25.198
To judge whether one AI tool finds
documents better than another,

5
00:00:25.198 --> 00:00:27.883
you first need to understand how it works.

6
00:00:29.103 --> 00:00:33.356
When an AI assistant searches
your SharePoint or your knowledge base,

7
00:00:33.356 --> 00:00:36.435
the large language model
is not doing the searching.

8
00:00:36.793 --> 00:00:38.147
It can't.

9
00:00:38.503 --> 00:00:42.294
An LLM generates text
from the context it's given;

10
00:00:42.294 --> 00:00:45.794
it doesn't reach into other
systems and pull documents.

11
00:00:46.631 --> 00:00:49.425
That work is done by a retriever.

12
00:00:49.782 --> 00:00:51.861
It's a separate piece of software,

13
00:00:51.861 --> 00:00:56.236
a separate service with its own logic,
and its whole job is to take a query

14
00:00:56.236 --> 00:00:59.193
and return the most
relevant pieces of content.

15
00:00:59.550 --> 00:01:04.355
The LLM only enters the picture
once the retriever has done its job.

16
00:01:05.192 --> 00:01:09.444
So when two AI tools give different
answers from the same data,

17
00:01:09.444 --> 00:01:12.442
your first instinct might
be to blame the model.

18
00:01:12.799 --> 00:01:15.906
Almost always, it's the retriever.

19
00:01:17.126 --> 00:01:20.450
There are two main ways
a retriever searches.

20
00:01:20.807 --> 00:01:23.547
The oldest is keyword search.

21
00:01:23.904 --> 00:01:27.314
It builds an index of every
word in every document,

22
00:01:27.314 --> 00:01:30.942
a bit like the index at the back
of a book, and returns the documents

23
00:01:30.942 --> 00:01:33.138
that contain the words in your query.

24
00:01:33.975 --> 00:01:37.258
The limitation follows from the mechanism.

25
00:01:37.615 --> 00:01:43.430
Ask how we handle major incidents, and if
the document says P1 escalation procedure,

26
00:01:43.430 --> 00:01:46.048
keyword search finds nothing useful.

27
00:01:46.404 --> 00:01:50.408
The concept is the same,
but the words are different.

28
00:01:51.245 --> 00:01:53.876
Semantic search solves that.

29
00:01:54.233 --> 00:01:59.246
A separate model, called an embedding
model, turns text into a list of numbers,

30
00:01:59.246 --> 00:02:03.480
a vector, so that text with similar
meaning ends up close together.

31
00:02:03.837 --> 00:02:06.799
Your query is turned into a vector too,

32
00:02:06.799 --> 00:02:10.001
and the retriever returns
whatever sits nearest.

33
00:02:10.358 --> 00:02:14.497
And again, the embedding
model is not the LLM.

34
00:02:15.717 --> 00:02:20.196
Before any of that can happen,
the documents have to be indexed.

35
00:02:20.553 --> 00:02:23.637
They're broken into smaller
pieces called chunks,

36
00:02:23.637 --> 00:02:25.960
each chunk is turned into a vector,

37
00:02:25.960 --> 00:02:28.931
and the vectors are stored
in a vector database.

38
00:02:29.768 --> 00:02:32.485
Each chunk only gets one vector,

39
00:02:32.485 --> 00:02:35.837
so that vector has to represent
everything in the chunk.

40
00:02:36.193 --> 00:02:41.528
A chunk focused on one procedure
or one policy sits cleanly in one place.

41
00:02:41.885 --> 00:02:45.798
A chunk that cuts a table in half,
or spans two topics,

42
00:02:45.798 --> 00:02:49.779
gets pulled in two directions, so it's
missed for some of the right queries

43
00:02:49.779 --> 00:02:51.772
and returned for the wrong ones.

44
00:02:52.609 --> 00:02:57.269
In other words, the chunking decision
shapes what the LLM receives,

45
00:02:57.269 --> 00:02:59.615
and so the quality of every answer.

46
00:03:00.835 --> 00:03:05.183
The strongest retrievers run keyword
and semantic search together,

47
00:03:05.183 --> 00:03:07.284
which is called hybrid search.

48
00:03:07.641 --> 00:03:12.967
Keyword search catches exact matches
like product codes and policy references,

49
00:03:12.967 --> 00:03:15.571
and semantic search catches the meaning.

50
00:03:16.408 --> 00:03:19.492
Then a reranker,
another separate component,

51
00:03:19.492 --> 00:03:23.065
scores the combined results
for how well they answer the question,

52
00:03:23.065 --> 00:03:25.737
and only the best of them go to the LLM.

53
00:03:26.957 --> 00:03:30.000
So when you're evaluating
an AI knowledge tool,

54
00:03:30.000 --> 00:03:33.080
the question is not which LLM it uses.

55
00:03:33.437 --> 00:03:38.432
It's what the retrieval layer looks like,
and how much control you have over it.

56
00:03:39.269 --> 00:03:42.443
Ask what chunking strategy is used.

57
00:03:42.800 --> 00:03:45.418
Ask whether search is hybrid.

58
00:03:45.775 --> 00:03:48.311
Ask whether there's a reranker.

59
00:03:48.667 --> 00:03:51.657
And if you're connecting
through a pre-built connector,

60
00:03:51.657 --> 00:03:55.293
you're accepting someone else's
decisions on all of those.

61
00:03:56.130 --> 00:04:01.633
Two tools can sit on the same SharePoint
site, use models from the same provider,

62
00:04:01.633 --> 00:04:03.761
and give very different answers.

63
00:04:04.118 --> 00:04:08.298
The model gets the blame,
but the retriever is the cause.
