Embeddings and Vector Search
Meaning as geometry: embeddings, similarity search, vector stores and nearest-neighbour reality checks.
Before this: LLM Fundamentals
Objective
Turn semantic similarity into computable distance: embed text into vectors, compare with cosine similarity, retrieve nearest neighbors through a store - while learning how chunking choices and model mismatches quietly destroy result quality.
Vocabulary
The words to use when directing your agent.
- embedding
- A list of numbers representing a text's meaning, positioned so similar texts sit close together.
- cosine similarity
- The standard closeness score between vectors - higher means semantically nearer.
- vector store
- Storage built for fast nearest-neighbor lookup across millions of vectors.
- nearest neighbor
- The stored vectors closest to a query vector - the raw results of semantic search.
- chunk
- A piece of a document embedded as one unit - too big dilutes meaning, too small loses context.
Understand it
Chunking decides what meaning survives
Documents must be split before embedding. Chunks too large average away their own point - a page-long chunk about many topics belongs nowhere precisely. Chunks too small lose referents - 'it was approved' embeds meaninglessly without its sentence. Practical defaults exist (paragraph-to-section sized, slight overlaps), but the honest method is empirical: embed YOUR data, query YOUR questions, inspect misses, adjust.
One space, one model - forever per index
Embeddings from different models occupy incompatible spaces: distances between mixed-model vectors are noise dressed as scores. An index must be built and queried with the SAME model, and switching models means re-embedding everything. Record model identity alongside stored vectors - future-you must know which space the numbers live in.
Core concepts
Similarity becomes geometry
Embedding maps text to a point in high-dimensional space where distances track meaning: 'coffee with client' lands near 'business brunch', far from 'server rack'. Search stops requiring keyword overlap - you query by MEANING. This single trick powers recommendations, dedupe, semantic filters and every RAG system you will meet next module.
Nearest neighbor is not truth
Vector search returns mathematical proximity, not correctness: irrelevant-but-lexically-nearby items sneak in, and quality hinges on decisions made upstream - chunk boundaries, embedding model choice, score thresholds. Retrieval gives candidates; judgment layers still needed. Garbage chunks in, confidently-ranked garbage out.
Examples
Search by meaning, not keywords
'food last week' should find 'Lunch at Sakura Wednesday' despite sharing zero significant keywords. Embedded, their meanings sit close enough to surface together - keyword search structurally cannot make this match, semantic search finds it trivially. That gap is the entire value proposition.
The threshold reality
Query a small corpus and top hits score suspiciously similar regardless of question - because SOMEthing is always nearest. Raw nearest-neighbor answers need score context: meaningful cutoffs calibrated on your data, or UI language admitting 'best guess' status when scores run weak.
Common mistakes
Common mistake
Mixing embedding models within one index after a mid-project switch.
Fix
Stamp model identity on every stored vector; refuse queries whose query-model differs from index-model. Rebuild explicitly when migrating.
Common mistake
Copying someone else's chunk size without checking retrieval on your actual content.
Fix
Run retrieval evaluations on YOUR corpus: known questions, expected documents, measure hit rates. Tune chunks against observed misses.
Working with your AI agent
Require an inspection tool from your agent early: given a query, show top-k with scores AND the source text of each hit - eyeballing misses teaches more in five minutes than theory in five days. Ask for score-distribution output across your corpus to calibrate what 'high' versus 'weak' similarity actually looks like HERE.
Build it
The module’s real project. Your agent builds; you direct and inspect.
Project: Meaning Finder
Build Meaning Finder: semantic search over the Ledger's own data - embed expense notes and categories through the adapter layer (mock vectors offline), cosine-rank results with visible scores and source text, configurable chunking for longer notes, and a small retrieval eval proving known queries hit expected records.
Requirements
- Corpus embedded through the adapter pattern - deterministic mock vectors offline, real embeddings via env-var proxy optional
- Search endpoint/CLI returning top-k with cosine scores AND human-readable source text
- Model-identity stamped on stored vectors; mismatched query-model refused loudly
- Chunking configuration exposed (size, overlap) for multi-sentence notes
- Retrieval eval: curated set of at least eight known-query-to-expected-record pairs, hit-rate reported
- Score distributions printed to help calibrate what weak versus strong similarity looks like here
Context for your agent
Build semantic search over my Ledger's expense data. Embed notes through my existing adapter layer - deterministic MOCK embedding mode by default so everything runs offline; real provider stays env-var proxied. Cosine ranking with scores and source snippets shown; stamp model identity into stored vectors and refuse cross-model queries. Expose chunk-size/overlap config. Build the retrieval-eval runner with me curating eight-plus known pairs, reporting hit rate, and printing score distributions so we can calibrate thresholds honestly.
Verify it - twice
Check every item yourself first. Then hand your agent the prompt and compare verdicts.
Verification prompt - paste into your AI agent
Stress-test my Meaning Finder as a retrieval skeptic who trusts neither scores nor demos. Attempt each: (1) craft paraphrased queries around my eval pairs - where does hit-rate degrade and does my report admit it?, (2) hunt for cross-user leakage through semantically similar-but-foreign records, (3) feed a query under a DIFFERENT model id - confirm refusal, (4) inspect my weakest chunking decision by finding a record that SHOULD surface but does not - explain what the chunk boundary destroyed. Grade DoD items, then judge my score-calibration story: would a newcomer set thresholds correctly from my documentation alone?
Run this against “Embeddings and Vector Search” after your own checklist pass. Where the agent and you disagree, investigate before moving on.
Review questions
Answer out loud. If you cannot, revisit before moving on.
Show a query your semantic search nails that keyword search structurally cannot.
Why must one index live in exactly one embedding model's space?
What did score distribution inspection change about your thresholds?
You can now
- Implement semantic search over real data with inspectable scores
- Control chunking empirically instead of copying folklore
- Guard embedding-space integrity across models and accounts
Up next
RAG Applications