Most people’s mental model of AI goes something like: you ask a question, a large language model thinks very hard, and an answer appears. That model is wrong in a way that matters if you’re building anything with AI.
The language model doesn’t know your data. It knows what it was trained on — a snapshot of the internet, frozen in time. Ask it about your company’s pricing, your internal policies, or what happened in last Tuesday’s meeting, and it will either hallucinate a confident-sounding answer or politely admit it doesn’t know.
This is the fundamental problem that RAG solves. And vector databases are the engine that makes RAG fast enough to be useful.
I’ve spent the past few months building AI assistants for small businesses — chatbots that answer customer questions about specific services, prices, and opening hours. The data changes weekly. The language model was trained months ago. Bridging that gap is the entire game.
The Gap Between What the Model Knows and What You Need
A language model like Claude or GPT is trained on enormous amounts of text. It learns patterns, grammar, reasoning, facts. But that training has a cutoff. And even within its training window, it doesn’t know anything about your specific business, your customers, or your data.
You could fine-tune the model on your data. This is expensive, slow, and means retraining every time your data changes. For a business whose opening hours change seasonally, that’s absurd.
The alternative: don’t change the model. Change what you show it.
flowchart LR
subgraph NAIVE["Naive approach"]
Q1["User question"] --> LLM1["Language Model"]
LLM1 --> A1["Answer based on<br/>training data only"]
end
subgraph RAG["RAG approach"]
Q2["User question"] --> RETRIEVE["Retrieve relevant<br/>context from your data"]
RETRIEVE --> COMBINE["Combine question<br/>+ retrieved context"]
COMBINE --> LLM2["Language Model"]
LLM2 --> A2["Answer grounded<br/>in your actual data"]
end
style NAIVE fill:#fde2e2,stroke:#c0392b
style RAG fill:#e2f4e6,stroke:#27ae60That’s RAG in one sentence: retrieve relevant context first, then let the model generate an answer using that context. The model doesn’t need to memorise your data. It just needs to see it at the right moment.
How Text Becomes Searchable: Vectors and Embeddings
Here’s the core insight that makes RAG work. You can’t search text by meaning using traditional databases. If someone asks “What time do you close on Saturdays?” and your database has a row that says “Weekend hours: 9am to 4pm”, a keyword search for “close” and “Saturdays” won’t find it. The words don’t match.
Humans understand that “close” relates to “hours” and “Saturdays” is part of “weekend.” Computers need a different trick.
An embedding model converts a piece of text into a vector — a list of numbers, typically between 768 and 1,536 of them. These numbers capture the semantic meaning of the text, not the exact words. Two sentences that mean similar things end up as vectors that are close together in this high-dimensional space.
flowchart TD
subgraph EMBED["Embedding Model"]
T1["'What time do you close on Saturdays?'"]
T2["'Weekend hours: 9am to 4pm'"]
T3["'Our Saturday roast menu starts at noon'"]
T1 --> V1["[0.23, 0.87, -0.14, 0.55, ...]"]
T2 --> V2["[0.21, 0.84, -0.11, 0.58, ...]"]
T3 --> V3["[0.67, 0.12, 0.43, -0.31, ...]"]
end
V1 -. "cosine similarity: 0.96<br/>(very close)" .-> V2
V1 -. "cosine similarity: 0.34<br/>(not related)" .-> V3
style V1 fill:#e2f4e6,stroke:#27ae60
style V2 fill:#e2f4e6,stroke:#27ae60
style V3 fill:#fde2e2,stroke:#c0392bThe question about closing time and the statement about weekend hours produce vectors that are nearly identical — even though they share almost no words. The Saturday roast menu, despite containing “Saturday,” ends up far away because its meaning is different.
This is what people mean by “semantic search.” You’re not matching keywords. You’re matching meaning.
What a Vector Database Actually Does
A vector database is purpose-built for one job: given a query vector, find the most similar vectors in the collection, fast.
Traditional databases index by exact values — find all rows where status = 'active' or price < 50. They use B-trees, hash indexes, and inverted indexes. These are useless for vectors, because you’re not looking for exact matches. You’re looking for proximity in a space with hundreds of dimensions.
Vector databases use specialised index structures:
| Index Type | How It Works | Trade-off |
|---|---|---|
| HNSW (Hierarchical Navigable Small World) | Builds a graph of nearest neighbours, searches by hopping between nodes | Fast queries, high memory, slow to build |
| IVF (Inverted File Index) | Partitions vectors into clusters, only searches relevant clusters | Good balance, needs tuning |
| Flat / brute force | Compares the query against every vector | Perfect accuracy, doesn’t scale |
For most business applications with fewer than a million documents, HNSW is the default choice. It gives you sub-millisecond search times with excellent accuracy.
The key operation is similarity search — typically using cosine similarity or Euclidean distance. Cosine similarity measures the angle between two vectors: 1.0 means identical direction (same meaning), 0.0 means unrelated, -1.0 means opposite meaning.
Putting It Together: How RAG Actually Works
Here’s the full pipeline, step by step:
sequenceDiagram
participant User
participant App as Your Application
participant Embed as Embedding Model
participant VDB as Vector Database
participant LLM as Language Model
Note over App,VDB: Setup (done once, updated as data changes)
App->>Embed: Convert all business data to vectors
Embed-->>App: Vectors [0.23, 0.87, ...]
App->>VDB: Store vectors + original text
Note over User,LLM: Query time (every user question)
User->>App: "What time do you close on Saturday?"
App->>Embed: Convert question to vector
Embed-->>App: Query vector [0.21, 0.84, ...]
App->>VDB: Find top 5 most similar vectors
VDB-->>App: "Weekend hours: 9-4pm", "We're closed bank holidays", ...
App->>LLM: System prompt + retrieved context + user question
LLM-->>App: "We close at 4pm on Saturdays."
App-->>User: "We close at 4pm on Saturdays."Step 1: Index your data. Break your content into chunks (paragraphs, FAQ entries, product descriptions). Run each chunk through an embedding model. Store the vector alongside the original text in your vector database.
Step 2: Receive a question. When a user asks something, run their question through the same embedding model to get a query vector.
Step 3: Retrieve. Search the vector database for the chunks most similar to the query vector. Typically you retrieve the top 3 to 10 results.
Step 4: Generate. Pass the retrieved chunks as context to the language model, along with the user’s question. The model reads the context and generates an answer grounded in your actual data.
The language model never needs to have seen your data during training. It just needs to be good at reading and synthesising text — which is exactly what it was trained to do.
The Part Nobody Talks About: Chunking
The difference between a RAG system that works and one that hallucinates is often not the model, not the vector database, and not the embedding. It’s how you split your data into chunks.
Too large (entire pages) and the retrieved context is bloated — the model struggles to find the relevant bit among paragraphs of noise. Too small (individual sentences) and you lose context — “9am to 4pm” without knowing that refers to Saturday is useless.
There’s no universal answer. What works depends on your data:
- FAQ pairs (question + answer) are natural chunks. Don’t split them.
- Product descriptions work well as whole items — one product, one chunk.
- Long documents need splitting with overlap. A 200-word chunk with 50 words of overlap into the next chunk prevents losing meaning at boundaries.
- Structured data (opening hours, pricing tables) often works better as small, typed chunks: “Monday: 9am-5pm” as its own chunk with metadata tagging the type.
In our chatbot system, we don’t actually use a vector database. The total content per tenant — pricing, services, hours, FAQs — fits comfortably within the language model’s context window. So we skip the retrieval step entirely and inject all the content directly into the system prompt. This works because we’re dealing with small businesses with bounded content sets, not enterprises with thousands of documents.
That’s an important point: RAG with a vector database is not always the right architecture. If your data fits in the context window, direct injection is simpler, faster, and has no retrieval accuracy to worry about.
When You Need a Vector Database (and When You Don’t)
| Scenario | Data Size | Approach | Why |
|---|---|---|---|
| Small business chatbot | < 50 pages of content | Direct injection into prompt | Simpler, no retrieval errors, all context always available |
| Company knowledge base | 50 - 5,000 documents | RAG with vector DB | Too large for context window, need semantic search |
| Legal / medical corpus | 10,000+ documents | RAG with vector DB + re-ranking | Need precision, relevance ranking, citation tracking |
| Code search | Entire codebase | RAG with specialised embeddings | Code semantics differ from natural language |
The threshold is roughly the model’s context window. Claude can handle around 200,000 tokens of context — that’s roughly 150,000 words, or a 500-page book. If your data fits, inject it directly. If it doesn’t, you need retrieval.
Multi-Tenancy and RAG: The Isolation Problem
If you’re building a multi-tenant AI product — one system serving multiple customers — there’s a critical isolation concern. Each tenant’s data must be retrievable only for that tenant’s queries.
flowchart TD
Q["Query from Tenant A"] --> EMBED["Embedding Model"]
EMBED --> SEARCH["Vector Search"]
SEARCH --> FILTER{"Metadata filter:<br/>tenant_id = A"}
FILTER -->|"Tenant A docs"| RESULTS_OK["Relevant context<br/>(correct tenant)"]
FILTER -.->|"Tenant B docs"| RESULTS_BAD["Blocked by filter<br/>(data leak prevented)"]
RESULTS_OK --> LLM["Language Model"]
LLM --> ANSWER["Answer grounded<br/>in Tenant A's data"]
style RESULTS_OK fill:#e2f4e6,stroke:#27ae60
style RESULTS_BAD fill:#fde2e2,stroke:#c0392bThree approaches, in order of increasing isolation:
Metadata filtering (pool model): Store all tenants’ vectors in one collection, tag each with tenant_id, and always filter on it during search. This is the cheapest approach and works well if your vector database supports efficient metadata filtering. Most do — Pinecone, Weaviate, and AWS Bedrock Knowledge Bases all support this natively.
Namespace separation: Some vector databases support namespaces or collections within a single instance. Each tenant gets their own namespace. Stronger isolation than metadata filtering, similar cost.
Separate instances: Each tenant gets their own vector database. Maximum isolation, maximum cost. Only justified for enterprise clients with strict data residency requirements.
The right choice mirrors the multi-tenancy decision in your main database. For most SaaS products serving small businesses, metadata filtering with consistent enforcement is the right default.
The Practical Takeaway
RAG isn’t magic. It’s a pipeline: embed your data, store the vectors, retrieve the relevant bits, and let the model do what it’s good at — reading and synthesising.
The vector database makes the retrieval step fast and accurate at scale. Without it, you’d be comparing your query against every document linearly. With it, you get sub-millisecond semantic search across millions of documents.
But the engineering is in the details: how you chunk your data, how you handle multi-tenancy, when you skip the vector database entirely, and how you keep the indexed data in sync with the source of truth.
The best RAG systems I’ve seen share one trait: they treat the retrieval quality as a first-class metric, measured and monitored alongside the model’s response quality. Because the best language model in the world can only work with what you give it. If the retrieval step hands it irrelevant context, the answer will be confidently wrong — the worst kind of failure in a system people trust.
Start simple. Inject what fits. Add retrieval when you outgrow the context window. And always, always filter by tenant.