Understanding Embeddings and Vector Databases

Most explanations of vector databases spend their time comparing Pinecone against Weaviate against pgvector, as if the database is the hard part. It usually isn’t. The variable that actually determines whether retrieval works or quietly fails is chunking strategy — how you cut your source text into pieces before embedding it — and that’s the part most comparisons skip entirely.

Quick answer

An embedding is a fixed-length list of numbers that represents a piece of text in a way that captures its meaning — text with similar meaning ends up as vectors that are numerically close together, even if the words themselves are completely different. A vector database stores these embeddings alongside a reference to the original text, builds an index so it doesn’t have to compare every vector one by one, and answers a query by returning the closest matches. Together, they power retrieval-augmented generation (RAG): embed the query, find the most similar stored chunks, inject them into the model’s context, and generate an answer grounded in that retrieved text instead of the model’s training data alone.

Key takeaways

• Embeddings capture meaning, not keywords — a search for “car” can surface a document that only ever says “automobile,” because the two end up numerically close in vector space.
• Chunking strategy, not database choice, is usually the actual bottleneck in retrieval quality — the 2026 practitioner consensus favors 256-512 token chunks with roughly 50-token overlap over embedding whole documents.
• Using different embedding models for ingestion and query silently breaks retrieval — vectors from different models live in different numerical spaces, so a query embedded with Model B against content embedded with Model A returns effectively random results, with no error to flag it.
• Vector search is approximate, not exact, by design (ANN — approximate nearest neighbor) — it trades a small, usually negligible accuracy loss for the speed needed to search millions of vectors in real time.

What an embedding actually is

An embedding model converts a piece of text into a vector — a fixed-length list of numbers, typically somewhere between a few hundred and a few thousand dimensions depending on the model. What makes this useful isn’t the numbers themselves, but the geometry they create: text with similar meaning produces vectors that sit close together in that multidimensional space, while unrelated text ends up far apart. This is what allows semantic search to work: a query for “how to reduce cloud costs” can retrieve a document titled “Optimizing AWS Spend” even though the two share almost no exact words, because their embeddings land near each other in vector space.

Embedding models have their own context window, separate from and usually much smaller than an LLM’s. If the text you feed in exceeds that limit, the excess is typically truncated — silently cut off — before it’s converted to a vector, rather than throwing an error. This is one reason chunking matters even before you consider retrieval quality: feeding an oversized document straight into an embedding model can quietly lose the back half of it with no warning — see our guide to how tokens work for the underlying mechanics of context windows generally.

How a vector database actually works

A vector database does three specific jobs, and understanding them separately makes the whole system easier to reason about:

1. Stores each vector alongside a reference to its original text. The vector is what gets searched; the reference is what lets the system hand back the actual passage once a match is found.
2. Builds an index for fast lookup. Comparing a query against every stored vector one by one doesn’t scale past a small collection — the index lets the database narrow the search dramatically before doing detailed comparisons.
3. Answers a query by returning the closest vectors. “Closest” is typically measured by cosine similarity, and the database returns the top-K matches — the K most similar chunks, where K is a number you choose based on how much context you want to retrieve.

That search is approximate by design — the common term is ANN, approximate nearest neighbor. An exact nearest-neighbor search across millions of vectors is too slow for real-time use, so production vector databases trade a small amount of accuracy for a large amount of speed. In practice this means the returned results are very likely, but not absolutely guaranteed, to be the true closest matches — a distinction that rarely matters in practice but is worth knowing when debugging an unexpected retrieval result.

The four-step RAG loop

Embeddings and vector databases combine into retrieval-augmented generation through a consistent pattern, regardless of which specific database or embedding model is involved:

1. Embed the query. The user’s question is converted into a vector using the same embedding model used at ingestion time.
2. Search. The vector database returns the top-K most similar stored chunks.
3. Inject. Those chunks are inserted into the model’s context alongside the original question, typically in the user message rather than the system prompt — see our guide to system prompts for why that split matters — since retrieved content changes with every query.
4. Generate. The model produces an answer grounded in the retrieved text, rather than relying solely on what it learned during training.

This pattern is what makes RAG useful for private or fast-changing data the model was never trained on, and for reducing hallucination on domain-specific questions — the model is answering from text actually in front of it, not just from memory. Worth noting: since retrieved chunks change with every query, they typically don’t benefit from prompt caching the way a genuinely stable system prompt does — only the unchanging parts of a RAG pipeline’s prompt, like fixed instructions, are realistic caching candidates. Our RAG Pipeline Cost Calculator breaks down what each of these steps costs separately if you’re pricing out a real implementation.

Why chunking matters more than which database you pick

When retrieval quality is disappointing, the instinct is often to switch vector databases. In practice, the database is rarely the actual problem — chunking strategy is. A single vector represents a fixed-size piece of text, and how you cut that text up directly determines what the vector can and can’t capture.

Embedding an entire 20-page document as one vector blends everything in it into a single, blurry point in vector space. A query about one specific paragraph on page 14 has to compete, in that same vector, with everything else the document covers — the embedding can’t represent “mostly about X, except for this one relevant part,” so retrieval quality suffers even though nothing is technically broken. The current practitioner consensus for text-heavy content is chunks of roughly 256-512 tokens, with about a 50-token overlap between consecutive chunks so that information sitting near a chunk boundary isn’t split away from its surrounding context entirely.

Fixed-size chunking — cutting text every N tokens regardless of what’s actually there — is simple to implement but loses context at boundaries, sometimes splitting a sentence or a table row in half. Semantic chunking, which splits along paragraph or concept boundaries instead of a fixed token count, generally preserves meaning better at the cost of slightly more implementation complexity. For most applications, starting with semantic chunking at the paragraph or section level, then adjusting based on actual retrieval results, is a more reliable starting point than optimizing chunk size in the abstract.

A worked example: chunking a support knowledge base

Say you’re building retrieval over a 200-page product documentation set for a support chatbot. Embedding each of the roughly 50 top-level pages as one vector each is the fastest approach to implement — and the one most likely to disappoint. A user question about a specific error message buried in page 32 has to compete against everything else that page covers, and the resulting vector is too diluted to reliably surface that specific paragraph.

Splitting the same documentation into roughly 400-token chunks with 50-token overlap, aligned to actual section breaks rather than an arbitrary character count, produces several thousand chunks instead of 50 pages. Each one now represents a focused, specific piece of content — a single error explanation, a single configuration step — which is exactly the granularity a similarity search needs to surface the right answer instead of a whole page containing it somewhere.

The tradeoff is more vectors to store and search, which is a real but usually minor cost — see our token counter for a sense of how document length translates to token count, and by extension, how many chunks a given chunking strategy produces.

The mistake that silently breaks retrieval

Beyond chunking, one specific implementation mistake causes retrieval to fail in a way that’s hard to diagnose because nothing throws an error: using different embedding models for ingestion and for querying. Vectors produced by different models live in different numerical spaces — there’s no shared coordinate system between them. A query embedded with one model, searched against content embedded with a different model, returns results that are essentially random, because “closeness” in one model’s vector space has no defined relationship to “closeness” in another’s.

This typically happens when a team switches embedding providers or upgrades to a newer model version without re-embedding their existing content — the old vectors stay in the database, new queries get embedded with the new model, and retrieval quality degrades in a way that looks like a database problem but isn’t. The fix is straightforward once identified: lock the embedding model at ingestion, and if you ever change it, re-embed the entire existing corpus rather than mixing vector spaces.

Common mistakes

• Embedding whole documents instead of chunks. A single vector for an entire long document blends all its meaning into one point, making targeted retrieval nearly impossible.
• Mixing embedding models between ingestion and query. This produces results that look plausible but are essentially meaningless, since the two vector spaces share no relationship.
• Choosing chunk size without testing against real queries. The 256-512 token range is a reasonable starting point, not a guarantee — content structure (dense technical text versus conversational prose) changes what actually works best.
• Ignoring the embedding model’s own context window. Text that exceeds it gets silently truncated before embedding, which can quietly drop the most important part of a long chunk with no error to catch it.

Advanced tips

Add metadata filters alongside vector search rather than relying on similarity alone. Filtering by source, date, or document type before or alongside the similarity search often improves precision more than tuning the embedding or chunking strategy further, particularly when your corpus spans multiple document types with different relevance patterns.

Test retrieval quality with real queries, not synthetic ones. Chunk size and overlap that look reasonable in the abstract can still underperform on your actual query patterns — a handful of representative real questions run against different chunking configurations reveals more than any general guideline can.

Consider a reranking step for high-stakes retrieval. Vector search returns the top-K most similar chunks quickly, but a separate, more computationally expensive reranking pass over just those K candidates can meaningfully improve final ordering — worth the added latency and cost for applications where retrieval accuracy matters more than raw speed.

The takeaway

Embeddings and vector databases are conceptually simple — text becomes a vector, similar meaning ends up numerically close, and a database returns the closest matches quickly. Where implementations actually go wrong is almost never that core mechanism. It’s chunking text too coarsely to be useful, or quietly mixing embedding models between ingestion and query. Get those two things right, and the specific database you choose — Pinecone, Weaviate, pgvector, or something else — matters far less than most comparisons suggest. This is also exactly the retrieval layer our guide to how AI coding assistants work touches on when discussing how agentic tools retrieve relevant code rather than loading an entire repository into context at once.

FAQ

Do I need a dedicated vector database, or can I use a regular database?

Several mainstream databases, including Postgres via the pgvector extension, now support vector similarity search directly, and this is often sufficient for smaller-scale applications already using that database for everything else — avoiding the operational overhead of running a separate system just for vector search. Dedicated vector databases generally win at larger scale, where their purpose-built indexing handles millions of vectors with better performance than a general-purpose database retrofitted for the task — the crossover point varies by workload, but teams typically feel it once vector search latency starts noticeably affecting the rest of the application.

How many dimensions should an embedding have?

This is set by whichever embedding model you choose, not something you configure independently — common models produce vectors anywhere from a few hundred to a few thousand dimensions. Higher dimensionality generally captures more nuance but costs more to store and search; most applications do fine with a mid-range model rather than the largest available one — our AI Model Comparison Tool covers current model options if you’re deciding which one to standardize on.

Can embeddings work for images and other non-text data?

Yes — the same fixed-length-vector concept applies to images, audio, and other data types, using embedding models trained specifically for that modality rather than the text-focused models discussed throughout the rest of this guide. Multimodal embedding models that place text and images into the same shared vector space also exist, enabling searches like finding an image using a text description.

Why did my retrieval quality drop after I updated my embedding model?

This is almost always the mixed-embedding-model problem described above — new queries are being embedded with the updated model while existing content in the database still reflects the old one. The fix is re-embedding the full existing corpus with the new model, not just the content going forward.

Is RAG the same thing as fine-tuning a model on my data?

No — they solve a similar problem differently. RAG retrieves relevant information at query time and injects it into context, leaving the underlying model unchanged. Fine-tuning actually adjusts the model’s weights based on training examples. RAG is generally faster to set up and easier to keep current as source data changes; fine-tuning can better teach a model a specific style or reasoning pattern that retrieval alone doesn’t capture.

How is this different from just giving a model a longer context window?

A longer context window still has a limit, and stuffing an entire large knowledge base into every request is both expensive and often less accurate than retrieval, since the model has to weigh relevant and irrelevant content equally across a huge context. Retrieval narrows the field first, so the model only sees the specific chunks likely to matter for that particular question — see our AI Model Cost Calculator for how much a large, unfiltered context actually costs per request compared to a targeted retrieval approach.

Does chunk overlap waste storage and money?

Marginally, yes — a 50-token overlap on 512-token chunks adds roughly 10% more stored vectors than no overlap at all. In nearly every practical case, the retrieval quality improvement from preserving context across chunk boundaries is worth that modest storage cost, especially since vector storage is typically a small fraction of an application’s total infrastructure spend.