Vector Databases: What They Actually Do

If you've spent any time around the engineering community lately, you've probably heard the term "Vector Database" thrown around like it's the next big thing. For a while, I honestly thought it was just another buzzword that hype-chasers were using to sell more infrastructure.

But when I finally sat down to understand how these things work under the hood, it completely changed how I think about searching and storing data. I want to share what I learned—diving deep into the actual math and mechanics, but keeping it grounded in plain English.

The Problem with Traditional Search

Think about how a standard SQL database works. If you want to find something, you query for exact matches. SELECT * FROM products WHERE color = 'red'

It's rigid. It relies on exact keywords.

But what if you wanted to search for "ocean-like shades"? A traditional database would just shrug at you. It doesn't understand meaning or context. For a long time, we patched this problem using inverted indexes (like Elasticsearch) that count how often words appear. But even those don't actually understand what the words mean. They just know that the string "ocean" appeared three times.

Enter Vectors: Turning Meaning into Coordinates

To understand a vector database, you first have to understand a Vector.

Imagine you are trying to describe different animals using a scoring system from 0 to 10. Let's pick two traits: "How fluffy is it?" and "How big is it?"

• A Golden Retriever might score an 8 on fluffiness and a 6 on size. We write as [8, 6].
• A Cat might score a 7 on fluffiness and a 3 on size. We write this as [7, 3].
• A T-Rex might score a 0 on fluffiness and a 10 on size. We write this as [0, 10].

In math, this list of numbers is a vector.

Because these animals are just numbers now, we can plot them on a 2D graph. If you look at the graph, the Golden Retriever [8, 6] and the Cat [7, 3] are located pretty close to each other. The T-Rex [0, 10] is way off in the corner by itself.

By simply measuring the distance between the points, the computer suddenly understands that a dog is more similar to a cat than it is to a dinosaur.

The 1,536-Dimensional Space

Scoring animals on two traits is easy. But how do you score the meaning of an entire sentence? Or an image?

This is where AI embedding models (like OpenAI's text-embedding-ada-002) come in. Instead of just 2 traits, these models score text across 1,536 different dimensions. You and I don't even know what all 1,536 dimensions are—the neural network learned them on its own during training. One dimension might loosely correlate with "is this about food?", another with "is this sarcastic?", and so on.

When you feed a paragraph into this model, it outputs a list of 1,536 numbers. This massive coordinate point represents the true, deep, semantic meaning of your paragraph.

How Do We Measure "Similarity"? (The Math)

So you have millions of these 1,536-dimensional coordinates in your database. A user types a search query: "comfortable shoes for running." You run that query through the AI model to get its vector coordinate. Now, you need to find the vectors in your database that are closest to the user's query vector.

How does the database actually measure distance in 1,536 dimensions? Usually, it uses Cosine Similarity.

Instead of measuring the physical distance between two points (Euclidean distance), Cosine Similarity measures the angle between two vectors if you drew lines from the origin [0,0,0...] to each point.

• If the angle is 0 degrees, the vectors point in the exact same direction (highly similar).
• If the angle is 90 degrees, they are completely unrelated.
• If the angle is 180 degrees, they mean the exact opposite.

Cosine similarity is incredibly efficient for computers to calculate, and it ignores the "length" of the vector, focusing purely on the direction (the semantic meaning).

The Core Problem: The Nearest Neighbor Search

Here is the catch: finding the closest point among millions of points in a space with 1,536 dimensions is incredibly computationally expensive.

If you used a traditional database, it would have to calculate the cosine similarity between the user's query and every single row in the database (known as K-Nearest Neighbors, or KNN). It would be perfectly accurate, but your server would crawl to a halt. It simply doesn't scale.

We need a way to find the closest vectors without checking every single one. We need Approximate Nearest Neighbors (ANN).

How Vector Databases Actually Work (HNSW)

This is where true Vector Databases (like Pinecone, Milvus, or Qdrant) earn their keep. They don't just store vectors; they organize them using brilliant algorithms so you can search them in milliseconds.

The most popular algorithm powering modern vector databases is HNSW (Hierarchical Navigable Small World).

To understand HNSW, think about how you navigate a road system to drive to a friend's house in another state:

1. You start on the Interstate Highway (the top layer) to cross the country rapidly.
2. You take an exit onto a State Highway (middle layer) to get to their city.
3. You get onto Local Streets (bottom layer) to find their exact house.

HNSW builds a literal multi-layered graph of your vectors.

• The bottom layer contains every single vector in your database, connected to its closest neighbors.
• The layer above it contains only 10% of the vectors, serving as "exits."
• The top layer might contain just 1% of the vectors, serving as the "highways."

When you search, the database drops your query into the top layer. It quickly hops between the few "highway" vectors to find the general neighborhood. Then, it drops down a layer, hones in closer, and drops down again to the local streets. By doing this, the database only has to calculate the distance to a few hundred vectors, rather than millions, allowing it to return results in milliseconds.

The Metadata Challenge

A vector database isn't just an algorithm like HNSW. It's a full database system. One of the hardest things it has to do is combine vector search with traditional filtering.

Imagine you search for "red running shoes" (Semantic Vector Search) but you also click a filter that says Size = 10 and In Stock = True (Traditional SQL Filter).

If the database finds the closest vectors first (Post-filtering), it might return 10 pairs of red shoes, only to realize none of them are Size 10, leaving the user with zero results. If it filters for Size 10 first (Pre-filtering), it breaks the carefully constructed HNSW highway system, forcing the database to do a slow, full scan of the remaining items.

Modern vector DBs have developed complex techniques (like Single-Stage Filtering) where the HNSW graph is traversed while simultaneously evaluating the metadata filters at every node. Building this infrastructure from scratch is why dedicated vector databases exist.

Wrapping Up

Learning about vector databases was one of those rare moments where the hype actually felt justified. It's not just a faster database; it's a completely new way for computers to understand, organize, and search through human information by treating meaning as a mathematical landscape.

Whether you are building RAG applications for LLMs, semantic search bars, or context-aware recommendation engines, vectors are the primitive that makes modern AI possible.