How Semantic Search Works: The Transformer Behind It

Type “how do I get my money back” into a modern search box and it finds a page titled “requesting a refund” — even though you didn’t share a single word with it. That is semantic search: searching by meaning instead of by matching keywords. And understanding how semantic search works means understanding one downstream gift of the 2017 Transformer paper, which taught machines to turn language into numbers that capture what words actually mean in context.

An honest note before we start, because the history gets garbled: the famous “Attention Is All You Need” paper was about translating languages, not about search. Everything in this article is a ripple — what the Transformer enabled through its descendants, not something the 2017 paper did itself. With that boundary clear, let’s follow the meaning. If you want the mechanism first, our cornerstone on what attention is has you covered; here we put it to work.

How semantic search works — a query lands in an embedding space and lights up the nearest documents by meaning, not keywords

🔢 The One Idea Underneath It All: Embeddings

Every semantic search system rests on a single concept: the embedding. An embedding turns a piece of text into a list of numbers — a vector — positioned in a space where things with similar meaning sit close together and unrelated things sit far apart. “Refund,” “money back,” and “reimbursement” land in the same neighborhood; “photosynthesis” lands across town.

If that idea is new to you, our explainer on what an embedding is is the foundation for everything here — and image embeddings shows the exact same trick for pictures. Once meaning is a location in space, “find related things” becomes “find nearby points,” which is something computers are very good at. The only question is: how good are the embeddings? And that is precisely where the Transformer changed everything.

📈 The Leap: From “One Meaning per Word” to Context

Before the Transformer, the best word representations were static. Methods like word2vec and GloVe (from around 2013) gave each word exactly one vector, forever. That was a breakthrough at the time — but it had an obvious flaw: a word means different things in different sentences.

Consider the word “bank.” In “I sat on the river bank” and “I deposited cash at the bank,” it means two completely different things — yet a static embedding gives it the same vector both times. The context is thrown away.

The Transformer’s descendants fixed this. Because the attention mechanism lets every word’s representation be built from all the other words around it, the resulting embeddings are contextual: “bank” by a river gets a different vector than “bank” holding your money. The same word, read in two sentences, produces two different fingerprints — each reflecting its actual meaning in that spot. That single upgrade, from static to contextual embeddings, is the reason modern search feels like it understands you. It’s the Transformer’s real gift to search.

The model that made this mainstream for language was BERT (2018). Here a small but important detail: BERT is an encoder-only Transformer — it kept just the “understanding” half of the original encoder-decoder design. (The generating half became the GPT line.) Search runs mostly on the encoder side; keep that distinction and the family tree stays clear.

Static vs contextual embeddings — the word 'bank' getting one fixed vector versus two different vectors depending on its sentence

🔎 How a Search Actually Runs: Embed, Then Find Neighbors

With good contextual embeddings in hand, a semantic search engine does something refreshingly simple. It’s the same “embed, then find the nearest neighbors” recipe LIG describes for pictures in how reverse image search works — just with a text encoder instead of an image one:

Offline (ahead of time): run every document — every help article, product description, or paragraph — through a Transformer encoder and store the resulting embedding vectors in a special database called a vector index.
At query time: run the user’s query through the same encoder to get its embedding.
Find neighbors: retrieve the stored vectors sitting closest to the query’s vector (by a distance measure like cosine similarity), and return those documents.

Because the query “get my money back” lands near the stored “requesting a refund,” the right page surfaces — no shared keywords required. That’s the whole trick. And you don’t need a research lab to run it: our hands-on guide to Postgres as a vector database walks through building exactly this pipeline yourself, storing embeddings and querying for neighbors on your own machine.

One practical wrinkle worth knowing: comparing a query against millions of stored vectors exactly would be slow, so production systems use approximate nearest-neighbor search — clever shortcuts that find almost the closest matches while skipping most of the haystack, trading a sliver of accuracy for enormous speed.

🧠 A Detail That Makes It Scale: Sentence Embeddings

There’s a subtle engineering problem hiding in step 2. Vanilla BERT is fantastic at judging how similar two sentences are — but only if you feed it both at once, which means to compare your query against a million documents you’d have to re-run the model a million times. Hopeless at scale.

The fix was Sentence-BERT (2019), which adapts the model to produce a single, self-contained embedding for each text independently. Now you embed every document once, offline, and store the vectors; at query time you embed only the query and compare. That’s what makes web-scale semantic search practical, and it’s the pattern behind every modern text-embedding service (the embedding APIs from OpenAI, Cohere, and open-source families like E5 and BGE — all Transformer encoders, all descendants of 2017).

🤖 Where This Leads: RAG and Smarter AI Answers

Semantic search isn’t just a better search box — it’s the retrieval half of one of the most important patterns in modern AI: retrieval-augmented generation, or RAG.

RAG is two Transformer descendants working as a team. An encoder embeds your question and retrieves the most relevant documents (the semantic search we just described). Then a decoder — a generating model like the ones behind ChatGPT and Claude — is handed those documents and writes an answer grounded in them. Instead of answering from memory (and risking making things up), the AI answers from sources it just looked up, and can cite them.

That’s why semantic search matters far beyond search itself: it’s how you give an AI assistant access to your private documents, or to current information it was never trained on. The question of how to retrieve — plain vector similarity versus something structured — is an active and fascinating debate, which LIG explores in GraphRAG vs Vector RAG and graph search vs vector search. But whichever retrieval method you choose, the embeddings underneath were made possible by the Transformer.

⚠️ The Honest Limits

Semantic search is powerful, not perfect, and its weaknesses follow from how it works. Two vectors landing close together means the texts are similar in meaning — not that either is true, authoritative, or what you specifically wanted. Keyword search still wins when you’re hunting an exact phrase, a product code, or a precise name, which is why serious systems often blend semantic and keyword search (“hybrid search”). And every embedding inherits the biases of the data its encoder learned from, so it will judge some kinds of text more reliably than others. Meaning-matching is a strong default, not an oracle — treat a top result as “most relevant,” not “definitely right.”

🧭 The Big Picture

Step back and the lineage is clean. A 2017 paper about translation introduced an architecture whose encoder turned out to be extraordinary at turning language into meaning-rich, context-aware numbers. Its descendants — BERT, Sentence-BERT, and the embedding models that followed — made those numbers cheap and practical, and semantic search, vector databases, and RAG are what we built on top.

None of that was in the original paper. But it’s why, today, a search box can understand that “get my money back” and “requesting a refund” are the same wish. The Transformer didn’t set out to fix search. It fixed meaning — and search was one of the first things to benefit.

❓ Frequently Asked Questions

What is semantic search? Search by meaning rather than by matching exact words. It turns your query and all the documents into embeddings — numeric fingerprints of meaning — and returns the documents whose fingerprints sit closest to your query’s, so “get my money back” can find a page titled “requesting a refund” with no shared keywords.

How does the Transformer relate to search? Indirectly but crucially. The 2017 Transformer paper was about translation, but its descendants (starting with BERT in 2018) produce contextual embeddings — where a word’s vector reflects its meaning in that specific sentence. Those high-quality embeddings are what make modern semantic search work. It’s a downstream ripple, not something the original paper did.

What’s the difference between semantic search and keyword search? Keyword search matches exact words; semantic search matches meaning. Semantic search finds relevant results even with no shared words, but keyword search is better for exact phrases, names, or codes. Many real systems combine both (“hybrid search”).

What are contextual embeddings? Word or text representations where the numbers depend on the surrounding context. Unlike older static embeddings (word2vec, GloVe) that gave a word one fixed vector, contextual embeddings from Transformer encoders give “bank” a different vector in “river bank” than in “savings bank” — capturing its actual meaning in each sentence.

How does semantic search connect to RAG and AI chatbots? RAG (retrieval-augmented generation) uses semantic search to fetch relevant documents, then hands them to a generative AI model so it answers from real sources instead of memory. This lets an assistant answer questions about your private documents or recent events, and cite where the answer came from.

📚 Further Reading

What Is “Attention” in AI? — the mechanism behind the contextual embeddings that power semantic search.
What Is an Embedding? and Image Embeddings Explained — the foundation, for text and for pictures.
Postgres as a Vector Database — build the embed-then-search pipeline yourself, hands-on.
GraphRAG vs Vector RAG and Graph Search vs Vector Search — the active debate over how to retrieve, one layer up from the embeddings.

📚 Recommended Resources

Disclosure: some links below are Amazon affiliate links. If you buy through them, I earn a small commission — at no extra cost to you — which helps keep this site running.