You’ve noticed it happen. You search for “budget road trips” and Google returns results about “affordable travel” — words you never typed. Spotify serves you a song you’ve never heard and it’s somehow perfect. ChatGPT seems to understand what you mean even when you phrase things clumsily. Something connects those dots. That something is called an embedding.
Embeddings are the core technology behind how modern AI understands language, finds meaning, and makes recommendations. They sound deeply technical, but the underlying idea is surprisingly intuitive — and once you get it, you’ll see them everywhere.
🗺️ What is an Embedding?
Imagine a library where books don’t sit on shelves by author or publication date. Instead, they float in a room and gradually drift toward other books on similar topics. Books about dogs end up near books about veterinarians. Books about quantum physics float far away, near other books about mathematics and physics. Over time, the physical location of each book tells you something about its meaning.
That room is an embedding space. And the position of each book is its embedding.
More precisely: an embedding is a list of numbers — called a vector — that represents something (a word, a sentence, an image, a product) in a multi-dimensional mathematical space. Things that are similar in meaning end up with similar numbers, which means they land close together in that space.
Here’s the classic demonstration that makes this concrete. In a well-trained word embedding model, you can do arithmetic on meaning:
king − man + woman ≈ queen
Subtract the “man-ness,” add the “woman-ness,” and you arrive at queen. The model didn’t learn a rule. It learned through patterns. And the result lives in the numbers.
🧠 How Embeddings Are Created
No one sits down and manually assigns numbers to words. Embeddings emerge from training.
A model is shown billions of sentences from books, websites, Wikipedia, and anywhere else text exists. It’s given a task: predict what word is likely to appear near this word. “Coffee” appears near “morning,” “cup,” “hot,” and “caffeine.” It rarely appears near “submarine” or “parliament.”
Through millions of iterations, the model adjusts its internal numbers to get better at this prediction. When training is done, those internal numbers — the embeddings — have quietly absorbed the structure of language. Words that appear in similar contexts end up with similar vectors.
The model never read a dictionary. It learned meaning the same way a child does: from context, over and over, until the patterns stick.
📦 From Words to Sentences to Everything
Early embeddings worked at the word level — one fixed vector per word. That created a famous blind spot: the word “bank” gets the same embedding whether you’re talking about a riverbank or a bank account. Context didn’t change the number.
Modern models fixed that. Contextual embeddings (from models like BERT and GPT) generate a different vector for “bank” depending on the surrounding sentence. The embedding is dynamic, shaped by context.
And the technology didn’t stop at words. Today, you can embed:
- Sentences and paragraphs — enabling semantic search across documents
- Images — so you can search photos using text descriptions
- Audio — for music recommendations and speech recognition
- Code — powering tools like GitHub Copilot
- Products — the foundation of recommendation engines
- User behavior — how Netflix knows what you want to watch next
Google’s CLIP model famously maps images and text into the same embedding space. Type “a dog running on a beach” and you can find matching photos — even ones that were never labeled with those words. The image’s meaning and the sentence’s meaning ended up in the same neighborhood.
🔧 The Algorithms Behind Embeddings
You don’t need to know these algorithms to use embeddings, but it helps to recognize the names since they come up constantly in AI conversations.
Word2Vec
Released by Google in 2013, Word2Vec was the breakthrough that made word embeddings mainstream. It trains a shallow neural network to predict words from their neighbors (or neighbors from a word). Fast, efficient, and still widely used for specialized domains.
GloVe
Stanford’s GloVe (Global Vectors) takes a different approach — it builds a giant table of how often every word appears near every other word across an entire corpus, then learns vectors that reflect those global statistics. Often produces slightly richer analogical relationships than Word2Vec.
FastText
Meta’s FastText extends Word2Vec by breaking words into character pieces. This means it can generate embeddings for words it has never seen before. It handles typos, rare words, and languages with complex word structures far better than its predecessors.
BERT and Beyond
Google’s BERT (2018) changed everything by producing contextual embeddings — a different vector per word per context. Modern language models (GPT-4, Claude, Gemini) all use variations of this approach internally. The embeddings they produce are vastly more powerful than static word vectors.
OpenAI Embeddings API
For most people building with AI today, you don’t train your own embeddings. You call an API. OpenAI’s text-embedding-3-large model turns any text you send it into a vector of up to 3,072 numbers. One API call, no training required, and the results are excellent for semantic search and classification.
🌍 Where You Already Use Embeddings (Without Knowing It)
Embeddings aren’t a research curiosity. They’re in production everywhere.
Google Search uses semantic embeddings to understand what you mean, not just what you typed. That’s why searching “how to fix a leaky faucet” returns plumbing guides even if the page never contains the word “fix” — the meanings are close in embedding space.
Spotify and Netflix build an embedding for each song, movie, or show, and another embedding for each user based on their listening and watching history. Recommendation = find the content embedding closest to your user embedding.
ChatGPT’s memory and retrieval features embed your conversations and documents, then use nearest-neighbor search to pull in relevant context when you ask a question. This pattern — embed your documents, search them at query time — is called RAG (Retrieval-Augmented Generation), and it’s how most AI-powered tools work with private data.
Semantic search in Notion and Obsidian finds notes that are about the same topic as your query, even if they use different words. The query gets embedded, your notes get embedded, and the closest matches surface.
🗄️ Vector Databases: Where Embeddings Live
Once you have embeddings, you need somewhere to store and search them efficiently. A regular database finds exact matches — find the row where name = "Alice". That doesn’t work for vectors, where you want to find the most similar match, not an exact one.
A vector database is built specifically for this kind of search. It stores millions of vectors and finds the nearest neighbors to any query vector in milliseconds. Companies like Pinecone, Weaviate, Chroma, and Qdrant specialize in this. Standard databases like Postgres can also handle it via the pgvector extension.
The search technique is called approximate nearest neighbor (ANN) search. Instead of comparing your query to every stored vector (impossibly slow at scale), it uses clever indexing to find close matches fast.
This is the invisible infrastructure under ChatGPT’s file uploads, Perplexity’s source retrieval, and every enterprise AI tool that “knows” your company documents.
❓ Frequently Asked Questions
What’s the difference between a token and an embedding? A token is how text gets chopped up before processing — roughly a word or word-piece. An embedding is the vector representation assigned to that token. Tokenization happens first, then each token gets embedded. They’re sequential steps, not the same thing.
Do I need to be a developer to use embeddings? Not to benefit from them — they power tools you already use every day. If you want to build with them, the OpenAI Embeddings API requires just a few lines of code. No machine learning degree needed.
How many dimensions does an embedding have? It varies by model. Word2Vec typically uses 100–300 dimensions. OpenAI’s ada-002 uses 1,536. OpenAI’s text-embedding-3-large supports up to 3,072. More dimensions generally means finer-grained meaning, at the cost of storage and compute.
What is cosine similarity? The most common way to measure how close two embedding vectors are. Rather than straight-line distance, it measures the angle between two vectors. A score of 1.0 means identical direction (very similar), 0 means perpendicular (unrelated), -1.0 means opposite. It’s preferred because it’s not thrown off by differences in vector magnitude — only direction matters.
What’s a RAG pipeline? Retrieval-Augmented Generation. You embed your own documents, store them in a vector database, and retrieve relevant chunks at query time to give an LLM specific, up-to-date context before it answers. This is how Notion AI, Perplexity, and custom ChatGPT integrations work with private data — the LLM doesn’t memorize your documents, it searches them on demand.
📚 Further Reading
- The Illustrated Word2Vec — Jay Alammar’s visual, intuition-first walkthrough is the best starting point for anyone who wants to go deeper without losing the thread
- OpenAI Embeddings Guide — practical documentation for building with embeddings via API, including code examples
- Sentence Transformers — open-source library for generating high-quality sentence and paragraph embeddings without needing OpenAI
- Pinecone Learning Center — plain-English explainers on vector databases, RAG, and semantic search from the people who build them
📚 Recommended Resources
Disclosure: some links below are Amazon affiliate links. If you buy through them, I earn a small commission — at no extra cost to you — which helps keep this site running.
Build a Large Language Model (From Scratch)
Sebastian Raschka · Book · Manning Publications
The most direct follow-up to this article. Raschka walks you through building a GPT-style model from first principles in PyTorch — including the embedding layers, attention mechanism, and training loop. If you want to go from “I understand what an embedding is” to “I built the thing that makes embeddings,” this is the book.
Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow
Aurélien Géron · Book · O’Reilly Media
The standard practical ML reference. Covers word embeddings, recurrent networks, and transformers with working code throughout. Strong on intuition before implementation — the same approach this article tries to take, just taken much further.
Designing Machine Learning Systems
Chip Huyen · Book · O’Reilly Media
Where this article ends — RAG pipelines, vector stores, semantic retrieval — is where Huyen’s book begins. She covers the full architecture of production ML systems: feature stores, model deployment, monitoring, and the data infrastructure behind the tools you use every day. Essential reading if you’re building with AI rather than just learning about it.
Natural Language Processing with Python
Steven Bird, Ewan Klein & Edward Loper · Book · O’Reilly Media
The classic NLTK-based introduction to how text is processed, tokenised, and analysed before it ever reaches an embedding model. Older than the transformer era but still the clearest explanation of what’s happening to language at the character and token level — worth reading if you want to understand the pipeline from raw text to vector.