Every AI-generated image you have ever seen — every Midjourney portrait, every DALL·E illustration, every Stable Diffusion render — began its life as a screen of pure, meaningless static. Understanding how diffusion models work means understanding how a computer learns to start from that random noise and, step by careful step, remove it until a coherent picture emerges. It sounds like magic. It is, in fact, one of the most elegant ideas in modern artificial intelligence — the diffusion model — and, reassuringly, one you can grasp without a single equation.
This is the comprehensive guide. We will build the idea up from its simplest form (a network that learns to un-blur an image), through the real machinery behind Stable Diffusion (the U-Net, the VAE, the text encoder), the frontier techniques that made generation nearly instant, the honest limits and controversies, and the wider world diffusion now touches — video, audio, 3D, even protein design. It is long on purpose: this is meant to be the one page that answers your questions and links you to deeper dives on each piece. Let’s begin.
Where Diffusion Fits: The Generative-AI Family
Before the mechanics, the map. Diffusion is one member of a family called generative AI — systems that create new content rather than just classifying existing content. And it sits at a specific point in the recent history of AI, which is easiest to see as a trilogy:
- Recognition came first. In 2012, convolutional neural networks (CNNs) taught computers to recognize what’s in an image — the “big bang” of modern deep learning.
- Language came next. In 2017, the Transformer taught computers to handle text, leading to large language models like ChatGPT.
- Generation — creating brand-new images — is where diffusion won. And, as we’ll see, it didn’t replace the other two; it is built on top of them.
Diffusion also wasn’t the first way to generate images. Two earlier approaches dominated first, and knowing them clarifies what diffusion is:
| Approach | How it generates | Strengths | Weaknesses |
|---|---|---|---|
| VAE (2013) | Encode data to a compact code, then decode | Fast, stable to train | Outputs often blurry |
| GAN (2014) | A generator and a critic compete | Sharp, fast at generation | Unstable training; limited diversity (“mode collapse”) |
| Diffusion (2015/2020) | Iteratively remove noise | High quality and diversity; stable training | Slow (many steps) |
| Autoregressive | Predict the image pixel/patch by patch | Flexible | Slow; historically lower quality for images |
For most of the 2010s, GANs were the undisputed champion of AI image generation. Diffusion was an obscure idea almost nobody used. The story of how it overtook GANs is the story of this guide.
The Core Idea: Learning to Remove Noise
Here is the whole concept in one analogy. Imagine a sculptor who works not with marble but with television static. They start with a screen of pure random noise and, with thousands of tiny, confident adjustments, remove exactly the right specks until a face, a landscape, or a spaceship emerges from the chaos. A diffusion model is that sculptor — except it taught itself the craft by watching the process happen in reverse. That self-taught denoising is the heart of how diffusion models work, and the next three subsections build it up piece by piece.
The forward process: adding noise on purpose
Training starts with a counter-intuitive move: you take a real training image and deliberately destroy it, adding a little random Gaussian noise, then a little more, then more, across many steps, until the image is indistinguishable from pure static. This is the forward (or “diffusion”) process — the name borrowed from physics, where it describes how a drop of ink spreads through water until it’s evenly, randomly distributed. Crucially, at every step you know exactly how much noise you added.
The reverse process: the network that undoes it
Now the clever part. You train a neural network to do the opposite: given a noisy image, predict what noise was added so it can be removed, nudging the image one small step back toward clarity. Do this across millions of images and billions of noising steps, and the network becomes an expert at one narrow, powerful skill — looking at a noisy picture and estimating what’s noise versus what’s real signal. This is the reverse (denoising) process, introduced for images in the landmark 2020 paper Denoising Diffusion Probabilistic Models (DDPM), building on a 2015 idea from Sohl-Dickstein and colleagues.
How generation actually works
Once trained, generating a brand-new image is beautifully simple: start from a fresh screen of pure random static — noise the network has never seen — and ask it to denoise, step by step, all the way down. Because the network learned the statistical shape of real images, its repeated “remove the noise” nudges don’t reconstruct any specific training photo; they hallucinate a new, plausible one out of the randomness. Different starting noise gives a different image. That is the entire generative trick: not summoning a picture from nothing, but carving one out of noise using a very well-honed sense of what real images look like.
The training objective, in one line
If you remember one technical detail, make it this: the network is trained to predict the noise. That’s the actual DDPM objective — not “draw a cat,” just “given this noisy image, what’s the noise?” Everything else — the gorgeous outputs, the prompt-following — emerges from getting that one humble prediction very, very good. And to be clear about what this is not: the model has no understanding of what a cat is. It has learned an extraordinarily detailed statistical map of how pixels tend to arrange themselves. It is denoising, not comprehending.
Inside the Machine: The Denoiser Architecture
“A network that predicts noise” is the job. What kind of network actually does it? This is where diffusion quietly connects back to the rest of AI.
The U-Net — a CNN with attention
The classic diffusion denoiser is a U-Net: a convolutional neural network shaped like the letter U. It first compresses the noisy image down through a series of convolutional layers (capturing the big-picture structure), then expands it back up to full resolution, with “skip connections” that shuttle fine detail across the U so nothing is lost. Modern U-Nets also sprinkle in attention layers — the same attention mechanism that powers language models — so the network can relate distant parts of the image to each other.
Sit with that for a second: the image generator’s engine is a CNN with attention baked in. Diffusion doesn’t compete with the recognition and language architectures — it uses them.
Why a U-Net
Denoising has to happen at every scale at once. Is that speckle in the sky noise, or a distant bird? You can only tell by considering both the local pixels and the global scene. The U-Net’s compress-then-expand shape lets the network reason about coarse structure and fine texture simultaneously, which is exactly what removing noise well requires.
Diffusion Transformers (DiT)
More recently, researchers replaced the U-Net’s convolutional backbone with a pure Transformer, producing the Diffusion Transformer (DiT). The idea: chop the image into patches and treat them like the tokens of a sentence, letting attention do the heavy lifting. DiT scales beautifully with size and data, and it is widely reported to be the architecture behind OpenAI’s Sora video model. The takeaway reinforces the theme — the denoiser can be a CNN (U-Net) or a Transformer (DiT); diffusion is the process that runs on top of whichever you choose.
The Efficiency Breakthrough: Latent Diffusion
Here is the single most important engineering leap, and the reason you can run image generation on a gaming laptop instead of a supercomputer. It’s also the thing pop explanations most often get wrong.
The pixel-space problem
Running the full noise-then-denoise process directly on pixels is brutally expensive. A high-resolution image is millions of numbers, and you’re processing it dozens or hundreds of times per generation. Early pixel-space diffusion models cost on the order of hundreds of GPU-days to train — firmly out of reach for anyone but a big lab.
The VAE compression trick
The 2022 paper High-Resolution Image Synthesis with Latent Diffusion Models solved this with a wonderfully practical idea: don’t run diffusion on the pixels — run it on a compressed summary. A separate autoencoder (a VAE) first squeezes the image into a much smaller “latent” representation — think of it as a compact sketch that keeps the meaning but throws away redundant pixel detail. Diffusion happens entirely in that small latent space, and only at the very end does the VAE decode the finished latent back into a full-resolution image. Because the latent is many times smaller, the whole process becomes dramatically cheaper.
Why this is Stable Diffusion
This technique — latent diffusion — is exactly what Stable Diffusion is. When people say “Stable Diffusion,” they mean a latent diffusion model. This is the guardrail worth nailing: Stable Diffusion does not run diffusion on pixels; it runs on a compressed latent space via a VAE. That compression is the whole reason image generation escaped the data center and landed on consumer GPUs in 2022, setting off the cultural explosion. Miss this, and you’ve missed the reason any of this reached your laptop.
From Words to Images: Conditioning
So far the model generates some plausible image. How does your prompt — “a corgi astronaut, oil painting” — steer it toward that image? Through a process called conditioning, and it requires a second component the diffusion model doesn’t have on its own.
The text encoder — a separate model that reads the prompt
A diffusion model cannot read English. It needs a text encoder — a completely separate neural network — to turn your prompt into a list of numbers (an embedding) that captures its meaning. Stable Diffusion uses CLIP, a model trained on hundreds of millions of image-caption pairs so that a picture and its description land near each other in a shared “meaning space.” Google’s Imagen uses a large language text encoder (T5) instead. Either way, this is a two-component system: text-to-image needs a text encoder AND a diffusion model — never one alone. It’s the same “turn meaning into vectors” idea behind semantic search, pointed at a paintbrush.
Cross-attention — how the prompt steers each step
That text embedding is injected into the U-Net at every denoising step through cross-attention: the image-in-progress “queries” the text, and the text “answers,” pulling the denoising toward regions of image-space that match the words. So the prompt isn’t consulted once — it guides every one of the dozens of denoising steps, continuously tugging the emerging picture toward your description.
Classifier-free guidance — the CFG slider
Finally, a knob you may have seen: guidance scale (or CFG). Introduced as classifier-free guidance, it controls how strongly the model obeys your prompt versus how much creative freedom it takes. Turn it up and the image adheres more tightly to the words but can become over-saturated or artifact-ridden; turn it down and you get more variety but looser prompt-following. That slider in your image tool is literally this parameter. It’s the dial between “do exactly what I said” and “surprise me.”
Sampling: The Speed-vs-Quality Tradeoff
We keep saying “step by step.” How many steps? This is diffusion’s defining weakness and the focus of years of research.
DDPM’s many steps
The original DDPM formulation used a lot of denoising steps — often on the order of a thousand — to go from static to a finished image. Each step is a full pass through the U-Net, so a thousand steps means a thousand passes. That’s why early diffusion was slow: generating one image could take a noticeable while, even on good hardware. Be honest about this — diffusion is not, in its classic form, instant.
DDIM and faster samplers
The 2020 Denoising Diffusion Implicit Models (DDIM) paper showed you could take bigger, smarter steps and get a comparable image in far fewer of them — dozens instead of a thousand — by reformulating the reverse process as a more direct path. A whole family of “samplers” (the Euler, DPM++, and UniPC methods you see in the settings) followed, each a different mathematical shortcut through the denoising trajectory. Roughly:
| Denoising steps | Speed | Typical quality |
|---|---|---|
| ~1,000 (classic DDPM) | Slowest | Excellent, but overkill |
| ~50 (DDIM / DPM++) | Fast | Very good — the common default |
| ~20 | Faster | Good; slight quality dip |
| ~4 (distilled/LCM) | Near-instant | Good for many uses (see next section) |
The trend is unmistakable: the field has spent enormous effort dragging the step count down, because fewer steps means faster, cheaper generation. Which brings us to the frontier.
The Frontier: How Modern Diffusion Got Fast
If you’ve tried an AI image tool recently and watched the picture appear almost the instant you stopped typing, you’ve seen the payoff of this research. The story of modern diffusion is largely the story of crushing the step count from ~1,000 to ~1.
Smarter solvers
First came better math for taking the same denoising path in fewer, larger steps. DPM-Solver and related methods treat denoising as solving a differential equation and apply advanced numerical shortcuts, getting quality results in ~10–20 steps. No retraining needed — just a cleverer route down.
Consistency models and LCM
The bigger leap was consistency models (2023), which are trained so that any point on the denoising path can jump straight to the finished image in a single step. Applied to Stable Diffusion, Latent Consistency Models (LCM) brought high-quality generation down to around 4 steps — an order-of-magnitude speedup that made real-time, type-and-watch generation feasible.
Distillation: Turbo and Lightning
Finally, distillation techniques teach a fast “student” model to mimic a slow, high-quality “teacher” in a handful of steps. Stability AI’s SD-Turbo and SDXL-Lightning produce good images in 1–4 steps, fast enough to update live as you type. The honest framing: these methods trade a little peak quality for a massive speed gain, and for most everyday uses the tradeoff is well worth it.
| Classic DDPM (2020) | Modern practice (2024+) | |
|---|---|---|
| Steps to generate | ~1,000 | 1–4 (distilled) to ~20–50 (quality) |
| Speed | Seconds to minutes | Near-instant |
| Runs on | Data-center GPUs | Consumer GPUs, even phones |
| Space it runs in | Pixels | Compressed latent (VAE) |
The Theory, Briefly: Noise, Score, and SDEs Are One Idea
You can skip this section and lose nothing practical — but it’s a lovely piece of the puzzle. Diffusion was discovered from three different directions that turned out to be the same thing.
- The thermodynamics view (Sohl-Dickstein, 2015) framed it as slowly destroying and then restoring structure, borrowing the physics of diffusion.
- The “predict the noise” view (DDPM, 2020) framed it as a simple denoising task — the version we’ve used throughout.
- The score-based view (Song & Ermon) framed it as learning the “score” — the direction toward more-realistic images at each noise level — and showed the whole process is a reverse stochastic differential equation (SDE).
A 2020 paper unified all three: predicting the noise, estimating the score, and solving a reverse SDE are three languages for one underlying idea. You don’t need the math, but the fact that independent lines of research converged on the same mechanism is part of why diffusion felt, in retrospect, inevitable. (For genuinely excellent deeper reading, the blogs of researchers Sander Dieleman and Yang Song are the gold standard.)
What’s Replacing Classic Diffusion: Flow Matching
Here’s a sign the field is still moving fast: the newest top-tier models don’t use classic diffusion at all — they use a close cousin called flow matching (or “rectified flow”).
Intuitively, classic diffusion learns a wiggly, roundabout path from noise to image. Flow matching trains the model to follow a straighter path from noise to data — and a straighter path can be traversed in fewer steps, which means faster generation and, in practice, often better quality. It’s the same broad idea (transform noise into a sample), reformulated for efficiency. Stable Diffusion 3 and the FLUX models use flow matching rather than the original DDPM formulation. If you learn diffusion and then hear “flow matching,” don’t be thrown: it’s the next refinement of the same core insight, not a different universe.
Measuring Quality: FID and Its Limits
How do you prove one image generator is “better” than another? You need a number — and that number comes with the same honesty caveats as every AI benchmark.
What FID measures
The standard metric is FID (Fréchet Inception Distance). Roughly, it runs a big batch of real images and a big batch of generated images through a pretrained vision network, summarizes each batch as a statistical distribution of features, and measures the distance between those two distributions. Lower FID = generated images are statistically closer to real ones. It rewards both quality and diversity, which is why it became the field’s yardstick.
“Diffusion beats GANs”
The turning point had a name. The 2021 paper Diffusion Models Beat GANs on Image Synthesis showed diffusion achieving lower (better) FID than the best GANs on standard benchmarks. That result — a slower, once-obscure method decisively outscoring the reigning champion — is what flipped the field. Within roughly a year, essentially every major image generator was diffusion-based.
FID’s blind spots
But FID is a proxy, not truth — exactly like top-5 error for images or BLEU for translation. It’s sensitive to fiddly details like how images are resized, it depends entirely on the feature network you plug in, and a lower FID doesn’t guarantee a human would prefer the images. Researchers have documented ways it gets gamed and have proposed replacements (Google’s 2024 “Rethinking FID” work, for instance). So read a headline FID the way you’d read any benchmark number: useful, comparable within the same setup, and not the same thing as “better pictures.” For prompt-alignment specifically, a complementary CLIP score measures how well the image matches the text — also a proxy, also imperfect.
Diffusion vs GANs vs VAEs
We’ve met all three generative families now; here they are side by side, since “how is diffusion different from a GAN?” is one of the most common questions.
| VAE | GAN | Diffusion | |
|---|---|---|---|
| Core mechanism | Encode → decode | Generator vs critic compete | Iteratively remove noise |
| Generation speed | Fast (one pass) | Fast (one pass) | Slower (many steps; now faster via distillation) |
| Training stability | Stable | Notoriously unstable | Stable |
| Image quality | Often blurry | Sharp | Sharp and diverse |
| Diversity | Good | Can collapse to few modes | Excellent |
| Peak era | 2013–2016 | 2014–2021 | 2021–present |
Diffusion won because it combined GAN-level sharpness with VAE-level training stability and strong diversity — paying only the price of slower sampling, which the frontier techniques have since largely paid down. (Fun connection: latent diffusion actually uses a VAE as its compression front-end, so diffusion didn’t so much beat the VAE as absorb it.)
Diffusion vs CNNs vs Transformers
This is the comparison that confuses the most people, so let’s be precise. You’ll hear “Stable Diffusion uses a CNN” and “the newest models use Transformers instead” and reasonably conclude these are three competing technologies. They’re not — and the confusion comes from comparing different kinds of thing.
A CNN and a Transformer are network architectures — engines. Diffusion is a generative process — a way of using an engine. A CNN is great at images; a Transformer is great at sequences and attention. Diffusion is neither — it’s the noise-then-denoise recipe that runs on one of those engines (a U-Net CNN, or a DiT Transformer).
| CNN | Transformer (enc-dec) | Diffusion | |
|---|---|---|---|
| What it is | A network architecture | A network architecture | A generative process |
| Native job | Recognize images (image → label) | Transform sequences (e.g. translation) | Create images (noise → image) |
| Discriminative or generative? | Usually discriminative | Either | Generative |
| Relationship | Used by diffusion (the U-Net) | Used by diffusion (DiT; text encoder) | Uses CNNs and Transformers |
So the honest answer to “diffusion vs CNN vs Transformer” is: it’s not a versus. Diffusion is the car; the CNN or Transformer is the engine under the hood. That’s why this hub ties LIG’s vision and language content together — diffusion sits on top of both. (We unpack this further in the dedicated companion piece.)
Controlling the Output: The Practical Ecosystem
Knowing how diffusion works is one thing; knowing how people actually use it is another — and it’s where a curious reader goes from “neat” to “I get this field.” Almost this entire ecosystem exists for one reason: Stable Diffusion’s weights were released openly in 2022, so anyone could not just use it but modify it. That single decision spawned a toolkit no closed model has.
Controlling composition — ControlNet
Out of the box, you can describe what appears but not where. ControlNet fixes that: you feed it a control image — a stick-figure pose, a depth map, an edge outline, a rough scribble — and the model generates an image that follows that structure while obeying your prompt. It’s the jump from “a knight, somewhere” to “a knight in exactly this pose.” ControlNet solved diffusion’s biggest practical weakness: precise, spatial control.
Teaching it new things — LoRA and DreamBooth
A base model doesn’t know your face, your product, or one specific art style. Two techniques teach it cheaply. DreamBooth fine-tunes the model on a handful of images of a subject so it can place that subject in new scenes. LoRA (Low-Rank Adaptation) does something similar far more efficiently — it trains a tiny add-on file (megabytes, not gigabytes) that plugs into the base model to add a character, face, or style. LoRAs are why there are hundreds of thousands of community-made add-ons for a single base model.
The knobs that matter
A quick glossary you’ll meet in any image tool: steps (more = slower, better up to a point), sampler (Euler, DPM++, DDIM — different shortcuts down the denoising path), CFG/guidance scale (prompt adherence vs freedom), seed (the starting noise — same seed and prompt give the same image), negative prompt (what to avoid), and denoising strength (in image-to-image, how far to wander from the input). Interfaces range from the feature-packed AUTOMATIC1111 to the node-based ComfyUI (increasingly the professional standard) to simplified apps like Fooocus.
The thesis to carry: this rich, controllable ecosystem is a direct consequence of open weights. Closed models are more polished out of the box but offer none of this modularity.
The Landscape: Who Makes the Models
The image-generation world splits into two camps, and the split explains almost everything about how the field developed.
| Model family | Company | Open or closed | Best known for |
|---|---|---|---|
| DALL·E 3, Sora | OpenAI | Closed API | Strong prompt-following; Sora (video) |
| Stable Diffusion / SDXL / SD3 | Stability AI | Open weights | Democratizing image gen; the whole open ecosystem |
| Midjourney (v6/v7) | Midjourney | Closed | Aesthetic quality (architecture undisclosed) |
| Imagen / Veo | Mostly gated | Photorealism; video (Veo) | |
| Firefly | Adobe | Closed | “Commercially safe” — trained on licensed data; Photoshop Generative Fill |
| FLUX.1 | Black Forest Labs | Open + Pro | State-of-the-art open image gen (2024) |
| Ideogram | Ideogram | Closed | Legible text inside images |
Two threads worth noticing. First, open-weights (Stable Diffusion, FLUX) vs closed-API (DALL·E, Midjourney, Imagen) is the field’s defining fault line: open unlocked the ControlNet/LoRA ecosystem and also the misuse; closed offers safety and polish but no control. Second, the researcher diaspora — just like the Transformer’s authors, diffusion’s key people scattered into startups: the latent-diffusion team founded Black Forest Labs (FLUX); DDPM’s Jonathan Ho co-founded Ideogram. (Note: Midjourney publishes nothing about its architecture, so we describe it as believed diffusion-based, not confirmed.)
Editing With Diffusion: Inpainting, Outpainting, and img2img
Diffusion doesn’t only generate from scratch — the same denoising machinery powers a suite of editing tools:
- Inpainting — regenerate a masked region (remove an object, fix a hand) while keeping the rest.
- Outpainting — extend an image beyond its borders, inventing plausible surroundings.
- Image-to-image (img2img) — start the denoising from a real image plus some noise instead of pure static, so the output resembles your input but shifts toward your prompt. The “denoising strength” knob controls how far it drifts.
A common misconception worth clearing up directly: Stable Diffusion is not searching the internet for your picture. It stores no images. Editing works by regenerating pixels using the model’s learned sense of what looks right — it’s generation, not retrieval. (That distinction is exactly why this sits apart from true semantic search.)
Beyond Images: Video, Audio, 3D, and Science
Here is the reason diffusion is a genuinely big deal and not just an image trick: the same “learn to remove noise” idea generalizes to almost anything you can add noise to.
- Video. Add the dimension of time and you get video generators — Stable Video Diffusion, Runway, Google’s Veo, and OpenAI’s Sora (a diffusion transformer). The hard part is temporal consistency: keeping a face or object coherent frame to frame. It’s the fastest-moving frontier.
- Audio and music. Treat a sound’s spectrogram (a visual representation of audio) like an image and diffusion generates speech and music — AudioLDM, Stable Audio, and diffusion-based text-to-speech vocoders like DiffWave. (Some popular music tools’ exact architectures are undisclosed.)
- 3D. Methods like DreamFusion use a trick called score distillation to turn a 2D image diffusion model into a generator of 3D objects from a text prompt.
- Science. This is the most consequential corner. Diffusion is used to design proteins (the Baker lab’s RFdiffusion) and molecules — generating plausible new structures the way it generates plausible images — and a diffusion module sits inside DeepMind’s AlphaFold3. Here “hallucinating something plausible” becomes a feature: proposing novel candidate molecules for scientists to test.
- Restoration. The quiet, genuinely-deployed corner: super-resolution (upscaling), deblurring, colorization, and denoising real photos — diffusion excels at “fill in the missing detail plausibly.”
The throughline: whatever the medium, the recipe is the same — corrupt it with noise during training, learn to reverse the corruption, then generate by denoising from scratch. That generality is why diffusion, not any single image model, is the thing to understand — and why how diffusion models work is worth learning once, deeply, as you just have.
The Honest Limits: Slowness, Copyright, Deepfakes, and Bias
A guide that only sold the magic would mislead you. Diffusion has real limitations and real controversies.
- It’s computationally heavy. Even with fast samplers, high-quality generation is more expensive than a single-pass model, and training frontier models still costs enormous compute — one reason it leans on the GPUs that made deep learning possible.
- It can be confidently wrong. Like all generative AI, it produces plausible output, not correct output — hence the infamous extra fingers, garbled text, and impossible physics. It has no understanding to check itself against.
- Copyright and consent are unresolved. Models were trained on huge web-scraped image sets (like LAION), raising live, unsettled questions about artist consent and style mimicry. There are active lawsuits (Getty Images v. Stability AI; artists v. Stability, Midjourney, and others). Adobe’s “commercially safe” Firefly is the industry’s attempt to sidestep this by training on licensed data. (These cases are moving; verify the current status of any specific ruling before relying on it.)
- Deepfakes and misuse are a genuine harm. The same tool that makes a birthday card can fabricate a convincing “photo” of a real person — see why your face is data you can’t change. Defenses exist (C2PA content credentials, Google’s SynthID watermarking, safety filters) but are imperfect and, on open models, removable. Practical media literacy — be skeptical of unverified images — matters more than ever.
- It inherits bias. Trained on internet images, it reproduces their skews. Responsible use means treating outputs critically, not neutrally.
None of this makes diffusion less remarkable. It makes it a powerful tool that, like any powerful tool, has to be understood and handled with care.
A Short History
To close the loop, the timeline in brief:
- 2015 — Sohl-Dickstein and colleagues introduce diffusion probabilistic models, inspired by thermodynamics. Almost nobody notices; GANs rule.
- 2020 — Ho, Jain, and Abbeel’s DDPM makes diffusion produce genuinely great images. Song and Ermon develop the parallel score-based view.
- 2021 — Dhariwal and Nichol prove Diffusion Models Beat GANs. The tide turns.
- 2022 — The explosion: DALL·E 2, Midjourney, and — crucially — the open-source release of Stable Diffusion put image generation in everyone’s hands.
- 2023–2026 — The ecosystem (ControlNet, LoRA), the speedups (consistency models, distillation), video (Sora), and the shift to flow matching (SD3, FLUX).
It’s the same shape as our other origin stories — AlexNet and the Transformer: an unfashionable idea, ignored for years, that suddenly won everything. (The full human story is its own piece in this series.)
❓ Frequently Asked Questions
How do diffusion models work, in simple terms? A diffusion model learns to turn random noise into an image by removing noise step by step. During training it takes real images, adds noise until they’re pure static, and learns to reverse that. To generate a new image, it starts from fresh random static and denoises repeatedly, “carving” a new, plausible picture out of the noise using its learned sense of what real images look like.
Is Stable Diffusion searching the internet for my image? No. It stores no images and searches nothing. It generates every pixel by denoising, using patterns it learned during training. Even editing tools like inpainting regenerate the region rather than retrieving anything — it’s generation, not search.
Why does Stable Diffusion need a VAE? Because running diffusion directly on millions of pixels is too slow and expensive. The VAE compresses the image into a small “latent” summary; diffusion runs in that compact space, and the VAE decodes the result back to full resolution at the end. This “latent diffusion” trick is what let image generation run on consumer GPUs.
How does a text prompt control the image? A separate text encoder (CLIP, or T5 in some models) turns your prompt into a numeric embedding, which is injected into the denoiser at every step via cross-attention — continuously steering the emerging image toward your words. The guidance scale (CFG) controls how strongly it obeys the prompt.
Is diffusion the same as a CNN or a Transformer? No — it’s a different kind of thing. A CNN and a Transformer are network architectures; diffusion is a generative process that runs on top of one of them (its denoiser is usually a CNN “U-Net,” or a Transformer in newer “DiT” models). So diffusion uses CNNs and Transformers rather than competing with them.
Why are AI images sometimes slow to generate? Classic diffusion needs many denoising steps — up to ~1,000 — each a full pass through the network. Faster samplers (DDIM, DPM++) cut this to dozens, and newer distillation/consistency methods (SD-Turbo, LCM) get it down to 1–4 steps for near-instant generation, trading a little peak quality for speed.
How is diffusion different from GANs? GANs pit a generator against a critic in a single fast pass, but are unstable to train and can lack diversity. Diffusion iteratively denoises — slower, but more stable to train and higher quality with more variety. Around 2021, diffusion overtook GANs as the dominant image-generation method.
What can diffusion models make besides images? Video (Sora, Veo, Runway), audio and music (via spectrograms), 3D objects (DreamFusion), and even proteins and molecules for scientific research (RFdiffusion, and a module inside AlphaFold3). The same “learn to denoise” recipe generalizes to any data you can add noise to.
📚 Further Learning
- What Is Generative AI? — the umbrella this sits under, alongside language and other generative models.
- How AI Learned to See and What Is “Attention” in AI? — the CNN and Transformer engines diffusion runs on.
- Midjourney Tips and Tricks — hands-on prompting once you understand the machinery.
- Best Deep Learning Books — to go deeper on the foundations. For diffusion specifically, the free standouts are Lilian Weng’s “What are Diffusion Models?”, Jay Alammar’s “Illustrated Stable Diffusion,” and the Hugging Face Diffusers course.
📚 Recommended Resources
Disclosure: some links below are Amazon affiliate links. If you buy through them, I earn a small commission — at no extra cost to you — which helps keep this site running.
Deep Learning with Python
François Chollet · Book · ~$50 on Amazon
The clearest hands-on introduction to the neural networks underneath diffusion — including the generative models (autoencoders, GANs) that set the stage for it. From the creator of Keras, and the best next step once this guide makes the concepts click.
Deep Learning
Ian Goodfellow, Yoshua Bengio & Aaron Courville · Book · ~$80 on Amazon
The foundational reference for the theory beneath it all — the U-Nets, the probability, the generative modeling. (Also free to read at deeplearningbook.org; buy the print copy for the shelf.) Note: Goodfellow also co-invented the GANs that diffusion later overtook.
Artificial Intelligence: A Guide for Thinking Humans
Melanie Mitchell · Book · ~$28 on Amazon
The clear-eyed companion for the bigger picture — what these generative systems actually do and don’t do, and how to think about the copyright, deepfake, and “is it creativity?” questions this guide raises.