Learning how AI image generators actually work looks like it should require a PhD. It doesn’t. If you want to know how to learn diffusion models — the technology behind Midjourney, DALL·E, and Stable Diffusion — the remarkable truth is that the very best material is free, and there’s exactly one paid book worth owning if you want the full practitioner treatment. The trick is climbing the ladder in the right order: intuition first, then concepts, then building.
This is that ladder, curated and ordered. Most of it costs nothing; where a purchase earns its place, we say so plainly. And a quick note on what this isn’t: it’s the diffusion-specific list. For the broader field we keep separate guides to the best deep learning books, a free deep learning path, and a Transformer/LLM reading ladder.
The Ladder at a Glance
| Rung | Best resource | Cost |
|---|---|---|
| 1. Visual intuition | Jay Alammar, “The Illustrated Stable Diffusion” | Free |
| 2. The concepts | Lilian Weng, “What are Diffusion Models?” | Free |
| 3. A textbook chapter | Simon Prince, Understanding Deep Learning (ch. on diffusion) | Free to read |
| 4. Hands-on build | Hugging Face Diffusers course; fast.ai Part 2 | Free |
| 5. The one paid book | David Foster, Generative Deep Learning (2nd ed.) | Paid |
Climb in order and each rung makes the next one easier. Here’s why each earns its place.
Rung 1: Free Visual Intuition (Start Here)
Before a single equation, spend an hour with Jay Alammar’s “The Illustrated Stable Diffusion.” Alammar is the same author behind the legendary “Illustrated Transformer,” and he brings the same gift: building up the whole Stable Diffusion pipeline — the text encoder, the U-Net denoiser, the VAE — with clear diagrams, one piece at a time. It’s the fastest way to get the shape of how a prompt becomes a picture before you dive into the mechanics. If you read only one thing, read this.
Rung 2: Free Concepts (The Canonical Explainer)
Once the pipeline makes visual sense, level up with Lilian Weng’s “What are Diffusion Models?” — the single most-cited technical blog post on the topic, written by a prominent AI researcher. It’s more mathematical than Alammar’s piece but still readable, and it connects the dots between the different views of diffusion (the “add noise / remove noise” framing and the score-based view). It’s the resource researchers themselves point newcomers to. Free, thorough, authoritative.
Rung 3: A Free Textbook Chapter
If you want the rigor of a textbook without buying one, Simon Prince’s Understanding Deep Learning (MIT Press) has a dedicated chapter on diffusion models — and the entire book is free to read online (the print copy is a paid purchase for the shelf). It’s the same “free-to-read, paid-to-own” pattern we recommend for other classic references, but newer and diffusion-specific. This is your bridge from “I get the intuition” to “I understand the math well enough to trust it.”
Rung 4: Free Hands-On Building
There’s a kind of understanding you only get by building. Two free, world-class options:
- The Hugging Face Diffusers course and docs. Hugging Face makes the library nearly everyone uses to run diffusion models, and their course walks you from concepts to actually generating and fine-tuning images in code. It’s the practical standard.
- fast.ai’s “Part 2: Deep Learning Foundations to Stable Diffusion.” This free course famously builds Stable Diffusion’s components up from foundations — the deep end, but deeply rewarding if you want to truly get it rather than just call an API.
Either one turns diffusion from an abstraction into something you’ve run with your own hands. That’s the rung that changes what you’re capable of.
Rung 5: The One Paid Book Worth Owning
If you prefer a single, well-structured book to tie it all together, the standout is David Foster’s Generative Deep Learning (2nd edition). It covers the whole family of generative models — autoencoders, GANs, and diffusion — with clear explanations and hands-on code, placing diffusion in the context of what came before it. It’s the best “one book on my shelf about generative AI” choice, and the 2nd edition added diffusion coverage precisely because the field moved. If you buy one book from this list, make it this one.
(A note on honesty: this is a genuinely good list precisely because so little of it costs money. Don’t feel you need to buy anything to learn diffusion well — rungs 1 through 4 are free and excellent. The book is for people who specifically prefer a bound, structured treatment.)
How to Actually Use This Ladder
Match the climb to your goal:
- “I just want to understand what everyone’s talking about.” Rungs 1 and 2 (Alammar, then Weng). A couple of free hours, and you’ll understand diffusion better than most people who use it daily.
- “I want to build with it.” Add Rung 4 (Hugging Face Diffusers) — the practical, code-first path.
- “I want to deeply, mathematically get it.” All five rungs, ending with the fast.ai course and Foster’s book for structure.
And if you want the primary sources, they’re free too: the original papers (the 2015 diffusion paper, the 2020 DDPM paper, the 2022 latent diffusion paper) are all on arXiv. Read them after an explainer or two, when you have the mental model to appreciate them. Our own comprehensive guide to how diffusion models work is the perfect primer to read first.
The Fastest Path: A Weekend Plan
If you want a concrete schedule rather than a menu, here’s a two-day plan that takes you from zero to genuine understanding — entirely free.
Saturday: get the picture
Spend the morning on Alammar’s “Illustrated Stable Diffusion,” slowly, letting the diagrams sink in. In the afternoon, read Lilian Weng’s post — don’t worry about following every equation on the first pass; aim to recognize the ideas (forward noising, reverse denoising, the score view) rather than derive them. By Saturday evening you’ll understand what a diffusion model is better than most people who use one daily.
Sunday: make one
Spend Sunday in the Hugging Face Diffusers course, actually running code — generate a few images, change the prompt, adjust the guidance scale and the number of steps, and watch what happens. Nothing cements the concepts like turning the knobs yourself and seeing the noise resolve into a picture. You don’t need to understand every line; you need to feel the pipeline work.
That’s it — one weekend, zero dollars, and you’ll have both the mental model and hands-on experience. From there, Prince’s textbook chapter and Foster’s book are for going deeper whenever the itch strikes. The point of the ladder isn’t to climb it all at once; it’s to always know which rung comes next.
❓ Frequently Asked Questions
What’s the best way to learn diffusion models? Climb a ladder: start with free visual intuition (Jay Alammar’s “Illustrated Stable Diffusion”), then free concepts (Lilian Weng’s “What are Diffusion Models?”), then a free textbook chapter (Simon Prince’s Understanding Deep Learning), then build hands-on (the Hugging Face Diffusers course or fast.ai Part 2). Add David Foster’s Generative Deep Learning if you want one structured book.
Can I learn diffusion models for free? Yes — remarkably well. The best material is free: Alammar’s illustrated guide, Weng’s canonical blog post, Prince’s free-to-read textbook chapter, the Hugging Face Diffusers course, fast.ai’s Stable Diffusion course, and the original papers on arXiv. You don’t need to spend anything to understand diffusion deeply.
Do I need to be good at math to learn diffusion models? Not to understand the ideas. The intuition (add noise, then learn to reverse it) needs no math, and resources like Alammar’s cover it visually. You only need real math if you want to implement the algorithms from scratch — and even then, courses like fast.ai’s build it up gradually for motivated learners.
What’s the single best book on diffusion models? For a structured, code-first treatment, David Foster’s Generative Deep Learning (2nd edition) is the standout — it covers diffusion alongside GANs and autoencoders, placing it in context. But note that the strongest free resources (Alammar, Weng, Hugging Face) rival any book, so start there.
How is this different from your other AI book lists? This list is specifically about diffusion models and AI image generation. Our best deep learning books list is broader, learn deep learning for free covers no-cost resources across the whole field, and books to learn about transformers is the language-model equivalent of this ladder.
📚 Further Reading
- How Diffusion Models Work: A Comprehensive Guide — the free LIG primer to read before (and alongside) these resources.
- Best Deep Learning Books and Learn Deep Learning for Free — the broader field, and the no-cost path.
- Books to Learn About Transformers — the sibling ladder for language models.
📚 Recommended Resources
Disclosure: some links below are Amazon affiliate links. If you buy through them, I earn a small commission — at no extra cost to you — which helps keep this site running. The strongest resources for learning diffusion are free (see the article) — these are for readers who prefer a bound, structured treatment.
Generative Deep Learning (2nd Edition)
David Foster · Book · ~$60 on Amazon
The one diffusion-focused book worth owning. Covers the whole generative family — autoencoders, GANs, and diffusion — with clear explanations and hands-on code, placing diffusion in the context of what came before it. If you buy one book from this ladder, make it this.
Deep Learning with Python
François Chollet · Book · ~$50 on Amazon
The clearest hands-on introduction to the neural networks underneath diffusion — the autoencoders, the attention, the generative modeling. From the creator of Keras, and a gentle on-ramp before Foster’s deeper treatment.
Hands-On Machine Learning with Scikit-Learn, Keras & TensorFlow
Aurélien Géron · Book · ~$80 on Amazon
The workhorse reference that takes you from ML basics all the way to generative models with production-ready code — ideal for turning “I understand diffusion” into “I can build with it.”