Transfer Learning With a Pretrained CNN (What People Actually Do)

Here is the honest version of applied computer vision in 2026: almost nobody trains a convolutional network from scratch, and transfer learning with a pretrained CNN is what practitioners actually do all day. You download a model that already learned to see on more than a million labeled images, keep the part that learned general visual features, and retrain a thin new layer on your own photos. That’s the whole trick. “Building an image classifier” has quietly become the discipline of adapting one — and the reason isn’t laziness, it’s data economics. You almost never have ImageNet-scale labeled data, and you almost never need to.

This is the hands-on companion to our Learn-pillar cornerstone How AI Learned to See, which tells the story of the 2012 AlexNet moment. Here we do the engineering: the two standard recipes, the preprocessing you cannot get wrong, and the three lines of code that do the actual work.

🧠 Why Nobody Trains a CNN From Scratch

PyTorch’s own transfer-learning tutorial puts it bluntly: “very few people train an entire Convolutional Network from scratch … because it is relatively rare to have a dataset of sufficient size.” That sentence is the entire justification for this article. AlexNet — the eight-layer network (5 convolutional + 3 fully-connected, ~60 million parameters in the original paper) that won ILSVRC-2012 — needed 1.2 million images across 1,000 classes and roughly five to six days on two GPUs to reach its reported numbers. You don’t have that dataset, that time budget, or a reason to spend either.

And you wouldn’t ship AlexNet even if you did. It’s a superb thing to understand and rebuild once and a poor thing to deploy: ResNet (2015) made 100-plus-layer networks trainable and cut ImageNet top-5 error to around 3.6%, EfficientNet pushed accuracy-per-FLOP much further, and Vision Transformers became the default for large-scale image work. AlexNet’s role today is pedagogical — a historical baseline, not a production choice. Transfer learning is how you get a modern backbone’s hard-won features onto your problem for the cost of an afternoon.

📦 What “Pretrained” Actually Buys You

A pretrained CNN is two things bolted together. There’s a feature extractor — the stack of convolutional layers that turns a raw image into a dense vector of visual features (edges, textures, parts, whole-object patterns, roughly in that order as you go deeper). And there’s a classifier head — the final fully-connected layer(s) that map those features to the 1,000 ImageNet categories.

The features are the treasure; the head is disposable. Those convolutional filters learned general-purpose visual structure from a million images, and that structure transfers remarkably well to images ImageNet never contained — your product photos, your X-rays, your drone footage. In fact, the activations of the penultimate layer are an image embedding: a compact numeric fingerprint you can compare, cluster, or feed to a downstream model. If that idea is fuzzy, our explainer What Is an Embedding? is the foundation — a CNN’s next-to-last layer is one of the cleanest examples of an embedding you’ll meet. Transfer learning is just the practice of keeping the extractor and replacing the head.

In torchvision, loading one is a single line:

from torchvision import modelsmodel = models.resnet18(weights='IMAGENET1K_V1')

That downloads ImageNet-pretrained weights. (The same call works for models.alexnet(weights='IMAGENET1K_V1') — torchvision’s AlexNet reports 56.5% top-1 / 79.1% top-5 accuracy and 61,100,840 parameters, slightly above the paper’s 60M because torchvision implements the single-stream “one weird trick” variant rather than the literal 2012 two-tower diagram.)

🎛️ Two Recipes: Frozen Extractor vs Full Fine-Tune

There are exactly two mainstream recipes, and choosing between them is mostly about how much data you have.

Recipe 1 — Fixed feature extractor. Freeze the entire backbone so its weights never update, and train only a fresh classifier head. This treats the pretrained CNN as a fixed function that turns images into features, and asks a small new layer to learn the mapping from those features to your classes. It’s fast, it’s cheap, it barely overfits, and it’s the right default when you have a small dataset (hundreds to low thousands of images).

for param in model.parameters():    param.requires_grad = False   # freeze the backbone

Recipe 2 — Full fine-tuning. Load the pretrained weights, swap the head, and train everything — but at a low learning rate, so you nudge the existing features rather than blowing them away. This can reach higher accuracy when your data differs meaningfully from ImageNet (medical scans, satellite imagery), but it needs more data and more care to avoid overfitting.

The honest heuristic: start with the frozen extractor. It’s the strong baseline you should beat before spending compute on full fine-tuning. A common middle path is to freeze first, train the head to convergence, then unfreeze the top few convolutional blocks and fine-tune the whole thing at a very low rate.

🔧 The Head Swap Is Three Lines

Whichever recipe you pick, you replace the 1,000-way ImageNet head with one sized to your problem. For a ResNet, the final layer is named fc, so you read its input width and assign a new Linear:

import torch.nn as nnnum_ftrs = model.fc.in_featuresmodel.fc = nn.Linear(num_ftrs, NUM_CLASSES)

That’s it — three lines, and the “hello world” of applied CV. The new layer starts with random weights (so its parameters are trainable even under Recipe 1), while everything behind it carries ImageNet’s learned features forward.

The pattern generalizes; only the head’s name changes per architecture. AlexNet’s classifier is an nn.Sequential, and the layer you want is the last one, index 6:

model.classifier[6] = nn.Linear(4096, NUM_CLASSES)

Same idea, different attribute path. When you meet a new backbone, the only research task is “what is the final layer called, and how wide is its input?” — then you swap it.

🧪 Get the Preprocessing Exactly Right

This is the step that silently ruins more transfer-learning runs than any other. A pretrained model expects its inputs to look statistically like the data it trained on. Feed it images normalized differently and accuracy quietly collapses — no error, no crash, just bad numbers. The ImageNet preprocessing pipeline every torchvision classification model expects is fixed and non-negotiable:

1. Resize the shorter side to 256 pixels.
2. Center-crop to 224 × 224 (the network’s expected input size).
3. Normalize each channel with mean [0.485, 0.456, 0.406] and std [0.229, 0.224, 0.225] — the per-channel statistics of the ImageNet training set.

from torchvision import transforms preprocess = transforms.Compose([    transforms.Resize(256),    transforms.CenterCrop(224),    transforms.ToTensor(),    transforms.Normalize(mean=[0.485, 0.456, 0.406],                         std=[0.229, 0.224, 0.225]),])

Those magic numbers aren’t arbitrary — they’re the actual channel statistics of ImageNet, and they must match the model you loaded. For training you’d swap the deterministic center-crop for augmentation (RandomResizedCrop(224) plus RandomHorizontalFlip) to fight overfitting, but keep the same Normalize. For validation and inference, use the exact resize-256 → crop-224 → normalize path above. Get these three numbers wrong and nothing else you do will save the run.

📉 A Sane Training Loop

Point torchvision.datasets.ImageFolder at a directory whose subfolders are your class names, wrap it in a DataLoader, and you have batches. From there the loop is ordinary supervised learning — cross-entropy loss, an optimizer, a few epochs — with two transfer-specific habits:

• Use a low learning rate. You’re adapting, not training from zero. Start around 1e-3 for a frozen-extractor head and drop to 1e-4 or lower when fine-tuning the backbone. Add a scheduler (a step decay every few epochs is fine) so the rate tapers as you converge.
• Only pass trainable parameters to the optimizer when you froze the backbone — optim.SGD(model.fc.parameters(), lr=1e-3, momentum=0.9) — so you’re not wasting work computing gradients you’ll throw away.

Then read a confusion matrix, not just top-line accuracy. On your own data, accuracy hides which classes the model confuses with which — and that’s where transfer learning either shines or exposes that two of your categories are genuinely ambiguous. For evaluation metrics done properly, the Measure pillar has you covered.

⚠️ What You’d Actually Reach For

To be explicit, so nobody deploys the wrong thing: use AlexNet to learn the mechanics, not to ship. For a real project, a pretrained ResNet-18 or ResNet-50 is the workhorse default — well-understood, well-supported, and strong. Reach for EfficientNet when you’re compute- or latency-constrained, and a Vision Transformer when you have a lot of data and want the current state of the art. The transfer-learning recipe is identical across all of them: load pretrained weights, freeze or fine-tune, swap the head, match the preprocessing. Learn it once on ResNet-18 and you’ve learned it for every backbone that will come after.

❓ Frequently Asked Questions

Should I freeze the backbone or fine-tune the whole network? Start frozen. A fixed feature extractor with a fresh head is fast, resists overfitting, and is the baseline you should beat before spending compute on full fine-tuning. Move to fine-tuning (at a low learning rate) when you have more data or when your images differ a lot from ImageNet.

Why does my accuracy collapse even though the code runs? Almost always preprocessing. A pretrained CNN expects resize-256 → center-crop-224 → normalize with ImageNet’s mean [0.485, 0.456, 0.406] and std [0.229, 0.224, 0.225]. Skip or change the normalization and the model sees inputs unlike anything it trained on — no error, just bad numbers.

Can I just use AlexNet since it’s the classic example? You can, and torchvision ships it as a one-liner, but you shouldn’t for production. It’s superseded — ResNet, EfficientNet, and Vision Transformers beat it on both accuracy and efficiency. Use AlexNet to understand the architecture; reach for a modern backbone to actually deploy.

How much data do I need for transfer learning? Far less than training from scratch — often a few hundred images per class for a frozen-extractor setup. That gap is the entire reason the technique exists: the pretrained backbone already learned general visual features from a million-plus images, so your data only has to teach the final mapping.

Is the pretrained model’s second-to-last layer really an embedding? Yes. Those penultimate activations are a dense feature vector — an image embedding — which is why the same backbone powers classification, similarity search, and clustering. Swapping the head just changes what you do with that embedding.

📚 Further Reading

• How AI Learned to See — the story of AlexNet and the 2012 ImageNet moment; the conceptual cornerstone this build tutorial pairs with.
• What Is an Embedding? — why a CNN’s penultimate layer is an embedding, and what that unlocks downstream.
• PyTorch’s Transfer Learning for Computer Vision tutorial and the torchvision.models documentation — the primary sources for the two recipes, the head swap, and the exact preprocessing constants above.

---

📚 Recommended Resources

Disclosure: some links below are Amazon affiliate links. If you buy through them, I earn a small commission — at no extra cost to you — which helps keep this site running.

---

Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow

Aurélien Géron · Book · ~$65 on Amazon

The most practical shelf reference for exactly this workflow — loading pretrained models, freezing layers, fine-tuning, and getting the preprocessing right. If you want one book that turns “I read a tutorial” into “I can build it,” this is it.

---

Deep Learning with Python

François Chollet · Book · ~$60 on Amazon

Written by the creator of Keras, with some of the clearest explanations anywhere of why transfer learning works — feature reuse, fine-tuning, and the intuition behind a convolutional feature extractor. The best conceptual bridge from “run the code” to “understand the code.”

---

Deep Learning

Ian Goodfellow, Yoshua Bengio & Aaron Courville · Book · ~$80 on Amazon

The canonical graduate-level foundation — convolutional networks, regularization, optimization — for readers who want the full mathematical grounding under the recipes. (The full text is also free to read in-browser at deeplearningbook.org; buy the print edition if you’d rather keep it on the desk.)