The fastest way to actually understand computer vision is to build a CNN image classifier by hand, once, from the ground up — and the best skeleton to build is AlexNet, the eight-layer network that won ILSVRC-2012 and kicked off the deep-learning era. It is small enough to read end to end, historically load-bearing, and it maps almost line-for-line onto a page of PyTorch. In this build we define AlexNet’s two blocks layer by layer, tie every line back to the 2012 paper, train it on a slice of CIFAR-10, and finish honestly: this is a teaching exercise, not a model you would ship.
If you want the story of why 2012 mattered, our Learn cornerstone How AI Learned to See tells it. And if you’re here to solve a real problem rather than learn the mechanics, skip straight to the sibling of this article — Transfer Learning With a Pretrained CNN — which is what practitioners actually do all day. This piece is deliberately the other path: the from-scratch build you do to understand what transfer learning is reusing.
🧱 The Architecture We’re Rebuilding
AlexNet is best understood as “LeNet, scaled up, plus three tricks that made scaling work.” The conv → pool → conv → … → fully-connected → softmax skeleton is Yann LeCun’s 1990s LeNet; AlexNet did not invent the convolutional network. What it proved was that a deep CNN, trained on a lot of data with a couple of then-new ideas, could win at ImageNet scale. Per the NIPS 2012 paper it has “60 million parameters and 650,000 neurons, consists of five convolutional layers, some of which are followed by max-pooling layers, and three fully-connected layers with a final 1000-way softmax.”
Two structural facts drive the whole build. First, the convolution kernels shrink with depth: an 11×11 filter with stride 4 up front (96 of them), then 5×5 (256), then three stacked 3×3 layers (384, 384, 256). Big receptive field early to survey the image cheaply, small filters later to refine detail. Second, and this is the surprising part, roughly 58 of the ~60 million parameters live in the two 4096-wide fully-connected layers, not in the convolutional feature extractor. The part doing the “seeing” is small; the part doing the “deciding” is enormous. That imbalance is exactly what later architectures attacked, and it’s why AlexNet’s checkpoint is heavy for its accuracy.
🔩 The features Block, Layer by Layer
PyTorch AlexNet is two nn.Sequential containers: features (the convolutional extractor) and classifier (the fully-connected head). Here is the extractor, written to follow the paper’s channel counts, with each design choice annotated:
import torchimport torch.nn as nn class AlexNet(nn.Module): def __init__(self, num_classes=10): super().__init__() self.features = nn.Sequential( # Layer 1 — 11x11 kernels, stride 4, 96 filters nn.Conv2d(3, 96, kernel_size=11, stride=4, padding=2), nn.ReLU(inplace=True), nn.LocalResponseNorm(size=5, alpha=1e-4, beta=0.75, k=2), # LRN nn.MaxPool2d(kernel_size=3, stride=2), # overlapping pool # Layer 2 — 5x5, 256 filters nn.Conv2d(96, 256, kernel_size=5, padding=2), nn.ReLU(inplace=True), nn.LocalResponseNorm(size=5, alpha=1e-4, beta=0.75, k=2), nn.MaxPool2d(kernel_size=3, stride=2), # Layers 3-5 — three stacked 3x3 convs, no pooling in between nn.Conv2d(256, 384, kernel_size=3, padding=1), nn.ReLU(inplace=True), nn.Conv2d(384, 384, kernel_size=3, padding=1), nn.ReLU(inplace=True), nn.Conv2d(384, 256, kernel_size=3, padding=1), nn.ReLU(inplace=True), nn.MaxPool2d(kernel_size=3, stride=2), ) self.avgpool = nn.AdaptiveAvgPool2d((6, 6)) # pin the map to 6x6 before flattenRead it against the paper and every line has a source:
nn.ReLU— the non-saturatingmax(0, x)activation. AlexNet did not invent ReLU; it popularized it for deep CNNs after reporting a four-layer network reached 25% training error on CIFAR-10 roughly six times faster with ReLUs than withtanh. That speed is why every layer above uses it.nn.LocalResponseNorm— Local Response Normalization, a lateral-inhibition scheme the paper credited with a ~1–2% error reduction. Include it here for historical fidelity, but know that it has since been abandoned; nobody adds LRN to new networks (more on that below).nn.MaxPool2d(kernel_size=3, stride=2)— overlapping pooling, because the 3-pixel window is wider than the 2-pixel stride. The paper reported this slightly reduced error and made the model marginally harder to overfit.AdaptiveAvgPool2d((6, 6))— a modern robustness touch (it’s what torchvision uses) that forces the final feature map to 6×6 no matter the exact input size, so the flatten into the classifier always lines up.
🎩 The classifier Block (Where the Weight Lives)
Now the fully-connected head — the part that holds nearly all the parameters:
self.classifier = nn.Sequential( nn.Dropout(p=0.5), # dropout in the first two FC layers nn.Linear(256 * 6 * 6, 4096), # 9216 -> 4096 nn.ReLU(inplace=True), nn.Dropout(p=0.5), nn.Linear(4096, 4096), nn.ReLU(inplace=True), nn.Linear(4096, num_classes), # 1000-way in the paper; 10 for CIFAR-10 ) def forward(self, x): x = self.features(x) x = self.avgpool(x) x = torch.flatten(x, 1) return self.classifier(x)The 256 6 6 = 9216 input width is where the extractor’s final 6×6×256 feature map gets flattened into a vector. Then two 4096-wide layers do the heavy lifting — and that’s where the parameter mass hides: a single Linear(9216, 4096) layer alone is ~37 million weights.
Two annotations matter here. nn.Dropout(p=0.5) sits in front of each of the first two fully-connected layers, randomly zeroing half the activations during training to fight overfitting. Dropout was not an AlexNet invention either — it was a then-recent idea from the same Hinton group — but this is one of its first high-profile uses. And that final Linear(4096, num_classes) is the entire reason transfer learning exists: swap this one layer and you’ve repurposed the whole network. That’s precisely the move the sibling transfer-learning guide is built around. Note too that the activations feeding that last layer — the 4096-dimensional vector — are an image embedding, the same idea our explainer What Is an Embedding? unpacks. A CNN’s penultimate layer is one of the cleanest embeddings you’ll ever meet.
🏋️ Training It on CIFAR-10
Here’s the honest friction: AlexNet expects 224×224 inputs, and CIFAR-10 images are 32×32. The faithful way to keep the architecture intact is to upsample the small images to 224 in the transform pipeline. It’s wasteful, but it means the network you built stays literally the paper’s network rather than a shrunken imitation.
from torch.utils.data import DataLoaderfrom torchvision import datasets, transforms train_tf = transforms.Compose([ transforms.Resize(224), # 32x32 -> 224x224 transforms.RandomHorizontalFlip(), # cheap augmentation, paper-style transforms.ToTensor(), transforms.Normalize((0.4914, 0.4822, 0.4465), (0.2470, 0.2435, 0.2616)),]) train = datasets.CIFAR10(root="data", train=True, download=True, transform=train_tf)loader = DataLoader(train, batch_size=128, shuffle=True, num_workers=2) device = "cuda" if torch.cuda.is_available() else "cpu"model = AlexNet(num_classes=10).to(device)optimizer = torch.optim.SGD(model.parameters(), lr=0.01, momentum=0.9, weight_decay=5e-4) # the paper's settingsloss_fn = nn.CrossEntropyLoss() for epoch in range(5): model.train() for images, labels in loader: images, labels = images.to(device), labels.to(device) optimizer.zero_grad() loss = loss_fn(model(images), labels) loss.backward() optimizer.step() print(f"epoch {epoch}: last-batch loss {loss.item():.3f}")The optimizer settings — SGD, learning rate 0.01, momentum 0.9, weight decay 5e-4 — are the paper’s own. To keep a first run quick on a laptop, wrap the dataset in a torch.utils.data.Subset of a few thousand images and drop to two or three epochs. What you should expect: the loss falls and accuracy climbs comfortably past the 10% you’d get from random guessing on ten classes. What you should not expect is anything near the paper’s ImageNet numbers. Those required 1.2 million images across 1,000 classes and roughly five to six days on two GPUs — and even then, the famous 15.3% top-5 figure was a seven-model ensemble, not one network. A single AlexNet scored 18.2%. This build is here to make the architecture concrete, not to compete.
📋 Kept vs Dropped Since 2012
If you take one table away from this build, take this one. Sorting AlexNet’s ingredients by what survived is the most useful mental model a CNN builder can have:
| AlexNet 2012 ingredient | Status today | What refined or replaced it |
|---|---|---|
| ReLU activation | Kept | Still the default; descendants like GELU |
| Dropout in FC layers | Kept (lighter touch) | Often reduced once normalization is in the conv stack |
| Data augmentation (crops, flips, PCA color) | Kept & expanded | RandAugment, mixup, CutMix |
| Overlapping max-pooling | Mostly folded away | Strided convolutions do the downsampling |
| Local Response Normalization (LRN) | Dropped | Batch normalization (2015) did the job better |
| Two-GPU model split | Dropped | The net fits on one modern GPU; single-stream |
| Giant 4096-wide FC layers | Dropped | Global average pooling (GoogLeNet, ResNet) |
A note on two of those. The two-GPU split in the original diagram was a memory workaround, not a design principle — the model didn’t fit in a single 2012 GPU’s 3 GB, so the authors spread it across two GTX 580s. Modern reimplementations, including torchvision’s, run single-stream; that’s also why torchvision’s AlexNet reports 61,100,840 parameters rather than the paper’s ~60M — it follows the trimmed single-stream “one weird trick” formulation, not the literal two-tower drawing. And LRN is genuinely gone: batch normalization arrived in 2015 and quietly retired it. Don’t retrofit batch norm onto “the 2012 model,” though — it wasn’t part of AlexNet.
⚠️ This Is a Learning Build, Not a Production Model
Say it plainly so nobody deploys the wrong thing: AlexNet is superseded. ResNet (2015) made 100-plus-layer networks trainable with residual connections and pushed ImageNet top-5 error to around 3.6%; EfficientNet (2019) improved accuracy-per-FLOP dramatically; Vision Transformers (2020) became the default for large-scale image work. AlexNet’s role in 2026 is pedagogical — a historical baseline you rebuild once to understand, then set down.
So having built it, what do you do with a real problem? You don’t train this from scratch — you load a pretrained modern backbone and adapt it, which needs a fraction of the data and an afternoon instead of a week. That’s the whole point of pairing this article with its sibling: build AlexNet once here to see the machinery, then reach for transfer learning every time after. The forty lines above are the map; transfer learning is the territory.
❓ Frequently Asked Questions
Do I need a GPU to build a CNN image classifier like this? For the architecture and a small subset run, no — it will train on a CPU, just slowly. The moment you upsample CIFAR-10 to 224×224 and use the full training set, a GPU (or a smaller input size) becomes worth it. Start with a Subset of a few thousand images and two or three epochs to confirm the network learns before committing more compute.
Why upsample 32×32 CIFAR-10 images to 224×224 instead of shrinking AlexNet? To keep the network literally the paper’s architecture rather than a scaled-down lookalike. Upsampling is wasteful but faithful, which is the right trade-off for a teaching build. Building an adapted, smaller “AlexNet-style” net for 32×32 inputs is a fine exercise too — just be clear it’s no longer the 2012 model.
Should I include Local Response Normalization? Only for historical accuracy. LRN was in the 2012 paper (worth a ~1–2% error reduction there), but it’s been abandoned in favor of batch normalization since 2015. Leaving it in teaches you what the original did; you’d never add it to a new network today.
Will this reproduce AlexNet’s famous accuracy? No, and it isn’t meant to. The paper’s results came from 1.2M ImageNet images over ~90 epochs on two GPUs, and the headline 15.3% top-5 number was a seven-model ensemble. On a CIFAR-10 subset in a few epochs you’ll see clear learning well above chance — enough to prove the build works.
When would I ever pick AlexNet for real? You wouldn’t. Use it to learn the conv-pool-FC mechanics, then deploy a pretrained ResNet, EfficientNet, or Vision Transformer via transfer learning. AlexNet is a superb thing to understand once and a poor thing to ship.
📚 Further Reading
- Transfer Learning With a Pretrained CNN — the sibling to this build, and the workflow you’ll actually use: keep a pretrained backbone, swap the head, match the preprocessing.
- How AI Learned to See — the story of the 2012 AlexNet moment and what “a computer learning to see” really means.
- What Is an Embedding? — why a CNN’s penultimate layer is an embedding, and what that unlocks for search and similarity.
- The NIPS 2012 paper (ImageNet Classification with Deep Convolutional Neural Networks) and the
torchvision.models.alexnetdocumentation — the primary sources for the architecture, parameter counts, and the single-stream reimplementation.
📚 Recommended Resources
Disclosure: some links below are Amazon affiliate links. If you buy through them, I earn a small commission — at no extra cost to you — which helps keep this site running.
Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow
Aurélien Géron · Book · ~$65 on Amazon
The most practical shelf reference for building networks like this one — convolutional layers, training loops, and the pooling/dropout/normalization choices you just annotated. If you want one book that turns “I followed a tutorial” into “I can build it from a blank file,” this is it.
Deep Learning with Python
François Chollet · Book · ~$60 on Amazon
Written by the creator of Keras, with some of the clearest explanations anywhere of why a convolutional feature extractor works the way it does — what each layer learns, and why the deep stack sees edges then textures then whole objects. The best conceptual bridge from “run the code” to “understand the code.”
Deep Learning
Ian Goodfellow, Yoshua Bengio & Aaron Courville · Book · ~$80 on Amazon
The canonical graduate-level foundation — convolutional networks, regularization (dropout, weight decay), and optimization (SGD, momentum) — for readers who want the full mathematical grounding under this build. (The full text is also free to read in-browser at deeplearningbook.org; buy the print edition if you’d rather keep it on the desk.)