You opened a shared notebook, pressed Run, watched the cells light up one by one, and got a chart. It feels like proof. It is not, and the largest study of the question explains why. Researchers tried to execute hundreds of thousands of published notebooks, and Jupyter notebook reproducibility turned out to be far lower than “it ran on my screen” suggests.
This is a companion to our guide to what Google Colab actually is. That guide makes the case that Colab removed the hardest step in learning to program with data, which is setting up the machine, and that the cost did not disappear. It moved. This piece is about where it moved to: the gap between “this notebook ran” and “this notebook gives anyone the same answer.”
Ran, and reproduced, are different claims
Two words get blurred together, so pull them apart first.
- It ran means the code finished without throwing an error, at least once, for at least one person.
- It reproduced means that when somebody else (or you, next month) runs it, the numbers and charts come out the same as the ones saved in the file.
A notebook can run and still not reproduce. Running proves the code is alive. It says nothing about whether the answer on the page is the answer the code produces today.
A shared notebook is a recipe, not a meal. The saved charts are a photograph of one meal someone cooked once, in their own kitchen, with whatever was in their pantry that day.
The study that actually counted
In 2019, Pimentel, Murta, Braganholo and Freire published “A Large-scale Study about Quality and Reproducibility of Jupyter Notebooks” at the Mining Software Repositories conference (MSR 2019). They did not ask people how they felt about notebooks. They collected notebooks from GitHub and ran them.
According to the paper (loaded 6 November 2026), the corpus was 1,159,166 unique notebooks from 264,023 GitHub repositories. Their headline, in the authors’ own words: “out of 863,878 attempted executions of valid notebooks (i.e., notebooks with defined Python version and execution order), only 24.11% executed without errors and only 4.03% produced the same results.”
The two numbers answer different questions:
| Stage | What it asks | Result in the study |
|---|---|---|
| Attempted | Notebooks with a defined Python version and execution order that the authors tried to run | 863,878 executions |
| Ran | Finished without an error | 24.11% (208,323 notebooks) |
| Reproduced | Finished and produced the same results as the outputs stored in the file | 4.03% |
| Ran but differed | Finished, yet produced different results from the stored ones | 173,487 notebooks |
The 4.03% is “produced the same results,” not “worked.” Do not merge the two figures into one scary statistic. The authors also note that their 24.11% run rate sits close to the 24.9% that Collberg and colleagues found for reproducibility in computer-systems research generally. Notebooks are not uniquely bad at running. On this evidence they are especially bad at agreeing with themselves.
One more caution. This is a study of notebooks published on GitHub, not of every notebook in the world. Private classroom notebooks, quick experiments and notebooks shared as Colab links are not in that sample, and nobody has measured them the same way.
Why they failed: three missing things
The failures sort into a pattern. Figures below are from the paper.
| Cause of failure | Share of executions | What it usually means |
|---|---|---|
| ImportError / ModuleNotFoundError | 29.23% | A library the notebook needs is not installed here |
| NameError | 14.53% | A variable is used that was never defined, the signature of cells run out of order |
| FileNotFoundError / IOError | 12.59% | The notebook points at a data file that was never shared, or at one machine’s folder path |
Notice what these have in common. None says the analysis was wrong. They say the recipe lists ingredients nobody supplied.
The ingredients list is missing
The authors write that “the notebook format does not encode its library dependencies with associated versions, which can make it hard (or even impossible) to reproduce the notebook.” They also measured how often people compensated. Of the 863,878 notebooks they executed, only 13.72% (118,483) came from repositories that declared their dependencies in a standard file.
Declaring them was not enough. Among notebooks from repositories with declared dependencies, 45.18% still failed with an import error, against 31.24% for repositories without any. The authors read that as a sign that many dependency files do not declare everything the notebook needs. A shopping list is only useful if it is complete.
The order of the steps is missing
The authors list “hidden states” and “unexpected execution order with fragmented code” among the format’s main criticisms, and they counted it too. Among notebooks with an unambiguous execution order, 36.36% had cells run out of order.
The data is missing
The paper says file errors happen “when users use absolute paths to access data files or do not include the data in the repositories.” A path that works on your laptop leads nowhere on mine.
Why hidden state is built in, not a flaw in the author
It is tempting to blame careless people. Jupyter’s own architecture documentation points at the design instead. It says the Jupyter server, not the kernel, is responsible for saving and loading notebooks, and adds: “The kernel doesn’t know anything about the notebook document: it just gets sent cells of code to execute when the user runs them.” (Loaded 6 November 2026.)
Think about what that means. The .ipynb file on disk is a record of the page. The variables you actually have are living in a separate running process that has never read that page. You can run cells in any order, run one twice, edit a cell after running it, or delete a cell after running it. The variable it created is still sitting in memory, and the saved file looks perfectly tidy either way.
That is why the same notebook can produce a chart for its author and an error for everyone else. The author’s session contains a hidden ingredient that the document never mentions.
Joel Grus gave a well-known talk on this theme, “I Don’t Like Notebooks,” at JupyterCon 2018. We could not load its slides or recording while writing this, so we point to its existence and quote nothing from it. The peer-reviewed measurements above carry the substance of the argument instead.
What Colab changes, and what it does not
Colab is built on the notebook, so it inherits every problem above. It also adds one of its own.
| Property | Notebook on your own machine | Notebook shared as a Colab link |
|---|---|---|
| Hidden state and run order | Yes, same design | Yes, same design |
| Dependencies recorded in the file | No | No |
| Your files after you close it | Stay on your disk | Live on a temporary machine that is cleared when it is recycled |
| Who controls the machine | You | Google, with allocation that varies |
| What a recipient receives | Whatever you send them | The document, not your machine |
On that last row, Google’s Colab FAQ is blunt (loaded 6 November 2026): “The virtual machine you’re using, including any custom files and libraries that you’ve setup, will not be shared.” Your custom files and libraries stay behind. The FAQ’s own advice is to include cells that install and load whatever the notebook needs.
The machine itself also moves under you. The same FAQ says that usage limits, idle timeouts, maximum VM lifetime and GPU types available “vary over time,” and that “Colab does not publish these limits, in part because they can vary over time.” So even a notebook that lists every library can meet a different machine on a different day.
A related drift lives in the preinstalled software. Google publishes snapshots of the Colab runtime’s package list in its googlecolab/backend-info repository. The file’s own header carries a caution: “Be aware that this list does not necessarily reflect the current state of the staging or production container.” It is still a useful reminder that the stack under “import pandas” is a moving target you did not choose.
Ephemerality is also not a free-tier quirk. Google’s documentation for Colab Enterprise, its paid and governed offering, says the same thing in its own terms: runtime file changes are not preserved. You cannot buy your way out of this trade. That is the architecture.
Three habits that cost almost nothing
None of these makes a notebook a meal. They make the recipe honest.
- Restart and run everything from the top before you share. If it fails after a fresh start, you have found the hidden state yourself, in thirty seconds, instead of leaving it for your reader. It is the cheapest reproducibility test there is.
- Write down the ingredients. Print the library versions your notebook actually used and paste them into the notebook, or save them as a requirements file next to it. A list of names without versions is the same gap Pimentel measured.
- Ship the data, or say exactly where it comes from. A download cell with a stable address beats a path on your own computer.
Each habit answers one of the three failures in the table: missing libraries, missing order, missing data.
Tools that help, if people adopt them
Real tools exist for the harder cases. nbdime diffs and merges notebooks, which are awkward to compare as plain files. repo2docker, from the Jupyter community, builds a runnable environment from a repository. They are good. But the study’s 13.72% is the warning: a tool nobody uses is an option, not a rebuttal.
Fair to the notebook
The critique is not a verdict that notebooks are bad. A notebook is often the right place to explore, and exploring is a legitimate mode of work. The honest reading is narrower: notebooks make it easy to get an answer once and give no guarantee that anyone, including you, can get it twice. For teaching, sketching and first looks that is fine. For a result that someone will rely on, it is not enough on its own.
Chattopadhyay and colleagues interviewed 20 data scientists and surveyed 156 (CHI 2020), and reproducing and reusing results was among the pain points practitioners described. That matches what the execution study counted from the outside.
❓ Frequently Asked Questions
What percentage of Jupyter notebooks are reproducible?
In the MSR 2019 study of 863,878 attempted executions of published GitHub notebooks, 24.11% ran without errors and 4.03% produced the same results as the stored outputs. It is a measurement of GitHub notebooks from that study, not a law.
Is Colab less reproducible than Jupyter?
Not in design terms: both inherit the notebook’s hidden state. Colab adds a temporary machine that Google controls, and a shared link carries the document without the environment.
Why does my notebook work for me and fail for a friend?
Usually one of three things: a library they lack, cells run in a different order, or a data file that exists only on your side. Those were the three biggest failure groups in the study.
What does “restart and run all” prove?
That the document alone can produce its results from a clean start. It does not prove the result is correct, only that it does not secretly depend on leftover state.
Does setting a random seed fix reproducibility?
It addresses one source of difference. It does nothing for missing libraries, missing data or moved software, and what a seed guarantees varies by library, so check your own library’s documentation before relying on one.
Should I stop using notebooks?
No. Use them to explore, then move what matters into something that records its own environment. Running is the start of a result, not the end.
📦 Recommended Resources
Some links below are Amazon affiliate links. If you buy through them this site earns a small commission at no extra cost to you.
- Software Engineering for Data Scientists: From Notebooks to Scalable Systems — Catherine Nelson · Book. The natural next step once a notebook needs to be something other people can run.
- Python Testing with pytest — Brian Okken · Book. Tests are how you find out that code still gives the same answer next month.
- Python for Data Analysis — Wes McKinney · Book. A grounded tour of the everyday data tools that most shared notebooks lean on.