Notebooks without hidden state: execution order, restart-and-run-all, and when to leave for a .py file

What a notebook is underneath

Jupyter, Colab, Kaggle and VS Code's notebook view all work the same way: a kernel — one Python process — sits behind the page, and each cell you run is sent to it. The kernel keeps every variable, import and function from every cell you have ever executed, in the order you executed them, regardless of where the cells sit on the page.

That is the whole power of a notebook — you load a large file once and explore it for an hour without reloading — and the whole danger. The page shows cells in one order. The kernel remembers a different one. The two drift apart the moment you edit a cell and re-run only some of them, and the number in the square brackets beside each cell — [7], [3], [12] — is the only visible record of the real order.

The bug it produces

You define clean_df in cell 4, use it in cells 5 to 12, then decide cell 4 was wrong and rewrite it — or delete it and do the cleaning in cell 2 under a different name. The kernel still has the old clean_df. Cells 5 to 12 keep working. You reach a result, save the notebook, and send it. The recipient runs it from the top and cell 5 fails with NameError, or — worse — cell 2's version has a different name so nothing fails and the results silently differ from yours.

Every experienced notebook user has shipped this bug. The defence is not care; it is a ritual.

Restart and Run All

Before you trust a result, before you commit, before you send: Kernel → Restart & Run All. It kills the process, throws away every variable, and runs the cells top to bottom. If the notebook still produces the number, the notebook says what it does. If it fails, you have found the hidden dependency while it is cheap.

Do it more often than feels necessary. A notebook that takes ten minutes to run from the top is telling you that the slow part — the data load, the embedding call — should be cached to disk with np.save or Parquet so that a restart costs seconds. That is a good pressure; give in to it.

Order of cells, and what goes where

Imports at the top, in one cell. Configuration — paths, model names, the seed — in the second. Then loading, cleaning, analysis, in the order a reader would want to follow. A notebook is read far more often than it is written, and a cell that uses a variable defined three cells below it is unreadable even when it runs.

Never rely on a cell having been run twice. df = df[df.price > 0] run twice is harmless; df["price"] = df["price"] * 100 run twice is a bug with no error message. Prefer transformations that are safe to repeat, or that build a new name from an old one so that re-running is a no-op.

The magics worth knowing

%timeit expr          # time a one-line expression, many runs
%%time                # time a whole cell, once
%load_ext autoreload  # then %autoreload 2: re-import edited .py files automatically
%debug                # after an exception: open the debugger at the failing frame
!pip install x        # a shell command; ! prefix

%autoreload 2 is the one that changes how you work: functions you edit in a .py file update in the running kernel without restarting. %debug is module 4's debugger, one word away, with the failed cell's variables in scope.

Leaving for a file

A notebook is for exploring — trying three approaches, looking at intermediate results, making plots. Once a piece of code has stopped changing, it belongs in a .py file that the notebook imports:

from myproject.clean import clean_orders
df = clean_orders(raw)

Three reasons. A function in a module can be tested with pytest, and a cell cannot. A module can be imported by a script or a server, and a notebook cannot without ceremony. And a module has no hidden state; what it does is what it says.

jupyter nbconvert --to script analysis.ipynb dumps a notebook to a .py file as a starting point; the real work is deciding which cells are functions and which were exploration to delete.

Notebooks in git

A .ipynb file is JSON that stores the code, the outputs — including images as base64 — and execution counts. Committing it after every run produces diffs that are thousands of lines of changed pixel data and a repository that grows by megabytes a day. nbstripout (free, pip install nbstripout && nbstripout --install) clears outputs on commit automatically, so the diff is the code. jupytext goes further and pairs each notebook with a plain .py twin that is what you actually version.

Outputs also leak. A cell that printed os.environ["OPENAI_API_KEY"] to check it was set has just written the key into the notebook file. Module 5 said never type a key into a cell; this is why, and nbstripout is the safety net.

Colab and Kaggle specifics

Both give you a free machine, often with a GPU, and both take it away: a Colab session ends after idleness or a fixed limit, and every variable and every file not saved to Drive is gone. Save intermediate results to disk early and often, mount Drive at the top, and design the notebook so that restarting from a saved checkpoint is one cell. The free tier is real and it is the best free computer most learners have; the discipline it enforces is the same one that makes a notebook trustworthy anywhere.

Try this now

Open any notebook you have, and Restart & Run All. If it fails, fix the order until it passes. Then install nbstripout, commit the notebook, and look at the diff. Finally move one function out into a .py file, turn on %autoreload 2, edit the function, and call it again without restarting.