{"course":{"slug":"python-for-ai","title":"Python, from Zero, for AI","url":"https://addaly.com/learn/python-for-ai","level":"beginner","track":"build"},"lesson":{"slug":"notebooks-without-hidden-state","title":"Notebooks without hidden state: execution order, restart-and-run-all, and when to leave for a .py file","url":"https://addaly.com/learn/python-for-ai/notebooks-without-hidden-state","position":79,"of":89,"minutes":8,"answer":"A notebook's kernel remembers every cell ever run in whatever order you ran them, so a result can depend on a cell you deleted.","keyIdea":"A notebook's kernel remembers every cell ever run in whatever order you ran them, so a result can depend on a cell you deleted; Restart and Run All is the only test that the notebook says what it does, and code that has stopped changing belongs in a module the notebook imports."},"body":{"format":"markdown","text":"## What a notebook is underneath\n\nJupyter, Colab, Kaggle and VS Code's notebook view all work the same way: a **kernel** (one Python process) sits behind the page, and each cell you run is sent to it. The kernel keeps every variable, import and function from every cell you have ever executed, in the order you executed them, regardless of where the cells sit on the page.\n\nThat is the whole power of a notebook (you load a large file once and explore it for an hour without reloading) and the whole danger. The page shows cells in one order. The kernel remembers a different one. The two drift apart the moment you edit a cell and re-run only some of them, and the number in the square brackets beside each cell (`[7]`, `[3]`, `[12]`) is the only visible record of the real order.\n\n## The bug it produces\n\nYou define `clean_df` in cell 4, use it in cells 5 to 12, then decide cell 4 was wrong and rewrite it, or delete it and do the cleaning in cell 2 under a different name. The kernel still has the old `clean_df`. Cells 5 to 12 keep working. You reach a result, save the notebook, and send it. The recipient runs it from the top and cell 5 fails with `NameError`, or, worse, cell 2's version has a different name so nothing fails and the results silently differ from yours.\n\nEvery experienced notebook user has shipped this bug. The defence is not care; it is a ritual.\n\n## Restart and Run All\n\nBefore you trust a result, before you commit, before you send: **Kernel → Restart & Run All**. It kills the process, throws away every variable, and runs the cells top to bottom. If the notebook still produces the number, the notebook says what it does. If it fails, you have found the hidden dependency while it is cheap.\n\nDo it more often than feels necessary. A notebook that takes ten minutes to run from the top is telling you that the slow part (the data load, the embedding call) should be cached to disk with `np.save` or Parquet so that a restart costs seconds. That is a good pressure; give in to it.\n\n## Order of cells, and what goes where\n\nImports at the top, in one cell. Configuration (paths, model names, the seed) in the second. Then loading, cleaning, analysis, in the order a reader would want to follow. A notebook is read far more often than it is written, and a cell that uses a variable defined three cells below it is unreadable even when it runs.\n\nNever rely on a cell having been run twice. `df = df[df.price > 0]` run twice is harmless; `df[\"price\"] = df[\"price\"] * 100` run twice is a bug with no error message. Prefer transformations that are safe to repeat, or that build a new name from an old one so that re-running is a no-op.\n\n## The magics worth knowing\n\n```\n%timeit expr          # time a one-line expression, many runs\n%%time                # time a whole cell, once\n%load_ext autoreload  # then %autoreload 2: re-import edited .py files automatically\n%debug                # after an exception: open the debugger at the failing frame\n!pip install x        # a shell command; ! prefix\n```\n\n`%autoreload 2` is the one that changes how you work: functions you edit in a `.py` file update in the running kernel without restarting. `%debug` is module 4's debugger, one word away, with the failed cell's variables in scope.\n\n## Leaving for a file\n\nA notebook is for exploring: trying three approaches, looking at intermediate results, making plots. Once a piece of code has stopped changing, it belongs in a `.py` file that the notebook imports:\n\n```python\nfrom myproject.clean import clean_orders\ndf = clean_orders(raw)\n```\n\nThree reasons. A function in a module can be tested with pytest, and a cell cannot. A module can be imported by a script or a server, and a notebook cannot without ceremony. And a module has no hidden state; what it does is what it says.\n\n`jupyter nbconvert --to script analysis.ipynb` dumps a notebook to a `.py` file as a starting point; the real work is deciding which cells are functions and which were exploration to delete.\n\n## Notebooks in git\n\nA `.ipynb` file is JSON that stores the code, the outputs, including images as base64, and execution counts. Committing it after every run produces diffs that are thousands of lines of changed pixel data and a repository that grows by megabytes a day. `nbstripout` (free, `pip install nbstripout && nbstripout --install`) clears outputs on commit automatically, so the diff is the code. `jupytext` goes further and pairs each notebook with a plain `.py` twin that is what you actually version.\n\nOutputs also leak. A cell that printed `os.environ[\"OPENAI_API_KEY\"]` to check it was set has just written the key into the notebook file. Module 5 said never type a key into a cell; this is why, and `nbstripout` is the safety net.\n\n## Colab and Kaggle specifics\n\nBoth give you a free machine, often with a GPU, and both take it away: a Colab session ends after idleness or a fixed limit, and every variable and every file not saved to Drive is gone. Save intermediate results to disk early and often, mount Drive at the top, and design the notebook so that restarting from a saved checkpoint is one cell. The free tier is real and it is the best free computer most learners have; the discipline it enforces is the same one that makes a notebook trustworthy anywhere.\n\n## Try this now\n\nOpen any notebook you have, and Restart & Run All. If it fails, fix the order until it passes. Then install `nbstripout`, commit the notebook, and look at the diff. Finally move one function out into a `.py` file, turn on `%autoreload 2`, edit the function, and call it again without restarting."},"check":{"question":"A notebook runs top to bottom without error and shows an accuracy of 0.91. A colleague opens it, chooses Restart Kernel and Run All, and cell 12 fails with `NameError: name 'clean_df' is not defined`. What does this reveal?","options":[{"text":"The colleague's machine is missing a package, and the import failure surfaced as a NameError further down.","correct":false,"tempting":"A missing package raises `ModuleNotFoundError` at the import cell, not a `NameError` on a variable twelve cells later."},{"text":"Jupyter runs cells in parallel on Run All to save time, and cell 12 happened to execute before the cell that defined `clean_df` had finished on the colleague's slower machine.","correct":false,"tempting":"Run All executes cells strictly in order, top to bottom; there is no parallelism."},{"text":"Variable names beginning with `clean_` are reserved by pandas and are cleared when the kernel restarts.","correct":false,"tempting":"There are no reserved variable-name prefixes; a restart clears every variable equally."},{"text":"`clean_df` came from a cell later edited or deleted; the author's kernel still held it, so the notebook seemed to work while depending on state no longer in the file.","correct":true}]},"prev":{"slug":"plotting-with-matplotlib","title":"matplotlib: a plot you can read, saved to a file, from a script or a server","url":"https://addaly.com/learn/python-for-ai/plotting-with-matplotlib"},"next":{"slug":"a-chat-loop-with-memory","title":"A chat loop with memory: keeping history, trimming it, and stopping cleanly","url":"https://addaly.com/learn/python-for-ai/a-chat-loop-with-memory"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Source and assessment review continues. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Notebooks without hidden state: execution order, restart-and-run-all, and when to leave for a .py file', in Python, from Zero, for AI. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/python-for-ai/notebooks-without-hidden-state (Accessed: 21 November 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Notebooks without hidden state: execution order, restart-and-run-all, and when to leave for a .py file","courseTitle":"Python, from Zero, for AI","module":{"index":8,"slug":"numbers-in-bulk","title":"Numbers in bulk: NumPy, pandas and the vector under every model"},"answer":null,"keyIdea":"A notebook’s kernel remembers every cell ever run in whatever order you ran them, so a result can depend on a cell you deleted; Restart and Run All is the only test that the notebook says what it does, and code that has stopped changing belongs in a module the notebook imports.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"what-a-notebook-is-underneath","runs":[{"text":"What a notebook is underneath"}]},{"t":"p","runs":[{"text":"Jupyter, Colab, Kaggle and VS Code’s notebook view all work the same way: a "},{"text":"kernel","b":true},{"text":" (one Python process) sits behind the page, and each cell you run is sent to it. The kernel keeps every variable, import and function from every cell you have ever executed, in the order you executed them, regardless of where the cells sit on the page."}]},{"t":"p","runs":[{"text":"That is the whole power of a notebook (you load a large file once and explore it for an hour without reloading) and the whole danger. The page shows cells in one order. The kernel remembers a different one. The two drift apart the moment you edit a cell and re-run only some of them, and the number in the square brackets beside each cell ("},{"text":"[7]","code":true},{"text":", "},{"text":"[3]","code":true},{"text":", "},{"text":"[12]","code":true},{"text":") is the only visible record of the real order."}]},{"t":"h2","id":"the-bug-it-produces","runs":[{"text":"The bug it produces"}]},{"t":"p","runs":[{"text":"You define "},{"text":"clean_df","code":true},{"text":" in cell 4, use it in cells 5 to 12, then decide cell 4 was wrong and rewrite it, or delete it and do the cleaning in cell 2 under a different name. The kernel still has the old "},{"text":"clean_df","code":true},{"text":". Cells 5 to 12 keep working. You reach a result, save the notebook, and send it. The recipient runs it from the top and cell 5 fails with "},{"text":"NameError","code":true},{"text":", or, worse, cell 2’s version has a different name so nothing fails and the results silently differ from yours."}]},{"t":"p","runs":[{"text":"Every experienced notebook user has shipped this bug. The defence is not care; it is a ritual."}]},{"t":"h2","id":"restart-and-run-all","runs":[{"text":"Restart and Run All"}]},{"t":"p","runs":[{"text":"Before you trust a result, before you commit, before you send: "},{"text":"Kernel → Restart & Run All","b":true},{"text":". It kills the process, throws away every variable, and runs the cells top to bottom. If the notebook still produces the number, the notebook says what it does. If it fails, you have found the hidden dependency while it is cheap."}]},{"t":"p","runs":[{"text":"Do it more often than feels necessary. A notebook that takes ten minutes to run from the top is telling you that the slow part (the data load, the embedding call) should be cached to disk with "},{"text":"np.save","code":true},{"text":" or Parquet so that a restart costs seconds. That is a good pressure; give in to it."}]},{"t":"h2","id":"order-of-cells-and-what-goes-where","runs":[{"text":"Order of cells, and what goes where"}]},{"t":"p","runs":[{"text":"Imports at the top, in one cell. Configuration (paths, model names, the seed) in the second. Then loading, cleaning, analysis, in the order a reader would want to follow. A notebook is read far more often than it is written, and a cell that uses a variable defined three cells below it is unreadable even when it runs."}]},{"t":"p","runs":[{"text":"Never rely on a cell having been run twice. "},{"text":"df = df[df.price > 0]","code":true},{"text":" run twice is harmless; "},{"text":"df[\"price\"] = df[\"price\"] * 100","code":true},{"text":" run twice is a bug with no error message. Prefer transformations that are safe to repeat, or that build a new name from an old one so that re-running is a no-op."}]},{"t":"h2","id":"the-magics-worth-knowing","runs":[{"text":"The magics worth knowing"}]},{"t":"code","lang":"","label":"Text","text":"%timeit expr          # time a one-line expression, many runs\n%%time                # time a whole cell, once\n%load_ext autoreload  # then %autoreload 2: re-import edited .py files automatically\n%debug                # after an exception: open the debugger at the failing frame\n!pip install x        # a shell command; ! prefix"},{"t":"p","runs":[{"text":"%autoreload 2","code":true},{"text":" is the one that changes how you work: functions you edit in a "},{"text":".py","code":true},{"text":" file update in the running kernel without restarting. "},{"text":"%debug","code":true},{"text":" is module 4’s debugger, one word away, with the failed cell’s variables in scope."}]},{"t":"h2","id":"leaving-for-a-file","runs":[{"text":"Leaving for a file"}]},{"t":"p","runs":[{"text":"A notebook is for exploring: trying three approaches, looking at intermediate results, making plots. Once a piece of code has stopped changing, it belongs in a "},{"text":".py","code":true},{"text":" file that the notebook imports:"}]},{"t":"code","lang":"python","label":"Python","text":"from myproject.clean import clean_orders\ndf = clean_orders(raw)"},{"t":"p","runs":[{"text":"Three reasons. A function in a module can be tested with pytest, and a cell cannot. A module can be imported by a script or a server, and a notebook cannot without ceremony. And a module has no hidden state; what it does is what it says."}]},{"t":"p","runs":[{"text":"jupyter nbconvert --to script analysis.ipynb","code":true},{"text":" dumps a notebook to a "},{"text":".py","code":true},{"text":" file as a starting point; the real work is deciding which cells are functions and which were exploration to delete."}]},{"t":"h2","id":"notebooks-in-git","runs":[{"text":"Notebooks in git"}]},{"t":"p","runs":[{"text":"A "},{"text":".ipynb","code":true},{"text":" file is JSON that stores the code, the outputs, including images as base64, and execution counts. Committing it after every run produces diffs that are thousands of lines of changed pixel data and a repository that grows by megabytes a day. "},{"text":"nbstripout","code":true},{"text":" (free, "},{"text":"pip install nbstripout && nbstripout --install","code":true},{"text":") clears outputs on commit automatically, so the diff is the code. "},{"text":"jupytext","code":true},{"text":" goes further and pairs each notebook with a plain "},{"text":".py","code":true},{"text":" twin that is what you actually version."}]},{"t":"p","runs":[{"text":"Outputs also leak. A cell that printed "},{"text":"os.environ[\"OPENAI_API_KEY\"]","code":true},{"text":" to check it was set has just written the key into the notebook file. Module 5 said never type a key into a cell; this is why, and "},{"text":"nbstripout","code":true},{"text":" is the safety net."}]},{"t":"h2","id":"colab-and-kaggle-specifics","runs":[{"text":"Colab and Kaggle specifics"}]},{"t":"p","runs":[{"text":"Both give you a free machine, often with a GPU, and both take it away: a Colab session ends after idleness or a fixed limit, and every variable and every file not saved to Drive is gone. Save intermediate results to disk early and often, mount Drive at the top, and design the notebook so that restarting from a saved checkpoint is one cell. The free tier is real and it is the best free computer most learners have; the discipline it enforces is the same one that makes a notebook trustworthy anywhere."}]},{"t":"h2","id":"try-this-now","runs":[{"text":"Try this now"}],"role":"try"},{"t":"p","runs":[{"text":"Open any notebook you have, and Restart & Run All. If it fails, fix the order until it passes. Then install "},{"text":"nbstripout","code":true},{"text":", commit the notebook, and look at the diff. Finally move one function out into a "},{"text":".py","code":true},{"text":" file, turn on "},{"text":"%autoreload 2","code":true},{"text":", edit the function, and call it again without restarting."}]}],"check":{"question":[{"text":"A notebook runs top to bottom without error and shows an accuracy of 0.91. A colleague opens it, chooses Restart Kernel and Run All, and cell 12 fails with "},{"text":"NameError: name 'clean_df' is not defined","code":true},{"text":". What does this reveal?"}],"options":[{"text":[{"text":"The colleague’s machine is missing a package, and the import failure surfaced as a NameError further down."}],"correct":false,"tempting":[{"text":"A missing package raises "},{"text":"ModuleNotFoundError","code":true},{"text":" at the import cell, not a "},{"text":"NameError","code":true},{"text":" on a variable twelve cells later."}]},{"text":[{"text":"Jupyter runs cells in parallel on Run All to save time, and cell 12 happened to execute before the cell that defined "},{"text":"clean_df","code":true},{"text":" had finished on the colleague’s slower machine."}],"correct":false,"tempting":[{"text":"Run All executes cells strictly in order, top to bottom; there is no parallelism."}]},{"text":[{"text":"Variable names beginning with "},{"text":"clean_","code":true},{"text":" are reserved by pandas and are cleared when the kernel restarts."}],"correct":false,"tempting":[{"text":"There are no reserved variable-name prefixes; a restart clears every variable equally."}]},{"text":[{"text":"clean_df","code":true},{"text":" came from a cell later edited or deleted; the author’s kernel still held it, so the notebook seemed to work while depending on state no longer in the file."}],"correct":true,"tempting":null}],"order":[2,1,0,3],"why":null},"prev":{"slug":"plotting-with-matplotlib","title":"matplotlib: a plot you can read, saved to a file, from a script or a server","minutes":8},"next":{"slug":"a-chat-loop-with-memory","title":"A chat loop with memory: keeping history, trimming it, and stopping cleanly","minutes":9}}}