{"course":{"slug":"fine-tuning","title":"Fine-Tuning, and When Not To","url":"https://addaly.com/learn/fine-tuning","level":"advanced","track":"deep"},"lesson":{"slug":"checkpointing-and-resuming","title":"Checkpointing and resuming","url":"https://addaly.com/learn/fine-tuning/checkpointing-and-resuming","position":55,"of":84,"minutes":8,"answer":"A resumable checkpoint holds optimiser state, scheduler position and RNG state as well as weights, so resuming is only correct when the data order and the number of devices are unchanged.","keyIdea":"A resumable checkpoint holds optimiser state, scheduler position and RNG state as well as weights, so resuming is only correct when the data order and the number of devices are unchanged; and on interruptible instances the checkpoint must live somewhere the instance does not."},"body":{"format":"markdown","text":"## What a checkpoint contains\n\nA checkpoint is not only weights. A complete one holds five things:\n\n1. The trainable weights: the adapter, or the full model.\n2. The optimiser state: AdamW's two moment buffers per trainable parameter.\n3. The learning-rate scheduler's position.\n4. The random number generator states, so data shuffling resumes identically.\n5. The trainer state: global step, epoch, logged metrics.\n\nOnly the first is needed to *use* the model. All five are needed to *continue* the run.\n\nThe size follows: a 42M-parameter LoRA is about 160 MB in bf16, and its optimiser state is another 340 MB, so a resumable checkpoint is roughly half a gigabyte. Keeping three is a gigabyte and a half, which is nothing. Keeping three full 8B checkpoints is 130 GB, which is not.\n\n## The settings\n\n```python\nSFTConfig(\n    save_steps=200,\n    save_total_limit=3,\n    save_safetensors=True,\n    load_best_model_at_end=False,   # see below\n    output_dir=\"out/triage-v3\",\n)\n```\n\nSet `save_steps` so that a crash costs you at most ten or fifteen minutes of work. At three seconds per step that is every 200 to 300 steps.\n\n`save_total_limit=3` keeps the disk bounded and is enough to step back from a checkpoint that turned out to be past the peak.\n\n`load_best_model_at_end=False` is a deliberate choice. \"Best\" is defined by validation loss, and (as the epochs lesson established) validation loss is not what you want to select on. Keep all three and choose on your task metric afterwards, with the canary set beside it.\n\n## Resuming correctly\n\n```python\ntrainer.train(resume_from_checkpoint=True)\n```\n\nThis restores all five components and fast-forwards the data loader to where it stopped. Two conditions make that correct, and both are easy to break:\n\n**The data must be in the same order.** Resuming replays the shuffle from the saved RNG state, so a dataset that has been re-shuffled with a different seed, or had rows added, produces a different sequence. The resumed run will not see the examples it was going to see, and may see some twice.\n\n**The world size must match.** Resuming a four-GPU run on two GPUs changes the effective batch size, which changes the number of optimiser steps per epoch, which puts the scheduler in the wrong place. It will run and it will not be the run you started.\n\n## Interruptible instances\n\nSpot and community instances are roughly half price and can be reclaimed with little warning. With checkpointing in place that is an inconvenience rather than a loss, provided one thing: **the checkpoint must not be on the instance's local disk.**\n\n```python\n# free Colab or Kaggle\noutput_dir = \"/content/drive/MyDrive/runs/triage-v3\"\n\n# or push each checkpoint to a private Hub repo\nSFTConfig(push_to_hub=True, hub_model_id=\"you/triage-v3\",\n          hub_private_repo=True, hub_strategy=\"checkpoint\")\n```\n\nPrivate repositories on the Hugging Face Hub are free, and `hub_strategy=\"checkpoint\"` pushes the resumable checkpoint rather than only the final weights. A reclaimed instance then costs you a restart, not a run.\n\n## Saving the adapter alone\n\nFor distribution you want the weights without the optimiser state:\n\n```python\ntrainer.model.save_pretrained(\"out/adapter\")    # ~160 MB\ntok.save_pretrained(\"out/adapter\")               # always alongside\n```\n\nSave the tokenizer with it, every time. An adapter without its tokenizer is a file that loads and produces subtly wrong results, because somebody six months later will pair it with a slightly different tokenizer revision and have no way to know.\n\n## Evaluating checkpoints without stopping the run\n\nCheckpoints accumulate while the job is still going, and evaluating them afterwards serialises two things that could overlap. Point a second process at the output directory and have it score each new checkpoint as it appears:\n\n```bash\nwhile true; do\n  for c in out/triage-v3/checkpoint-*; do\n    [ -f \"$c/.scored\" ] || { python evaluate.py \"$c\" >> results.jsonl; touch \"$c/.scored\"; }\n  done\n  sleep 120\ndone\n```\n\nOn a single-GPU machine this competes for the card, so it suits a rented second card or a CPU-servable small model. Where it works, the run finishes with its selection evidence already gathered rather than a queue of checkpoints to get through.\n\n## Two things to record inside the directory\n\nWrite a small `run.json` next to the checkpoint:\n\n```json\n{\"base_model\": \"meta-llama/Llama-3.1-8B-Instruct\",\n \"base_revision\": \"0e9e39f\",\n \"dataset_sha256\": \"9f2c1e…\",\n \"config_sha256\": \"71ba04…\",\n \"git_commit\": \"4f1a09\",\n \"seed\": 0,\n \"gpu\": \"RTX 4090\", \"transformers\": \"4.5x.y\", \"trl\": \"0.x.y\"}\n```\n\nThis is the only thing that makes a checkpoint identifiable later. Without it you have a directory of tensors and a memory. The `base_revision` field in particular matters: model repositories are updated in place, and \"the same base model\" six months later may not be the same bytes."},"check":{"question":"A four-GPU run is interrupted at step 900 and resumed on two GPUs with the same per-device batch size. Why is the resumed run not a continuation of the original?","options":[{"text":"The effective batch halves, so steps per epoch and the scheduler's position no longer match.","correct":true},{"text":"Checkpoints store per-device optimiser shards, so two GPUs cannot reconstruct state saved by four.","correct":false,"tempting":"True for some sharded strategies but not the general reason; the described failure occurs even with a fully gathered state dict."},{"text":"The random number generator state is per device, so two GPUs replay only half the shuffle and see half the dataset.","correct":false,"tempting":"Overstates the consequence; data sharding adjusts to world size and the run still sees the full dataset, just in a different order."},{"text":"Gradient accumulation is recomputed on resume, which resets the learning rate schedule to its warmup phase.","correct":false,"tempting":"Invents a reset behaviour; the scheduler restores its saved position, which is precisely why the mismatch matters."}]},"prev":{"slug":"logging-and-watching","title":"What to log, and how to read it","url":"https://addaly.com/learn/fine-tuning/logging-and-watching"},"next":{"slug":"reproducibility-and-seeds","title":"Reproducibility, and the noise floor","url":"https://addaly.com/learn/fine-tuning/reproducibility-and-seeds"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Checkpointing and resuming', in Fine-Tuning, and When Not To. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/fine-tuning/checkpointing-and-resuming (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Checkpointing and resuming","courseTitle":"Fine-Tuning, and When Not To","module":{"index":6,"slug":"running-the-job","title":"The mechanics of a run"},"answer":null,"keyIdea":"A resumable checkpoint holds optimiser state, scheduler position and RNG state as well as weights, so resuming is only correct when the data order and the number of devices are unchanged; and on interruptible instances the checkpoint must live somewhere the instance does not.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"what-a-checkpoint-contains","runs":[{"text":"What a checkpoint contains"}]},{"t":"p","runs":[{"text":"A checkpoint is not only weights. A complete one holds five things:"}]},{"t":"ol","start":1,"items":[[{"text":"The trainable weights: the adapter, or the full model."}],[{"text":"The optimiser state: AdamW’s two moment buffers per trainable parameter."}],[{"text":"The learning-rate scheduler’s position."}],[{"text":"The random number generator states, so data shuffling resumes identically."}],[{"text":"The trainer state: global step, epoch, logged metrics."}]]},{"t":"p","runs":[{"text":"Only the first is needed to "},{"text":"use","i":true},{"text":" the model. All five are needed to "},{"text":"continue","i":true},{"text":" the run."}]},{"t":"p","runs":[{"text":"The size follows: a 42M-parameter LoRA is about 160 MB in bf16, and its optimiser state is another 340 MB, so a resumable checkpoint is roughly half a gigabyte. Keeping three is a gigabyte and a half, which is nothing. Keeping three full 8B checkpoints is 130 GB, which is not."}]},{"t":"h2","id":"the-settings","runs":[{"text":"The settings"}]},{"t":"code","lang":"python","label":"Python","text":"SFTConfig(\n    save_steps=200,\n    save_total_limit=3,\n    save_safetensors=True,\n    load_best_model_at_end=False,   # see below\n    output_dir=\"out/triage-v3\",\n)"},{"t":"p","runs":[{"text":"Set "},{"text":"save_steps","code":true},{"text":" so that a crash costs you at most ten or fifteen minutes of work. At three seconds per step that is every 200 to 300 steps."}]},{"t":"p","runs":[{"text":"save_total_limit=3","code":true},{"text":" keeps the disk bounded and is enough to step back from a checkpoint that turned out to be past the peak."}]},{"t":"p","runs":[{"text":"load_best_model_at_end=False","code":true},{"text":" is a deliberate choice. “Best” is defined by validation loss, and (as the epochs lesson established) validation loss is not what you want to select on. Keep all three and choose on your task metric afterwards, with the canary set beside it."}]},{"t":"h2","id":"resuming-correctly","runs":[{"text":"Resuming correctly"}]},{"t":"code","lang":"python","label":"Python","text":"trainer.train(resume_from_checkpoint=True)"},{"t":"p","runs":[{"text":"This restores all five components and fast-forwards the data loader to where it stopped. Two conditions make that correct, and both are easy to break:"}]},{"t":"p","runs":[{"text":"The data must be in the same order.","b":true},{"text":" Resuming replays the shuffle from the saved RNG state, so a dataset that has been re-shuffled with a different seed, or had rows added, produces a different sequence. The resumed run will not see the examples it was going to see, and may see some twice."}]},{"t":"p","runs":[{"text":"The world size must match.","b":true},{"text":" Resuming a four-GPU run on two GPUs changes the effective batch size, which changes the number of optimiser steps per epoch, which puts the scheduler in the wrong place. It will run and it will not be the run you started."}]},{"t":"h2","id":"interruptible-instances","runs":[{"text":"Interruptible instances"}]},{"t":"p","runs":[{"text":"Spot and community instances are roughly half price and can be reclaimed with little warning. With checkpointing in place that is an inconvenience rather than a loss, provided one thing: "},{"text":"the checkpoint must not be on the instance’s local disk.","b":true}]},{"t":"code","lang":"python","label":"Python","text":"# free Colab or Kaggle\noutput_dir = \"/content/drive/MyDrive/runs/triage-v3\"\n\n# or push each checkpoint to a private Hub repo\nSFTConfig(push_to_hub=True, hub_model_id=\"you/triage-v3\",\n          hub_private_repo=True, hub_strategy=\"checkpoint\")"},{"t":"p","runs":[{"text":"Private repositories on the Hugging Face Hub are free, and "},{"text":"hub_strategy=\"checkpoint\"","code":true},{"text":" pushes the resumable checkpoint rather than only the final weights. A reclaimed instance then costs you a restart, not a run."}]},{"t":"h2","id":"saving-the-adapter-alone","runs":[{"text":"Saving the adapter alone"}]},{"t":"p","runs":[{"text":"For distribution you want the weights without the optimiser state:"}]},{"t":"code","lang":"python","label":"Python","text":"trainer.model.save_pretrained(\"out/adapter\")    # ~160 MB\ntok.save_pretrained(\"out/adapter\")               # always alongside"},{"t":"p","runs":[{"text":"Save the tokenizer with it, every time. An adapter without its tokenizer is a file that loads and produces subtly wrong results, because somebody six months later will pair it with a slightly different tokenizer revision and have no way to know."}]},{"t":"h2","id":"evaluating-checkpoints-without-stopping-the-run","runs":[{"text":"Evaluating checkpoints without stopping the run"}]},{"t":"p","runs":[{"text":"Checkpoints accumulate while the job is still going, and evaluating them afterwards serialises two things that could overlap. Point a second process at the output directory and have it score each new checkpoint as it appears:"}]},{"t":"code","lang":"bash","label":"Bash","text":"while true; do\n  for c in out/triage-v3/checkpoint-*; do\n    [ -f \"$c/.scored\" ] || { python evaluate.py \"$c\" >> results.jsonl; touch \"$c/.scored\"; }\n  done\n  sleep 120\ndone"},{"t":"p","runs":[{"text":"On a single-GPU machine this competes for the card, so it suits a rented second card or a CPU-servable small model. Where it works, the run finishes with its selection evidence already gathered rather than a queue of checkpoints to get through."}]},{"t":"h2","id":"two-things-to-record-inside-the-directory","runs":[{"text":"Two things to record inside the directory"}]},{"t":"p","runs":[{"text":"Write a small "},{"text":"run.json","code":true},{"text":" next to the checkpoint:"}]},{"t":"code","lang":"json","label":"JSON","text":"{\"base_model\": \"meta-llama/Llama-3.1-8B-Instruct\",\n \"base_revision\": \"0e9e39f\",\n \"dataset_sha256\": \"9f2c1e…\",\n \"config_sha256\": \"71ba04…\",\n \"git_commit\": \"4f1a09\",\n \"seed\": 0,\n \"gpu\": \"RTX 4090\", \"transformers\": \"4.5x.y\", \"trl\": \"0.x.y\"}"},{"t":"p","runs":[{"text":"This is the only thing that makes a checkpoint identifiable later. Without it you have a directory of tensors and a memory. The "},{"text":"base_revision","code":true},{"text":" field in particular matters: model repositories are updated in place, and “the same base model” six months later may not be the same bytes."}]}],"check":{"question":[{"text":"A four-GPU run is interrupted at step 900 and resumed on two GPUs with the same per-device batch size. Why is the resumed run not a continuation of the original?"}],"options":[{"text":[{"text":"The effective batch halves, so steps per epoch and the scheduler’s position no longer match."}],"correct":true,"tempting":null},{"text":[{"text":"Checkpoints store per-device optimiser shards, so two GPUs cannot reconstruct state saved by four."}],"correct":false,"tempting":[{"text":"True for some sharded strategies but not the general reason; the described failure occurs even with a fully gathered state dict."}]},{"text":[{"text":"The random number generator state is per device, so two GPUs replay only half the shuffle and see half the dataset."}],"correct":false,"tempting":[{"text":"Overstates the consequence; data sharding adjusts to world size and the run still sees the full dataset, just in a different order."}]},{"text":[{"text":"Gradient accumulation is recomputed on resume, which resets the learning rate schedule to its warmup phase."}],"correct":false,"tempting":[{"text":"Invents a reset behaviour; the scheduler restores its saved position, which is precisely why the mismatch matters."}]}],"order":[3,0,2,1],"why":null},"prev":{"slug":"logging-and-watching","title":"What to log, and how to read it","minutes":8},"next":{"slug":"reproducibility-and-seeds","title":"Reproducibility, and the noise floor","minutes":9}}}