{"course":{"slug":"fine-tuning","title":"Fine-Tuning, and When Not To","url":"https://addaly.com/learn/fine-tuning","level":"advanced","track":"deep"},"lesson":{"slug":"reproducibility-and-seeds","title":"Reproducibility, and the noise floor","url":"https://addaly.com/learn/fine-tuning/reproducibility-and-seeds","position":56,"of":84,"minutes":9,"answer":"Set a seed and record the git commit, config hash, dataset hash, base revision and library versions; then run three seeds to find your noise floor.","keyIdea":"Set a seed and record the git commit, config hash, dataset hash, base revision and library versions; then run three seeds to find your noise floor, because an improvement smaller than twice that spread is a coin flip you have named a decision."},"body":{"format":"markdown","text":"## Bit-identical is harder than it looks\n\n```python\nfrom transformers import set_seed\nset_seed(0)     # python, numpy, torch, cuda\n```\n\nThat covers weight initialisation, dropout, data shuffling and sampling. It does not give you a bit-identical run, because GPU reductions are not deterministic by default: floating-point addition is not associative, and the order in which thousands of parallel threads accumulate a sum varies between executions.\n\nFull determinism is available and it is expensive:\n\n```python\nimport torch, os\nos.environ[\"CUBLAS_WORKSPACE_CONFIG\"] = \":4096:8\"\ntorch.use_deterministic_algorithms(True)\ntorch.backends.cudnn.benchmark = False\n```\n\nThis costs throughput, and some operations have no deterministic implementation at all: the call will raise rather than silently be non-deterministic, which is the right behaviour and is also how you discover that your model uses one of them.\n\nAnd even with all of it, results differ across GPU models, driver versions and library versions. Reproducibility across machines is not achievable by flags.\n\n## Reproducible enough\n\nThe practical goal is not bit-identity. It is: *somebody with this repository can produce a model that performs the same within noise.* That needs a record, not a flag.\n\n```json\n{\"git_commit\": \"4f1a09\",\n \"config_sha256\": \"71ba04…\",\n \"dataset_sha256\": \"9f2c1e…\",\n \"base_model\": \"meta-llama/Llama-3.1-8B-Instruct\",\n \"base_revision\": \"0e9e39f\",\n \"seed\": 0,\n \"gpu\": \"1×RTX 4090\",\n \"versions\": {\"torch\": \"…\", \"transformers\": \"…\", \"trl\": \"…\", \"peft\": \"…\"}}\n```\n\nThe two hashes are the ones people omit and the two that matter most. \"We trained on the customer dataset\" is not a reproducible statement; a sha256 of the exact JSONL is.\n\nPin the base model revision too. Model repositories are edited (tokenizer configs get updated, chat templates get fixed) and `main` six months from now is a different artefact.\n\n## The measurement everybody skips\n\nHere is the part that changes how you read every result in this course.\n\nRun the same configuration three times with seeds 0, 1 and 2. Evaluate all three on your held-out set. The spread between them is your **noise floor**.\n\n```\nseed 0: 0.871\nseed 1: 0.858\nseed 2: 0.884\n→ range 2.6 points, sd ≈ 1.3\n```\n\nNow look at the last four \"improvements\" you recorded. If they were one or two points, you measured nothing. You measured seed variance and attributed it to a decision.\n\nThis is the single most common error in applied fine-tuning, and three extra runs (a few hours and a few pounds) is the whole cost of not making it. Until you know your noise floor, you cannot distinguish a real effect from a coin flip, and every ablation you run is uninterpretable.\n\nOnce you have it, the rule is simple: an improvement smaller than about twice your seed standard deviation is not an improvement. Either accept that, or run each configuration three times and compare the means, which is what the difference is worth measuring, if it is worth measuring.\n\n## Where the variance comes from\n\nWorth knowing, because it tells you how to reduce it:\n\n- **Data order** dominates on small datasets. With 800 examples, the sequence in which the model sees them changes the result materially.\n- **Adapter initialisation**: the random A matrix.\n- **Dropout masks.**\n- **Non-deterministic reductions**, the smallest contributor.\n\nMore data narrows the first and largest source, which is one more reason the dataset is the lever that matters.\n\n## Reporting a result with its noise\n\nOnce you have the floor, the honest form of a result is a mean and a range, not a single figure:\n\n```\nbaseline   0.871 ± 1.3  (3 seeds)\n+ replay   0.870 ± 1.1  (3 seeds)   task unchanged\n           canary 0.62 → 0.81       the actual finding\n```\n\nNote what that reporting does. It makes the task number a non-finding explicitly, and puts the canary movement (which is well outside the noise) where the reader's attention belongs. A single-figure table would have invited an argument about 0.871 versus 0.870.\n\n## The evaluation must be fixed\n\nSeed your evaluation too. If you generate at temperature 0.7 for evaluation, the same model scores differently on consecutive runs and you have added a second source of noise on top of the training one. Evaluate greedily, or with a fixed seed and enough samples to average, and record which you did.\n\nA result without a stated evaluation procedure is not a result. A result without a noise floor is not a comparison."},"check":{"question":"A team runs the same configuration with three seeds and gets 0.871, 0.858 and 0.884. They then report that switching LoRA rank from 16 to 32 improved their metric from 0.871 to 0.889. What is the problem?","options":[{"text":"Comparing against the seed-0 run alone is invalid; they should have used the highest of the three seeds as the baseline.","correct":false,"tempting":"Proposes picking the most favourable baseline, which biases the comparison in the opposite direction rather than fixing it."},{"text":"The 1.8-point gain is smaller than the 2.6-point spread they measured across seeds.","correct":true},{"text":"Rank 32 needs a different alpha, so the comparison is confounded by scaling rather than by noise.","correct":false,"tempting":"Names a real confound in rank comparisons, but the stated numbers are within the seed spread regardless of how alpha was set."},{"text":"Three seeds is too few to estimate variance, so the noise floor itself is unreliable and no comparison can be made.","correct":false,"tempting":"Demands more precision on the variance estimate while ignoring that even a rough spread already exceeds the claimed effect."}]},"prev":{"slug":"checkpointing-and-resuming","title":"Checkpointing and resuming","url":"https://addaly.com/learn/fine-tuning/checkpointing-and-resuming"},"next":{"slug":"multi-gpu-without-tears","title":"Multi-GPU without tears","url":"https://addaly.com/learn/fine-tuning/multi-gpu-without-tears"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Reproducibility, and the noise floor', in Fine-Tuning, and When Not To. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/fine-tuning/reproducibility-and-seeds (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Reproducibility, and the noise floor","courseTitle":"Fine-Tuning, and When Not To","module":{"index":6,"slug":"running-the-job","title":"The mechanics of a run"},"answer":null,"keyIdea":"Set a seed and record the git commit, config hash, dataset hash, base revision and library versions; then run three seeds to find your noise floor, because an improvement smaller than twice that spread is a coin flip you have named a decision.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"bit-identical-is-harder-than-it-looks","runs":[{"text":"Bit-identical is harder than it looks"}]},{"t":"code","lang":"python","label":"Python","text":"from transformers import set_seed\nset_seed(0)     # python, numpy, torch, cuda"},{"t":"p","runs":[{"text":"That covers weight initialisation, dropout, data shuffling and sampling. It does not give you a bit-identical run, because GPU reductions are not deterministic by default: floating-point addition is not associative, and the order in which thousands of parallel threads accumulate a sum varies between executions."}]},{"t":"p","runs":[{"text":"Full determinism is available and it is expensive:"}]},{"t":"code","lang":"python","label":"Python","text":"import torch, os\nos.environ[\"CUBLAS_WORKSPACE_CONFIG\"] = \":4096:8\"\ntorch.use_deterministic_algorithms(True)\ntorch.backends.cudnn.benchmark = False"},{"t":"p","runs":[{"text":"This costs throughput, and some operations have no deterministic implementation at all: the call will raise rather than silently be non-deterministic, which is the right behaviour and is also how you discover that your model uses one of them."}]},{"t":"p","runs":[{"text":"And even with all of it, results differ across GPU models, driver versions and library versions. Reproducibility across machines is not achievable by flags."}]},{"t":"h2","id":"reproducible-enough","runs":[{"text":"Reproducible enough"}]},{"t":"p","runs":[{"text":"The practical goal is not bit-identity. It is: "},{"text":"somebody with this repository can produce a model that performs the same within noise.","i":true},{"text":" That needs a record, not a flag."}]},{"t":"code","lang":"json","label":"JSON","text":"{\"git_commit\": \"4f1a09\",\n \"config_sha256\": \"71ba04…\",\n \"dataset_sha256\": \"9f2c1e…\",\n \"base_model\": \"meta-llama/Llama-3.1-8B-Instruct\",\n \"base_revision\": \"0e9e39f\",\n \"seed\": 0,\n \"gpu\": \"1×RTX 4090\",\n \"versions\": {\"torch\": \"…\", \"transformers\": \"…\", \"trl\": \"…\", \"peft\": \"…\"}}"},{"t":"p","runs":[{"text":"The two hashes are the ones people omit and the two that matter most. “We trained on the customer dataset” is not a reproducible statement; a sha256 of the exact JSONL is."}]},{"t":"p","runs":[{"text":"Pin the base model revision too. Model repositories are edited (tokenizer configs get updated, chat templates get fixed) and "},{"text":"main","code":true},{"text":" six months from now is a different artefact."}]},{"t":"h2","id":"the-measurement-everybody-skips","runs":[{"text":"The measurement everybody skips"}]},{"t":"p","runs":[{"text":"Here is the part that changes how you read every result in this course."}]},{"t":"p","runs":[{"text":"Run the same configuration three times with seeds 0, 1 and 2. Evaluate all three on your held-out set. The spread between them is your "},{"text":"noise floor","b":true},{"text":"."}]},{"t":"code","lang":"","label":"Text","text":"seed 0: 0.871\nseed 1: 0.858\nseed 2: 0.884\n→ range 2.6 points, sd ≈ 1.3"},{"t":"p","runs":[{"text":"Now look at the last four “improvements” you recorded. If they were one or two points, you measured nothing. You measured seed variance and attributed it to a decision."}]},{"t":"p","runs":[{"text":"This is the single most common error in applied fine-tuning, and three extra runs (a few hours and a few pounds) is the whole cost of not making it. Until you know your noise floor, you cannot distinguish a real effect from a coin flip, and every ablation you run is uninterpretable."}]},{"t":"p","runs":[{"text":"Once you have it, the rule is simple: an improvement smaller than about twice your seed standard deviation is not an improvement. Either accept that, or run each configuration three times and compare the means, which is what the difference is worth measuring, if it is worth measuring."}]},{"t":"h2","id":"where-the-variance-comes-from","runs":[{"text":"Where the variance comes from"}]},{"t":"p","runs":[{"text":"Worth knowing, because it tells you how to reduce it:"}]},{"t":"ul","items":[[{"text":"Data order","b":true},{"text":" dominates on small datasets. With 800 examples, the sequence in which the model sees them changes the result materially."}],[{"text":"Adapter initialisation","b":true},{"text":": the random A matrix."}],[{"text":"Dropout masks.","b":true}],[{"text":"Non-deterministic reductions","b":true},{"text":", the smallest contributor."}]]},{"t":"p","runs":[{"text":"More data narrows the first and largest source, which is one more reason the dataset is the lever that matters."}]},{"t":"h2","id":"reporting-a-result-with-its-noise","runs":[{"text":"Reporting a result with its noise"}]},{"t":"p","runs":[{"text":"Once you have the floor, the honest form of a result is a mean and a range, not a single figure:"}]},{"t":"code","lang":"","label":"Text","text":"baseline   0.871 ± 1.3  (3 seeds)\n+ replay   0.870 ± 1.1  (3 seeds)   task unchanged\n           canary 0.62 → 0.81       the actual finding"},{"t":"p","runs":[{"text":"Note what that reporting does. It makes the task number a non-finding explicitly, and puts the canary movement (which is well outside the noise) where the reader’s attention belongs. A single-figure table would have invited an argument about 0.871 versus 0.870."}]},{"t":"h2","id":"the-evaluation-must-be-fixed","runs":[{"text":"The evaluation must be fixed"}]},{"t":"p","runs":[{"text":"Seed your evaluation too. If you generate at temperature 0.7 for evaluation, the same model scores differently on consecutive runs and you have added a second source of noise on top of the training one. Evaluate greedily, or with a fixed seed and enough samples to average, and record which you did."}]},{"t":"p","runs":[{"text":"A result without a stated evaluation procedure is not a result. A result without a noise floor is not a comparison."}]}],"check":{"question":[{"text":"A team runs the same configuration with three seeds and gets 0.871, 0.858 and 0.884. They then report that switching LoRA rank from 16 to 32 improved their metric from 0.871 to 0.889. What is the problem?"}],"options":[{"text":[{"text":"Comparing against the seed-0 run alone is invalid; they should have used the highest of the three seeds as the baseline."}],"correct":false,"tempting":[{"text":"Proposes picking the most favourable baseline, which biases the comparison in the opposite direction rather than fixing it."}]},{"text":[{"text":"The 1.8-point gain is smaller than the 2.6-point spread they measured across seeds."}],"correct":true,"tempting":null},{"text":[{"text":"Rank 32 needs a different alpha, so the comparison is confounded by scaling rather than by noise."}],"correct":false,"tempting":[{"text":"Names a real confound in rank comparisons, but the stated numbers are within the seed spread regardless of how alpha was set."}]},{"text":[{"text":"Three seeds is too few to estimate variance, so the noise floor itself is unreliable and no comparison can be made."}],"correct":false,"tempting":[{"text":"Demands more precision on the variance estimate while ignoring that even a rough spread already exceeds the claimed effect."}]}],"order":[0,2,1,3],"why":null},"prev":{"slug":"checkpointing-and-resuming","title":"Checkpointing and resuming","minutes":8},"next":{"slug":"multi-gpu-without-tears","title":"Multi-GPU without tears","minutes":9}}}