{"course":{"slug":"how-software-changed","title":"How Software Development Changed","url":"https://addaly.com/learn/how-software-changed","level":"intermediate","track":"work"},"lesson":{"slug":"ci-as-the-gate","title":"Designing a pipeline that is actually a gate","url":"https://addaly.com/learn/how-software-changed/ci-as-the-gate","position":28,"of":87,"minutes":9,"answer":"A pipeline is only a gate if it is fast enough that nobody routes around it and tests the merge result rather than the branch, so blocking checks must be limited to what is deterministic.","keyIdea":"A pipeline is only a gate if it is fast enough that nobody routes around it and tests the merge result rather than the branch, so blocking checks must be limited to what is deterministic, quick and unambiguously wrong."},"body":{"format":"markdown","text":"## A gate has to be fast, or people route around it\n\nEvery CI pipeline is in a negotiation with the humans it blocks. If it takes forty minutes, people batch changes to avoid it, stop running it locally, start merging on a green from an older commit, and eventually give someone permission to bypass it \"just for this release\". None of that is written down and all of it happens.\n\nSo the first design constraint is a time budget, and a useful one is: **under ten minutes for the required checks, under two minutes for the checks that run on every push.** Everything that cannot fit goes into a second tier that runs after merge or nightly, and does not block.\n\nThat split (blocking and non-blocking) is the single most important decision in a pipeline, and most teams have never made it explicitly.\n\n## What blocks, and what only warns\n\n**Blocks.** Only things that are unambiguously wrong, deterministic, and fast:\n\n- Compilation or type checking\n- The unit test suite\n- A formatter and a linter, on error-level rules only\n- A secret scanner\n- A dependency vulnerability check at high severity\n- The migration applying cleanly to a copy of the production schema\n\n**Warns.** Everything that is a judgement, slow, or occasionally wrong:\n\n- Coverage deltas\n- Performance benchmarks, which are noisy on shared runners\n- Style suggestions and complexity metrics\n- Full end-to-end suites, if they are slow\n- Licence policy findings that need a human\n\nA check that blocks and is sometimes wrong will be disabled within a quarter. Better to have it warn permanently than to have it removed permanently.\n\n## Test the merge result, not the branch\n\nThis is the most common structural hole, and it is silent.\n\nBranch A passes. Branch B passes. Both merge. Main is broken, because A renamed a function and B added a call to it. Neither pull request was ever tested against the other, and both were green.\n\nThe fix is a merge queue: the platform merges each change onto the current main, runs the pipeline on that result, and only then lands it. GitHub, GitLab and Bazel-based systems all have a form of this, and `bors` and `mergify` are the older names for the same idea. On a small team the cheap version is a rule that a branch must be up to date with main before merging, which is not equivalent, because it does not protect against a change that lands during your run, but it catches most of it.\n\n## Make it reproducible, or the failures will be about the pipeline\n\n- **Pin everything.** Language version, runner image, tool versions, and dependencies via a committed lockfile. `npm ci`, not `npm install`. `pip install -r requirements.lock`, not a range.\n- **No network at test time** where you can avoid it. A test that downloads a package or calls an API imports the reliability of the whole internet into your gate.\n- **The same command locally and in CI.** If the pipeline runs a bespoke shell script that nobody can run on a laptop, debugging a failure means pushing commits to a branch and waiting, which is the worst debugging loop available. Put the whole thing in a `Makefile` or a `justfile` and have CI call the same target.\n\n## The generated-code specifics\n\nThree checks that are worth adding now and were marginal before:\n\n**Fail on a shrinking test suite.** A change that deletes tests, skips tests, or reduces the assertion count should require an explicit label. This is the mechanical version of the safety-subtraction norm.\n\n**Fail on new suppressions.** Count `# type: ignore`, `eslint-disable`, `@SuppressWarnings`, `#pragma warning disable` on main. If a pull request increases the count, it explains why.\n\n**Fail on unpinned or newly added dependencies without review.** A `CODEOWNERS` entry on the lockfile is one line and forces a specific human onto every supply-chain change.\n\n## The thing to check about your own pipeline today\n\nAsk: **when did this last fail for a real reason, and what happened next?**\n\nIf the answer is that it goes red often and people re-run it, the gate is already gone and you are paying for the compute anyway. If the answer is that it never goes red, either the team is unusually good or the pipeline is checking things that cannot fail; and the way to find out is to push a branch that deliberately breaks something and see whether it is caught.\n\nThat five-minute experiment is worth more than any amount of pipeline configuration, and almost nobody does it."},"check":{"question":"Two pull requests each pass CI. One renames a helper function; the other adds a call to the old name. Both merge within an hour and main breaks. What does this reveal about the pipeline?","options":[{"text":"The test suite lacks integration coverage, since a unit-level suite cannot catch a cross-module rename.","correct":false,"tempting":"Misdiagnoses the level: compilation or type checking would catch this instantly, on the combined code, which is what was never built."},{"text":"Branch protection was not requiring an up-to-date branch, which would have forced a rebase and caught the conflict.","correct":false,"tempting":"Names a mitigation that helps, but it does not protect against a change landing during your own pipeline run."},{"text":"It verified each branch in isolation and never built the combination that was actually merged.","correct":true},{"text":"The rename should have been done with a deprecation shim, so the failure is a process problem rather than a pipeline one.","correct":false,"tempting":"A reasonable practice for public interfaces, but it would leave the same gap for every change that is not a rename."}]},"prev":{"slug":"flaky-tests","title":"A suite that lies intermittently","url":"https://addaly.com/learn/how-software-changed/flaky-tests"},"next":{"slug":"release-as-verification","title":"Verification that continues after merge","url":"https://addaly.com/learn/how-software-changed/release-as-verification"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Designing a pipeline that is actually a gate', in How Software Development Changed. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/how-software-changed/ci-as-the-gate (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Designing a pipeline that is actually a gate","courseTitle":"How Software Development Changed","module":{"index":3,"slug":"oracles-and-verification","title":"Building oracles that stay ahead"},"answer":null,"keyIdea":"A pipeline is only a gate if it is fast enough that nobody routes around it and tests the merge result rather than the branch, so blocking checks must be limited to what is deterministic, quick and unambiguously wrong.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"a-gate-has-to-be-fast-or-people-route-around-it","runs":[{"text":"A gate has to be fast, or people route around it"}]},{"t":"p","runs":[{"text":"Every CI pipeline is in a negotiation with the humans it blocks. If it takes forty minutes, people batch changes to avoid it, stop running it locally, start merging on a green from an older commit, and eventually give someone permission to bypass it “just for this release”. None of that is written down and all of it happens."}]},{"t":"p","runs":[{"text":"So the first design constraint is a time budget, and a useful one is: "},{"text":"under ten minutes for the required checks, under two minutes for the checks that run on every push.","b":true},{"text":" Everything that cannot fit goes into a second tier that runs after merge or nightly, and does not block."}]},{"t":"p","runs":[{"text":"That split (blocking and non-blocking) is the single most important decision in a pipeline, and most teams have never made it explicitly."}]},{"t":"h2","id":"what-blocks-and-what-only-warns","runs":[{"text":"What blocks, and what only warns"}]},{"t":"p","runs":[{"text":"Blocks.","b":true},{"text":" Only things that are unambiguously wrong, deterministic, and fast:"}]},{"t":"ul","items":[[{"text":"Compilation or type checking"}],[{"text":"The unit test suite"}],[{"text":"A formatter and a linter, on error-level rules only"}],[{"text":"A secret scanner"}],[{"text":"A dependency vulnerability check at high severity"}],[{"text":"The migration applying cleanly to a copy of the production schema"}]]},{"t":"p","runs":[{"text":"Warns.","b":true},{"text":" Everything that is a judgement, slow, or occasionally wrong:"}]},{"t":"ul","items":[[{"text":"Coverage deltas"}],[{"text":"Performance benchmarks, which are noisy on shared runners"}],[{"text":"Style suggestions and complexity metrics"}],[{"text":"Full end-to-end suites, if they are slow"}],[{"text":"Licence policy findings that need a human"}]]},{"t":"p","runs":[{"text":"A check that blocks and is sometimes wrong will be disabled within a quarter. Better to have it warn permanently than to have it removed permanently."}]},{"t":"h2","id":"test-the-merge-result-not-the-branch","runs":[{"text":"Test the merge result, not the branch"}]},{"t":"p","runs":[{"text":"This is the most common structural hole, and it is silent."}]},{"t":"p","runs":[{"text":"Branch A passes. Branch B passes. Both merge. Main is broken, because A renamed a function and B added a call to it. Neither pull request was ever tested against the other, and both were green."}]},{"t":"p","runs":[{"text":"The fix is a merge queue: the platform merges each change onto the current main, runs the pipeline on that result, and only then lands it. GitHub, GitLab and Bazel-based systems all have a form of this, and "},{"text":"bors","code":true},{"text":" and "},{"text":"mergify","code":true},{"text":" are the older names for the same idea. On a small team the cheap version is a rule that a branch must be up to date with main before merging, which is not equivalent, because it does not protect against a change that lands during your run, but it catches most of it."}]},{"t":"h2","id":"make-it-reproducible-or-the-failures-will-be-about-the-pipel","runs":[{"text":"Make it reproducible, or the failures will be about the pipeline"}]},{"t":"ul","items":[[{"text":"Pin everything.","b":true},{"text":" Language version, runner image, tool versions, and dependencies via a committed lockfile. "},{"text":"npm ci","code":true},{"text":", not "},{"text":"npm install","code":true},{"text":". "},{"text":"pip install -r requirements.lock","code":true},{"text":", not a range."}],[{"text":"No network at test time","b":true},{"text":" where you can avoid it. A test that downloads a package or calls an API imports the reliability of the whole internet into your gate."}],[{"text":"The same command locally and in CI.","b":true},{"text":" If the pipeline runs a bespoke shell script that nobody can run on a laptop, debugging a failure means pushing commits to a branch and waiting, which is the worst debugging loop available. Put the whole thing in a "},{"text":"Makefile","code":true},{"text":" or a "},{"text":"justfile","code":true},{"text":" and have CI call the same target."}]]},{"t":"h2","id":"the-generated-code-specifics","runs":[{"text":"The generated-code specifics"}]},{"t":"p","runs":[{"text":"Three checks that are worth adding now and were marginal before:"}]},{"t":"p","runs":[{"text":"Fail on a shrinking test suite.","b":true},{"text":" A change that deletes tests, skips tests, or reduces the assertion count should require an explicit label. This is the mechanical version of the safety-subtraction norm."}]},{"t":"p","runs":[{"text":"Fail on new suppressions.","b":true},{"text":" Count "},{"text":"# type: ignore","code":true},{"text":", "},{"text":"eslint-disable","code":true},{"text":", "},{"text":"@SuppressWarnings","code":true},{"text":", "},{"text":"#pragma warning disable","code":true},{"text":" on main. If a pull request increases the count, it explains why."}]},{"t":"p","runs":[{"text":"Fail on unpinned or newly added dependencies without review.","b":true},{"text":" A "},{"text":"CODEOWNERS","code":true},{"text":" entry on the lockfile is one line and forces a specific human onto every supply-chain change."}]},{"t":"h2","id":"the-thing-to-check-about-your-own-pipeline-today","runs":[{"text":"The thing to check about your own pipeline today"}]},{"t":"p","runs":[{"text":"Ask: "},{"text":"when did this last fail for a real reason, and what happened next?","b":true}]},{"t":"p","runs":[{"text":"If the answer is that it goes red often and people re-run it, the gate is already gone and you are paying for the compute anyway. If the answer is that it never goes red, either the team is unusually good or the pipeline is checking things that cannot fail; and the way to find out is to push a branch that deliberately breaks something and see whether it is caught."}]},{"t":"p","runs":[{"text":"That five-minute experiment is worth more than any amount of pipeline configuration, and almost nobody does it."}]}],"check":{"question":[{"text":"Two pull requests each pass CI. One renames a helper function; the other adds a call to the old name. Both merge within an hour and main breaks. What does this reveal about the pipeline?"}],"options":[{"text":[{"text":"The test suite lacks integration coverage, since a unit-level suite cannot catch a cross-module rename."}],"correct":false,"tempting":[{"text":"Misdiagnoses the level: compilation or type checking would catch this instantly, on the combined code, which is what was never built."}]},{"text":[{"text":"Branch protection was not requiring an up-to-date branch, which would have forced a rebase and caught the conflict."}],"correct":false,"tempting":[{"text":"Names a mitigation that helps, but it does not protect against a change landing during your own pipeline run."}]},{"text":[{"text":"It verified each branch in isolation and never built the combination that was actually merged."}],"correct":true,"tempting":null},{"text":[{"text":"The rename should have been done with a deprecation shim, so the failure is a process problem rather than a pipeline one."}],"correct":false,"tempting":[{"text":"A reasonable practice for public interfaces, but it would leave the same gap for every change that is not a rename."}]}],"order":[1,2,0,3],"why":null},"prev":{"slug":"flaky-tests","title":"A suite that lies intermittently","minutes":9},"next":{"slug":"release-as-verification","title":"Verification that continues after merge","minutes":9}}}