Designing a pipeline that is actually a gate

A gate has to be fast, or people route around it

Every CI pipeline is in a negotiation with the humans it blocks. If it takes forty minutes, people batch changes to avoid it, stop running it locally, start merging on a green from an older commit, and eventually give someone permission to bypass it “just for this release”. None of that is written down and all of it happens.

So the first design constraint is a time budget, and a useful one is: under ten minutes for the required checks, under two minutes for the checks that run on every push. Everything that cannot fit goes into a second tier that runs after merge or nightly, and does not block.

That split (blocking and non-blocking) is the single most important decision in a pipeline, and most teams have never made it explicitly.

What blocks, and what only warns

Blocks. Only things that are unambiguously wrong, deterministic, and fast:

• Compilation or type checking
• The unit test suite
• A formatter and a linter, on error-level rules only
• A secret scanner
• A dependency vulnerability check at high severity
• The migration applying cleanly to a copy of the production schema

Warns. Everything that is a judgement, slow, or occasionally wrong:

• Coverage deltas
• Performance benchmarks, which are noisy on shared runners
• Style suggestions and complexity metrics
• Full end-to-end suites, if they are slow
• Licence policy findings that need a human

A check that blocks and is sometimes wrong will be disabled within a quarter. Better to have it warn permanently than to have it removed permanently.

Test the merge result, not the branch

This is the most common structural hole, and it is silent.

Branch A passes. Branch B passes. Both merge. Main is broken, because A renamed a function and B added a call to it. Neither pull request was ever tested against the other, and both were green.

The fix is a merge queue: the platform merges each change onto the current main, runs the pipeline on that result, and only then lands it. GitHub, GitLab and Bazel-based systems all have a form of this, and bors and mergify are the older names for the same idea. On a small team the cheap version is a rule that a branch must be up to date with main before merging, which is not equivalent, because it does not protect against a change that lands during your run, but it catches most of it.

Make it reproducible, or the failures will be about the pipeline

• Pin everything. Language version, runner image, tool versions, and dependencies via a committed lockfile. npm ci, not npm install. pip install -r requirements.lock, not a range.
• No network at test time where you can avoid it. A test that downloads a package or calls an API imports the reliability of the whole internet into your gate.
• The same command locally and in CI. If the pipeline runs a bespoke shell script that nobody can run on a laptop, debugging a failure means pushing commits to a branch and waiting, which is the worst debugging loop available. Put the whole thing in a Makefile or a justfile and have CI call the same target.

The generated-code specifics

Three checks that are worth adding now and were marginal before:

Fail on a shrinking test suite. A change that deletes tests, skips tests, or reduces the assertion count should require an explicit label. This is the mechanical version of the safety-subtraction norm.

Fail on new suppressions. Count # type: ignore, eslint-disable, @SuppressWarnings, #pragma warning disable on main. If a pull request increases the count, it explains why.

Fail on unpinned or newly added dependencies without review. A CODEOWNERS entry on the lockfile is one line and forces a specific human onto every supply-chain change.

The thing to check about your own pipeline today

Ask: when did this last fail for a real reason, and what happened next?

If the answer is that it goes red often and people re-run it, the gate is already gone and you are paying for the compute anyway. If the answer is that it never goes red, either the team is unusually good or the pipeline is checking things that cannot fail; and the way to find out is to push a branch that deliberately breaks something and see whether it is caught.

That five-minute experiment is worth more than any amount of pipeline configuration, and almost nobody does it.