{"course":{"slug":"how-software-changed","title":"How Software Development Changed","url":"https://addaly.com/learn/how-software-changed","level":"intermediate","track":"work"},"lesson":{"slug":"scanning-what-you-did-not-write","title":"Scanners, and the trick that makes them usable","url":"https://addaly.com/learn/how-software-changed/scanning-what-you-did-not-write","position":73,"of":87,"minutes":9,"answer":"Scan for the mechanical vulnerability classes and block only on findings the change introduced.","keyIdea":"Scan for the mechanical vulnerability classes and block only on findings the change introduced, because a baseline ratchet is the difference between a tool people act on and a wall of pre-existing findings that trains everyone to ignore alerts."},"body":{"format":"markdown","text":"## The asymmetry that makes automation necessary\n\nCode volume rose. Human attention did not. Scanners are one of the few things in this picture that scale the same way generation does, so the strategy is to automate every mechanical class in order to spend human attention on the classes that need judgment.\n\nThat is the whole argument. The rest is which tools, where they run, and the one configuration choice that decides whether anybody looks at the output.\n\n## The free toolchain, by what it catches\n\nEvery tool here is free and open source, runs locally, and needs no account.\n\n**Pattern-level static analysis.** `semgrep` matches structural patterns in source: concatenated SQL, disabled certificate verification, weak hash calls, dangerous deserialisation. Fast, language-agnostic, and its community rule set covers most of the previous lesson's list.\n\n**Dataflow analysis.** CodeQL asks a harder question: does attacker-controlled input reach a dangerous sink, through however many function calls? Slower, deeper, free for open-source projects. This is what catches the injection that `semgrep` misses because the concatenation happens three functions away from the request.\n\n**Language-specific linters.** `bandit` for Python, `gosec` for Go, `brakeman` for Rails. Narrow and well-tuned.\n\n**Dependency vulnerabilities.** `osv-scanner`, `trivy`, `pip-audit`, `cargo audit`, `npm audit`. These check what you have against public advisory databases. Seconds to run.\n\n**Secrets.** `gitleaks` or `trufflehog`, as a pre-commit hook.\n\n**Running applications.** OWASP ZAP for dynamic testing against a deployed instance. Finds things no source analysis can, such as a misconfigured header or an endpoint nobody knew was exposed.\n\n## The trick: only fail on what is new\n\nThis is the configuration that determines whether the whole exercise works.\n\nTurn a scanner on an existing codebase and you get hundreds of findings. Somebody triages forty, gets bored, and the pipeline gets a permanent exemption. Within a month the scanner is decoration, and, worse, people have learned that its output does not require action, which is a habit that outlasts the tool.\n\nSo compare against a baseline and block only on findings the change introduced.\n\n```bash\nsemgrep --config=auto --baseline-commit=origin/main --error\n```\n\nThat is a ratchet: the existing debt is recorded and does not block anybody, while nothing new gets added. It converts an unwinnable cleanup into a boundary you can actually hold, and the number goes down over time because people fix things while they are already in the file.\n\n> Figure: Where each check runs. Steps: 1. Pre-commit: secret scanning only (Under a second on a normal diff. Anything slower gets switched off, and it will be.); 2. Pull request: the fast ones, blocking on new findings (Pattern analysis against a baseline commit, dependency advisories, the language linter. Minutes.); 3. Nightly: the slow ones, blocking nothing (Dataflow analysis, a full dependency inventory, a dynamic scan against staging.); 4. A human: the classes no rule can express (Authorisation, business logic, and the design that is a correct implementation of a wrong decision.). The baseline is the configuration that decides whether anybody acts on the output. Existing debt is recorded and blocks nobody, while nothing new gets added.\n\n## Where each one runs\n\n- **Pre-commit:** secret scanning only. It must finish in under a second or people disable it, and they will.\n- **Pull request:** `semgrep` with a baseline, dependency scanning, and the language linter. Minutes, blocking on new findings.\n- **Nightly or weekly:** CodeQL, full dependency inventory, ZAP against staging. Slow things belong on a schedule, not in the path of a change.\n\n## What scanners cannot do, by mechanism\n\n**Authorisation.** Covered earlier and worth repeating because it is the most common serious finding: a tool cannot know who should be allowed to read which record. There is no rule to write.\n\n**Business logic.** A discount that can be applied twice, a refund that exceeds the payment, a state machine with a path from cancelled back to shipped. All valid code.\n\n**Design flaws.** Storing something you should never have collected. A token with no expiry. These are correct implementations of a wrong decision.\n\n**Anything they have no rule for.** Static analysis is a library of known patterns. A novel mistake is invisible by construction.\n\n## The failure mode to watch\n\nA noisy scanner is worse than no scanner, and the mechanism is behavioural rather than technical. Each false positive teaches the team that alerts from this tool are usually wrong. After thirty of them, a true positive arrives and gets dismissed in four seconds by somebody who has learned, correctly, what the base rate is.\n\nSo tune aggressively. Delete rules that produce noise in your codebase, even good rules, even ones your policy document likes. A small set of rules people believe is worth far more than a comprehensive set people skip.\n\n## Do not oversell this\n\nRunning these tools does not make generated code safe. It makes a specific, enumerable set of mistakes unlikely, which is valuable and is not the same claim.\n\nThe honest framing for a management conversation: these tools handle the mechanical classes so that review time goes to authorisation, business logic and design, which are the classes that actually cause the incidents you will be explaining."},"check":{"question":"Why does configuring a scanner to block only on new findings matter more than which rule set it uses?","options":[{"text":"New findings are more likely to be genuine, because recently written code is less well tested than older code.","correct":false,"tempting":"Assumes a quality difference by age; the argument is about the volume people face on day one."},{"text":"Baseline comparison reduces scan time substantially, which is what keeps the tool inside the pull request loop.","correct":false,"tempting":"Most scanners still analyse the whole tree; the baseline filters output rather than saving time."},{"text":"Blocking on all findings would prevent any deployment until the entire backlog is cleared, which is not feasible.","correct":false,"tempting":"Describes the immediate obstacle but not why the habit it creates persists after the exemption."},{"text":"A wall of pre-existing findings gets exempted, and the team learns that this tool's alerts need no action.","correct":true}]},"prev":{"slug":"instructions-hidden-in-data","title":"Instructions hidden in the things an agent reads","url":"https://addaly.com/learn/how-software-changed/instructions-hidden-in-data"},"next":{"slug":"threat-modelling-a-change","title":"Four questions before you merge","url":"https://addaly.com/learn/how-software-changed/threat-modelling-a-change"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Scanners, and the trick that makes them usable', in How Software Development Changed. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/how-software-changed/scanning-what-you-did-not-write (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Scanners, and the trick that makes them usable","courseTitle":"How Software Development Changed","module":{"index":8,"slug":"provenance-security-and-law","title":"Provenance, security and the unsettled law"},"answer":null,"keyIdea":"Scan for the mechanical vulnerability classes and block only on findings the change introduced, because a baseline ratchet is the difference between a tool people act on and a wall of pre-existing findings that trains everyone to ignore alerts.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"flow","steps":[{"label":"Pre-commit: secret scanning only","detail":"Under a second on a normal diff. Anything slower gets switched off, and it will be."},{"label":"Pull request: the fast ones, blocking on new findings","detail":"Pattern analysis against a baseline commit, dependency advisories, the language linter. Minutes."},{"label":"Nightly: the slow ones, blocking nothing","detail":"Dataflow analysis, a full dependency inventory, a dynamic scan against staging."},{"label":"A human: the classes no rule can express","detail":"Authorisation, business logic, and the design that is a correct implementation of a wrong decision."}],"title":"Where each check runs","caption":"The baseline is the configuration that decides whether anybody acts on the output. Existing debt is recorded and blocks nobody, while nothing new gets added."},"alt":"Figure: Where each check runs. Steps: 1. Pre-commit: secret scanning only (Under a second on a normal diff. Anything slower gets switched off, and it will be.); 2. Pull request: the fast ones, blocking on new findings (Pattern analysis against a baseline commit, dependency advisories, the language linter. Minutes.); 3. Nightly: the slow ones, blocking nothing (Dataflow analysis, a full dependency inventory, a dynamic scan against staging.); 4. A human: the classes no rule can express (Authorisation, business logic, and the design that is a correct implementation of a wrong decision.). The baseline is the configuration that decides whether anybody acts on the output. Existing debt is recorded and blocks nobody, while nothing new gets added."}},"blocks":[{"t":"h2","id":"the-asymmetry-that-makes-automation-necessary","runs":[{"text":"The asymmetry that makes automation necessary"}]},{"t":"p","runs":[{"text":"Code volume rose. Human attention did not. Scanners are one of the few things in this picture that scale the same way generation does, so the strategy is to automate every mechanical class in order to spend human attention on the classes that need judgment."}]},{"t":"p","runs":[{"text":"That is the whole argument. The rest is which tools, where they run, and the one configuration choice that decides whether anybody looks at the output."}]},{"t":"h2","id":"the-free-toolchain-by-what-it-catches","runs":[{"text":"The free toolchain, by what it catches"}]},{"t":"p","runs":[{"text":"Every tool here is free and open source, runs locally, and needs no account."}]},{"t":"p","runs":[{"text":"Pattern-level static analysis.","b":true},{"text":" "},{"text":"semgrep","code":true},{"text":" matches structural patterns in source: concatenated SQL, disabled certificate verification, weak hash calls, dangerous deserialisation. Fast, language-agnostic, and its community rule set covers most of the previous lesson’s list."}]},{"t":"p","runs":[{"text":"Dataflow analysis.","b":true},{"text":" CodeQL asks a harder question: does attacker-controlled input reach a dangerous sink, through however many function calls? Slower, deeper, free for open-source projects. This is what catches the injection that "},{"text":"semgrep","code":true},{"text":" misses because the concatenation happens three functions away from the request."}]},{"t":"p","runs":[{"text":"Language-specific linters.","b":true},{"text":" "},{"text":"bandit","code":true},{"text":" for Python, "},{"text":"gosec","code":true},{"text":" for Go, "},{"text":"brakeman","code":true},{"text":" for Rails. Narrow and well-tuned."}]},{"t":"p","runs":[{"text":"Dependency vulnerabilities.","b":true},{"text":" "},{"text":"osv-scanner","code":true},{"text":", "},{"text":"trivy","code":true},{"text":", "},{"text":"pip-audit","code":true},{"text":", "},{"text":"cargo audit","code":true},{"text":", "},{"text":"npm audit","code":true},{"text":". These check what you have against public advisory databases. Seconds to run."}]},{"t":"p","runs":[{"text":"Secrets.","b":true},{"text":" "},{"text":"gitleaks","code":true},{"text":" or "},{"text":"trufflehog","code":true},{"text":", as a pre-commit hook."}]},{"t":"p","runs":[{"text":"Running applications.","b":true},{"text":" OWASP ZAP for dynamic testing against a deployed instance. Finds things no source analysis can, such as a misconfigured header or an endpoint nobody knew was exposed."}]},{"t":"h2","id":"the-trick-only-fail-on-what-is-new","runs":[{"text":"The trick: only fail on what is new"}]},{"t":"p","runs":[{"text":"This is the configuration that determines whether the whole exercise works."}]},{"t":"p","runs":[{"text":"Turn a scanner on an existing codebase and you get hundreds of findings. Somebody triages forty, gets bored, and the pipeline gets a permanent exemption. Within a month the scanner is decoration, and, worse, people have learned that its output does not require action, which is a habit that outlasts the tool."}]},{"t":"p","runs":[{"text":"So compare against a baseline and block only on findings the change introduced."}]},{"t":"code","lang":"bash","label":"Bash","text":"semgrep --config=auto --baseline-commit=origin/main --error"},{"t":"p","runs":[{"text":"That is a ratchet: the existing debt is recorded and does not block anybody, while nothing new gets added. It converts an unwinnable cleanup into a boundary you can actually hold, and the number goes down over time because people fix things while they are already in the file."}]},{"t":"h2","id":"where-each-one-runs","runs":[{"text":"Where each one runs"}]},{"t":"ul","items":[[{"text":"Pre-commit:","b":true},{"text":" secret scanning only. It must finish in under a second or people disable it, and they will."}],[{"text":"Pull request:","b":true},{"text":" "},{"text":"semgrep","code":true},{"text":" with a baseline, dependency scanning, and the language linter. Minutes, blocking on new findings."}],[{"text":"Nightly or weekly:","b":true},{"text":" CodeQL, full dependency inventory, ZAP against staging. Slow things belong on a schedule, not in the path of a change."}]]},{"t":"h2","id":"what-scanners-cannot-do-by-mechanism","runs":[{"text":"What scanners cannot do, by mechanism"}]},{"t":"p","runs":[{"text":"Authorisation.","b":true},{"text":" Covered earlier and worth repeating because it is the most common serious finding: a tool cannot know who should be allowed to read which record. There is no rule to write."}]},{"t":"p","runs":[{"text":"Business logic.","b":true},{"text":" A discount that can be applied twice, a refund that exceeds the payment, a state machine with a path from cancelled back to shipped. All valid code."}]},{"t":"p","runs":[{"text":"Design flaws.","b":true},{"text":" Storing something you should never have collected. A token with no expiry. These are correct implementations of a wrong decision."}]},{"t":"p","runs":[{"text":"Anything they have no rule for.","b":true},{"text":" Static analysis is a library of known patterns. A novel mistake is invisible by construction."}]},{"t":"h2","id":"the-failure-mode-to-watch","runs":[{"text":"The failure mode to watch"}]},{"t":"p","runs":[{"text":"A noisy scanner is worse than no scanner, and the mechanism is behavioural rather than technical. Each false positive teaches the team that alerts from this tool are usually wrong. After thirty of them, a true positive arrives and gets dismissed in four seconds by somebody who has learned, correctly, what the base rate is."}]},{"t":"p","runs":[{"text":"So tune aggressively. Delete rules that produce noise in your codebase, even good rules, even ones your policy document likes. A small set of rules people believe is worth far more than a comprehensive set people skip."}]},{"t":"h2","id":"do-not-oversell-this","runs":[{"text":"Do not oversell this"}]},{"t":"p","runs":[{"text":"Running these tools does not make generated code safe. It makes a specific, enumerable set of mistakes unlikely, which is valuable and is not the same claim."}]},{"t":"p","runs":[{"text":"The honest framing for a management conversation: these tools handle the mechanical classes so that review time goes to authorisation, business logic and design, which are the classes that actually cause the incidents you will be explaining."}]}],"check":{"question":[{"text":"Why does configuring a scanner to block only on new findings matter more than which rule set it uses?"}],"options":[{"text":[{"text":"New findings are more likely to be genuine, because recently written code is less well tested than older code."}],"correct":false,"tempting":[{"text":"Assumes a quality difference by age; the argument is about the volume people face on day one."}]},{"text":[{"text":"Baseline comparison reduces scan time substantially, which is what keeps the tool inside the pull request loop."}],"correct":false,"tempting":[{"text":"Most scanners still analyse the whole tree; the baseline filters output rather than saving time."}]},{"text":[{"text":"Blocking on all findings would prevent any deployment until the entire backlog is cleared, which is not feasible."}],"correct":false,"tempting":[{"text":"Describes the immediate obstacle but not why the habit it creates persists after the exemption."}]},{"text":[{"text":"A wall of pre-existing findings gets exempted, and the team learns that this tool’s alerts need no action."}],"correct":true,"tempting":null}],"order":[1,3,2,0],"why":null},"prev":{"slug":"instructions-hidden-in-data","title":"Instructions hidden in the things an agent reads","minutes":10},"next":{"slug":"threat-modelling-a-change","title":"Four questions before you merge","minutes":8}}}