Asking the model to audit itself, and what it cannot see

The general review

“Check this app for security problems” returns a paragraph. It mentions input validation, recommends HTTPS, suggests keeping dependencies updated, and concludes that no critical issues were found. It is not wrong; it is not a review. A model given a whole codebase and a vague question produces the summary of security advice it has read most often, and the summary is the same for every codebase.

The first security lesson ended with a specific question: for every route, which line checks ownership. This lesson widens that into the review the model can actually do, and marks the edge of what it can do at all.

The checklist, per route

For each file that handles a request, five questions, asked as one prompt with the file in context:

For this route, answer each with a line number or "none":
1. Which line establishes who is making the request, from the session?
2. Which line decides whether that person may perform this action on this record?
3. Which line validates the body or parameters against a schema, and does anything
   below it use a raw body or raw parameter?
4. Which lines build SQL, a shell command, or a URL the server fetches, and is any
   value in them derived from the request?
5. What does this route return when the check in 2 fails, and what does it log?

The answers are checkable. A “none” on question 2 for a route that touches a user’s record is a finding. A line number on question 3 that sits below a database call is a finding. This is the review the model is good at: reading code it can see and locating the presence or absence of a pattern. Run it over every route, and keep the output, because next month’s review is a diff against it.

What no review of the code can see

Three categories are invisible in the text, whoever reads it.

What the code does not say. The list route in this lesson’s question is ordinary working code. Nothing in it is wrong as code; what is wrong is what it omits, and an omission is visible only to a reader who knows what should be there. The checklist supplies that knowledge for the common cases. For the uncommon (the workflow where a refund can be requested twice, the invite that can be accepted by the wrong person) you have to describe what should be true and ask whether it is enforced, and that description is the specification block’s job.

What lives elsewhere. The host’s settings, the bucket’s public flag, the database role’s privileges, the DNS, the provider’s retention terms. None of it is in the repository. A code review cannot tell you the bucket is public.

What is true at runtime. A dependency with a vulnerability disclosed yesterday; a header the framework sets that the CDN strips; the middleware that a crafted request skips. Reading code says what it intends. Only running it says what it does.

Free tools for the rest

npm audit for known vulnerabilities in dependencies, read with the honesty the dependency lesson asked for.

gitleaks for secrets in the history.

semgrep (free for local use) runs pattern rules over the code and finds string-built SQL, raw HTML injection, and a few hundred other shapes, faster and more consistently than a model reading file by file.

OWASP ZAP (free, desktop or command line) is a scanner that attacks a running site: a baseline scan against the preview deployment takes ten minutes, needs no configuration, and reports missing headers, exposed files and reflected input. Point it at the preview, never at production, and never at a site that is not yours.

A security.txt file at /.well-known/security.txt with an email address, so that a stranger who finds something has a way to tell you that is not a public post. Costs nothing, and it is the difference between a quiet email and a screenshot on social media.

When to pay a person

Other people’s money, health data, anything involving children, anything with a legal duty: the categories the closing lesson of this course names as unfit for unsupervised AI code. For these, a penetration test by a person is not optional, and it is not what a model or a scanner does: a tester chains small findings into a real path, reasons about your business rather than about patterns, and tells you what an attacker would actually do first. It costs real money, from a few hundred dollars for a small scoped test upward, and it is cheaper than the alternative in every case where the alternative has happened.

The review as a habit

Before each release: the per-route checklist on changed routes, the three-account script, the bundle and history scan, npm audit, and ten minutes of ZAP against the preview. An hour, most of it automated. The generated app started with the holes the first lesson of this block listed; this is the routine that keeps it from acquiring them again, one feature at a time.