Insecure defaults, inherited

Where the insecurity comes from

A generator produces what is typical in its training data. Its training data is public code. Public code is, on average, old, written under deadline, copied from an answer site, and never security-reviewed. The output inherits that distribution, and the inheritance is mechanical rather than malicious.

So the defects are not exotic. They are the ordinary ones, arriving at the rate at which you generate.

What actually turns up

String-built SQL. Parameterised queries are common in good code and concatenation is common in tutorial code. Both are in the corpus.

Old cryptography. MD5 or SHA-1 for password storage, because a decade of examples did that. ECB mode. A hard-coded initialisation vector. Math.random() or an unseeded PRNG for a session token or a password reset link, which is a full account takeover rather than a subtle weakness.

Verification switched off. verify=False on a Python request, rejectUnauthorized: false in Node, InsecureSkipVerify: true in Go. These appear constantly in public code because they are how people get past a certificate error while debugging, and the debugging line ships.

Deserialisation of untrusted input. pickle.loads on a request body is remote code execution, stated plainly.

Permissive by default. CORS set to *, an S3 bucket policy wider than needed, a database user with more rights than the application uses, an XML parser with external entities enabled.

The class no tool finds

One category is different in kind, and it is the most common serious finding in real applications: missing authorisation.

Ask for an endpoint that returns an invoice and you will very likely get authentication (is this a logged-in user) and very likely not get authorisation: is this their invoice.

@app.get("/invoices/{invoice_id}")
def get_invoice(invoice_id: int, user = Depends(current_user)):
    return db.query(Invoice).get(invoice_id)   # whose invoice?

That code is correct-looking, passes its tests, and lets any logged-in customer read every invoice in the system by changing a number in the URL. It is the top entry in the OWASP application security risks list for a reason.

The mechanism is worth being precise about. The model cannot know this is wrong, because the rule it violates is not in the code. It lives in a sentence somebody said in a meeting: a customer can see their own invoices and an administrator can see all of them, except finance staff, who can see any invoice but cannot see the customer’s address. Nothing in any corpus contains that. It is not a knowledge gap; it is an information that does not exist outside your organisation.

This is also why no scanner will find it. A static analyser can see that a query runs; it cannot know who should be allowed to run it.

The measurements, and what they are worth

Two studies are worth knowing.

Pearce and colleagues, Asleep at the Keyboard (2021), generated programs for scenarios drawn from the MITRE top-25 weakness list and found roughly 40 per cent of the resulting programs contained a relevant vulnerability.

Perry and colleagues at Stanford (published 2023) ran a controlled study with participants writing security-sensitive functions with and without an assistant. Those with the assistant wrote less secure code, and, separately, were more likely to believe their code was secure. That second result is the one to carry. It is the competence illusion from module seven, measured on the outcome that hurts most.

Both studies used models that are now several generations old, in constrained settings, with small samples. The specific percentages should not be quoted as current. The mechanism (a distribution containing insecure patterns, plus a reader whose confidence tracks fluency) has not changed.

What to do

Make the authorisation question a required review step. One sentence: show me the line that checks this user is allowed to touch this object. If the answer is a gesture at middleware, ask which middleware and what it checks. This single question finds more real vulnerabilities than any tool you can buy.

Ask for the secure variant explicitly, and still check. Adding “use parameterised queries and a constant-time comparison” to a request measurably improves the output, and it does not make the output trustworthy. It shifts the distribution; it does not verify anything.

Put the mechanical classes in the pipeline. Free tools catch concatenated SQL, weak hashes and disabled verification reliably: that is the next lesson. Automating the mechanical classes is how you buy human attention for the authorisation question, which is the one that needs a person.

Write the negative test. For every endpoint that returns something owned by somebody: a test where user B requests user A’s object and expects a 403 or 404. Generated suites almost never contain this test, because the happy path is what examples show.