How Software Development ChangedLesson 71 of 87

Contents

How Software Development Changed

Lesson 71 of 87. Sign in to keep your place.

Module 8 · Provenance, security and the unsettled law

What leaves the building when you type

9 min read

Know what is actually sent

A completion request is not just the line you are on. Depending on the tool it may include the surrounding file, other open tabs, files the tool judged related, a repository index, your recent edits, the last several terminal commands, and the diff you are about to commit.

The first useful action is to find out. Most tools document their context assembly somewhere unglamorous, and some will show you the request. If you cannot find out what is sent, assume the whole working tree, because that is the direction the products are moving.

The two questions that matter

They are separate and people routinely conflate them.

Is it retained? How long does the provider keep the request, in logs or otherwise?

Is it trained on? Does it enter a training set for a future model?

The answers differ sharply by tier, and almost always in the same direction: consumer and free tiers are permissive, business and enterprise tiers are restrictive and contractual. Many free tiers train on input by default with an opt-out buried in settings; most paid business tiers commit not to, in the agreement rather than the marketing page.

Read the data processing addendum, not the landing page. The landing page says “your data is secure”, which is not a statement about retention or training. The addendum says how long, where, and to whom.

The concrete leaks

These are the ones that actually happen, and none of them involve a sophisticated attacker.

  • .env open in a tab while completions run in the next file.
  • A test fixture built from a production export, with real names, real addresses and a real card BIN.
  • A connection string pasted into a chat to ask why it fails.
  • A stack trace pasted whole, carrying a session token in a header dump.
  • A migration script containing a customer identifier used as an example.

The pattern is that the secret was not in the file you were thinking about.

Controls that work

Do not have real secrets in the repository at all. This is the only control that fully survives a mistake. Secrets live in a manager (your cloud’s own, or sops with age, or Vault) and reach the process as environment variables at run time. direnv with an untracked .envrc is a free version that works on a laptop.

Use the tool’s ignore file. Most assistants honour a per-tool exclusion list alongside .gitignore: the name differs by product but the pattern is universal. Put .env*, *.pem, secrets/, fixture directories and anything containing production data in it. Verify it works rather than assuming, by checking whether completions still reference an excluded file’s contents.

Scan before the commit, not after the push. gitleaks and trufflehog are free and run as a pre-commit hook in under a second on a normal diff.

Bash
gitleaks protect --staged --redact

Generate the fixtures. Test data built by a factory (Faker or equivalent, free in every language) never contains a real person. Exporting from production “just for this test” is the most common route by which customer data ends up in a prompt.

Rotate what was exposed. If a live credential was in context, it is exposed. Not probably, not pending investigation. Rotate it and move on; the deliberation costs more than the rotation.

The regulatory layer, briefly and without advice

If a prompt contains personal data, sending it to a provider is a processing activity, and in most data-protection regimes that requires a lawful basis, a processor agreement, and a record. Code containing customer data is customer data: the fact that it is inside a Python file changes nothing.

Regimes differ in ways that matter operationally: where processing may occur, what must be disclosed, how long records are kept. This course is not the place to resolve any of that and cannot advise you. The operational point is narrower and holds everywhere: you cannot verify a retention claim, so the only control you truly hold is what you send. Data minimisation is not a legal strategy, it is the one thing that does not depend on somebody else keeping a promise.

Your own logs are part of this

Teams that route requests through an internal gateway for cost control usually log the full prompt. That log is now a store of source code and whatever was in context, sitting in your observability stack with whatever retention that has, readable by whoever can read dashboards.

If you build a gateway, decide the retention deliberately, redact on the way in rather than on the way out, and put the log behind the same access controls as the source it contains. Most teams discover this log exists during an audit.

Every lesson here is open. Your progress is recorded once you pass the AI at Work exam.

Check your understanding

A team moves to an enterprise tier whose contract forbids training on their inputs. What risk does this leave untouched?

Pick the one you would defend. Your answer stays private.

Next lesson · 10 minInstructions hidden in the things an agent reads

Stuck on something?Ask Larissa, your AI tutor

How to cite: Addaly (2026) ‘What leaves the building when you type’, in How Software Development Changed. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/how-software-changed/what-leaves-the-building (Accessed: 13 October 2026).