{"course":{"slug":"how-software-changed","title":"How Software Development Changed","url":"https://addaly.com/learn/how-software-changed","level":"intermediate","track":"work"},"lesson":{"slug":"what-the-tool-can-know","title":"What the tool can actually know about your codebase","url":"https://addaly.com/learn/how-software-changed/what-the-tool-can-know","position":34,"of":87,"minutes":9,"answer":"A model reasons only over the text in its window, so it invents helpers it cannot see.","keyIdea":"A model reasons only over the text in its window, so it invents helpers it cannot see, and the fix is naming specific files plus a short honest conventions file rather than expecting retrieval to find the constraint that is worded differently."},"body":{"format":"markdown","text":"## It invents your helper because it cannot see your helper\n\nThe most common frustration (it rewrote a function we already have, in a style we do not use, using a library we removed last year) has one explanation, and understanding it changes how you work.\n\nA model reasons over the text it is given. That text is a window: the files you attached, the ones the tool retrieved, the recent conversation, and any standing instructions. Your repository is not in the window. Neither is your team's history, your incident from March, or the convention that everyone follows and nobody wrote down.\n\nSo when it needs a date formatter and cannot see yours, it writes one. Not because it ignored yours. Because from where it stands, yours does not exist.\n\nThis reframes the whole activity. You are not instructing a colleague who knows the codebase. You are briefing a capable contractor on their first morning, who will not ask, and who will make something up rather than stop.\n\n## Retrieval helps and does not solve it\n\nModern coding tools search the repository and pull in what looks relevant: by keyword, by embedding similarity, by following imports. This works well and it fails in a specific, predictable way: it finds files that resemble the request.\n\nWhat it misses is the thing that does not resemble the request but governs it. The middleware that already handles authentication for every route. The config value that means this environment behaves differently. The comment in an unrelated module explaining why the obvious approach breaks. Retrieval is similarity search, and the constraint you most need is often the one phrased in different words.\n\nThe practical consequence: **name the files.** Three specific paths beat any amount of asking it to look around, and it is faster.\n\n## The conventions file\n\nEvery major coding tool now reads a plain-text instructions file from the repository root. The filenames differ by vendor and will keep changing; the idea does not. Write one, keep it short, and keep it true.\n\nWhat belongs in it: things that are true across the whole repository and cannot be inferred from a single file:\n\n```markdown\n# Working in this repository\n\n## Commands\n- Test: `make test` (not pytest directly — it sets the test DB URL)\n- Type check: `make types` — must pass before any commit\n- Never run `alembic upgrade` against a real database\n\n## Conventions\n- Money is always an integer of minor units, never a float. See `lib/money.py`.\n- All datetimes are timezone-aware UTC at the boundary. `naive_now()` does not exist here.\n- HTTP calls go through `lib/http.py`, which sets timeouts and retries.\n  Do not import `requests` directly.\n- Database access lives in `repo/`. Handlers never write SQL.\n\n## Things that look wrong and are not\n- `billing/fx.py` fails open on a rate-service outage. See docs/adr/0009.\n- The duplicate `format_date` in `reports/` is deliberate; the report format\n  is frozen by contract and must not follow app-wide changes.\n```\n\nWhat does not belong: anything a reader can see from the code, restatements of general good practice, aspirations nobody follows, and anything long. A file that has grown to 400 lines is competing with the actual code for the window, and the parts nobody maintains are actively misleading.\n\n**Keep it honest, or delete it.** A conventions file that describes a policy the team abandoned is worse than none, because it will be followed.\n\n## Attention decays across a long window\n\nOne practical property worth knowing: material in the middle of a very long context is used less reliably than material at the beginning or the end. This has been measured repeatedly across models and is often called the lost-in-the-middle effect. It is a tendency, not a rule, and it varies by model and by version.\n\nThe working implications are simple:\n\n- Put constraints at the end of the message, closest to the request, not buried at the top.\n- Prefer three relevant files over thirty. Filling the window is not the same as informing it.\n- On a long session, restate the important constraint rather than trusting that it is still in view from an hour ago.\n\n## The test that tells you whether it understands\n\nBefore letting anything write code against an unfamiliar part of your system, ask it to describe that part and check the description against reality.\n\n> Describe how authentication works for the `/api/v2` routes in this repository, and list the files involved.\n\nThirty seconds to verify. If it names a middleware that does not exist, you have found out before it built a change on that assumption, and you know exactly which file to attach next.\n\nThis is the same discipline as the whole course, in miniature: the cheap check that precedes the expensive commitment."},"check":{"question":"A team adds a 600-line conventions file describing every rule, style preference and past decision. Generated code still ignores the rule about not importing `requests` directly. What is the most likely mechanism?","options":[{"text":"Conventions files are advisory and no tool enforces them, so the only reliable fix is a lint rule rather than documentation.","correct":false,"tempting":"A lint rule is the right belt-and-braces answer, but it treats the file as useless when its length is what degraded it."},{"text":"The rule contradicts common practice in training data, and a model will always prefer the pattern it has seen most often.","correct":false,"tempting":"Priors do pull towards conventional imports, but a short, prominent instruction routinely overrides them: length is the variable that changed here."},{"text":"The file is long enough that the rule sits in the middle of the window, where material is used least reliably.","correct":true},{"text":"The file is read only at session start, so the instruction has fallen out of context by the time code is generated.","correct":false,"tempting":"Assumes a mechanism that most tools do not have; instruction files are typically re-supplied, which is why length rather than staleness is the problem."}]},"prev":{"slug":"decision-records","title":"Writing down the why","url":"https://addaly.com/learn/how-software-changed/decision-records"},"next":{"slug":"interfaces-before-implementations","title":"Design the seam, generate the filling","url":"https://addaly.com/learn/how-software-changed/interfaces-before-implementations"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'What the tool can actually know about your codebase', in How Software Development Changed. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/how-software-changed/what-the-tool-can-know (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"What the tool can actually know about your codebase","courseTitle":"How Software Development Changed","module":{"index":4,"slug":"saying-what-you-want","title":"Saying what you want"},"answer":null,"keyIdea":"A model reasons only over the text in its window, so it invents helpers it cannot see, and the fix is naming specific files plus a short honest conventions file rather than expecting retrieval to find the constraint that is worded differently.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"it-invents-your-helper-because-it-cannot-see-your-helper","runs":[{"text":"It invents your helper because it cannot see your helper"}]},{"t":"p","runs":[{"text":"The most common frustration (it rewrote a function we already have, in a style we do not use, using a library we removed last year) has one explanation, and understanding it changes how you work."}]},{"t":"p","runs":[{"text":"A model reasons over the text it is given. That text is a window: the files you attached, the ones the tool retrieved, the recent conversation, and any standing instructions. Your repository is not in the window. Neither is your team’s history, your incident from March, or the convention that everyone follows and nobody wrote down."}]},{"t":"p","runs":[{"text":"So when it needs a date formatter and cannot see yours, it writes one. Not because it ignored yours. Because from where it stands, yours does not exist."}]},{"t":"p","runs":[{"text":"This reframes the whole activity. You are not instructing a colleague who knows the codebase. You are briefing a capable contractor on their first morning, who will not ask, and who will make something up rather than stop."}]},{"t":"h2","id":"retrieval-helps-and-does-not-solve-it","runs":[{"text":"Retrieval helps and does not solve it"}]},{"t":"p","runs":[{"text":"Modern coding tools search the repository and pull in what looks relevant: by keyword, by embedding similarity, by following imports. This works well and it fails in a specific, predictable way: it finds files that resemble the request."}]},{"t":"p","runs":[{"text":"What it misses is the thing that does not resemble the request but governs it. The middleware that already handles authentication for every route. The config value that means this environment behaves differently. The comment in an unrelated module explaining why the obvious approach breaks. Retrieval is similarity search, and the constraint you most need is often the one phrased in different words."}]},{"t":"p","runs":[{"text":"The practical consequence: "},{"text":"name the files.","b":true},{"text":" Three specific paths beat any amount of asking it to look around, and it is faster."}]},{"t":"h2","id":"the-conventions-file","runs":[{"text":"The conventions file"}]},{"t":"p","runs":[{"text":"Every major coding tool now reads a plain-text instructions file from the repository root. The filenames differ by vendor and will keep changing; the idea does not. Write one, keep it short, and keep it true."}]},{"t":"p","runs":[{"text":"What belongs in it: things that are true across the whole repository and cannot be inferred from a single file:"}]},{"t":"code","lang":"markdown","label":"Markdown","text":"# Working in this repository\n\n## Commands\n- Test: `make test` (not pytest directly — it sets the test DB URL)\n- Type check: `make types` — must pass before any commit\n- Never run `alembic upgrade` against a real database\n\n## Conventions\n- Money is always an integer of minor units, never a float. See `lib/money.py`.\n- All datetimes are timezone-aware UTC at the boundary. `naive_now()` does not exist here.\n- HTTP calls go through `lib/http.py`, which sets timeouts and retries.\n  Do not import `requests` directly.\n- Database access lives in `repo/`. Handlers never write SQL.\n\n## Things that look wrong and are not\n- `billing/fx.py` fails open on a rate-service outage. See docs/adr/0009.\n- The duplicate `format_date` in `reports/` is deliberate; the report format\n  is frozen by contract and must not follow app-wide changes."},{"t":"p","runs":[{"text":"What does not belong: anything a reader can see from the code, restatements of general good practice, aspirations nobody follows, and anything long. A file that has grown to 400 lines is competing with the actual code for the window, and the parts nobody maintains are actively misleading."}]},{"t":"p","runs":[{"text":"Keep it honest, or delete it.","b":true},{"text":" A conventions file that describes a policy the team abandoned is worse than none, because it will be followed."}]},{"t":"h2","id":"attention-decays-across-a-long-window","runs":[{"text":"Attention decays across a long window"}]},{"t":"p","runs":[{"text":"One practical property worth knowing: material in the middle of a very long context is used less reliably than material at the beginning or the end. This has been measured repeatedly across models and is often called the lost-in-the-middle effect. It is a tendency, not a rule, and it varies by model and by version."}]},{"t":"p","runs":[{"text":"The working implications are simple:"}]},{"t":"ul","items":[[{"text":"Put constraints at the end of the message, closest to the request, not buried at the top."}],[{"text":"Prefer three relevant files over thirty. Filling the window is not the same as informing it."}],[{"text":"On a long session, restate the important constraint rather than trusting that it is still in view from an hour ago."}]]},{"t":"h2","id":"the-test-that-tells-you-whether-it-understands","runs":[{"text":"The test that tells you whether it understands"}]},{"t":"p","runs":[{"text":"Before letting anything write code against an unfamiliar part of your system, ask it to describe that part and check the description against reality."}]},{"t":"quote","runs":[{"text":"Describe how authentication works for the "},{"text":"/api/v2","code":true},{"text":" routes in this repository, and list the files involved."}]},{"t":"p","runs":[{"text":"Thirty seconds to verify. If it names a middleware that does not exist, you have found out before it built a change on that assumption, and you know exactly which file to attach next."}]},{"t":"p","runs":[{"text":"This is the same discipline as the whole course, in miniature: the cheap check that precedes the expensive commitment."}]}],"check":{"question":[{"text":"A team adds a 600-line conventions file describing every rule, style preference and past decision. Generated code still ignores the rule about not importing "},{"text":"requests","code":true},{"text":" directly. What is the most likely mechanism?"}],"options":[{"text":[{"text":"Conventions files are advisory and no tool enforces them, so the only reliable fix is a lint rule rather than documentation."}],"correct":false,"tempting":[{"text":"A lint rule is the right belt-and-braces answer, but it treats the file as useless when its length is what degraded it."}]},{"text":[{"text":"The rule contradicts common practice in training data, and a model will always prefer the pattern it has seen most often."}],"correct":false,"tempting":[{"text":"Priors do pull towards conventional imports, but a short, prominent instruction routinely overrides them: length is the variable that changed here."}]},{"text":[{"text":"The file is long enough that the rule sits in the middle of the window, where material is used least reliably."}],"correct":true,"tempting":null},{"text":[{"text":"The file is read only at session start, so the instruction has fallen out of context by the time code is generated."}],"correct":false,"tempting":[{"text":"Assumes a mechanism that most tools do not have; instruction files are typically re-supplied, which is why length rather than staleness is the problem."}]}],"order":[3,0,1,2],"why":null},"prev":{"slug":"decision-records","title":"Writing down the why","minutes":9},"next":{"slug":"interfaces-before-implementations","title":"Design the seam, generate the filling","minutes":9}}}