{"course":{"slug":"how-software-changed","title":"How Software Development Changed","url":"https://addaly.com/learn/how-software-changed","level":"intermediate","track":"work"},"lesson":{"slug":"the-questions-that-stay-open","title":"What this course could not tell you","url":"https://addaly.com/learn/how-software-changed/the-questions-that-stay-open","position":86,"of":87,"minutes":9,"answer":"The practical recommendations in this field rest on a small set of established mechanisms, while the headline empirical questions (productivity, skill, employment, law, maintainability) remain genuinely open.","keyIdea":"The practical recommendations in this field rest on a small set of established mechanisms, while the headline empirical questions (productivity, skill, employment, law, maintainability) remain genuinely open, so advice that depends on those is advice that expires."},"body":{"format":"markdown","text":"## A deliberate inventory\n\nEvery lesson here has flagged its own limits. This one collects them, because a course that ends by summarising its conclusions and not its ignorance is training you badly.\n\nFor each question: why it is unresolved, and what evidence would actually settle it. That second part is the useful one: it is how you judge the next study, the next blog post and the next person who tells you the answer with confidence.\n\n## Does it make teams faster overall?\n\n**Unresolved.** Controlled trials on fixed tasks show effects in both directions depending on the population and the task. Large observational deployments report throughput gains that cannot be separated from selection: teams that adopt eagerly differ from teams that do not. Self-reported savings measure belief.\n\n**What would settle it:** end-to-end delivery measures over a long period on comparable teams, with a defect and rework column, at organisations that did not choose their own condition. Expensive, slow, rarely funded, and each such study would speak only about the kind of work it covered.\n\n**What you can do meanwhile:** measure your own delivery at the end of the pipeline. Your four numbers beat any published figure for deciding what your team should do.\n\n## Does it degrade skill over a career?\n\n**Unresolved, and probably unresolvable.** The mechanism from module seven is well supported in adjacent laboratory literature. The magnitude in this setting is unknown, and getting it would require randomising people into decade-long careers.\n\n**What would settle it:** nothing clean. Expect the evidence to stay indirect (cohort comparisons, assessment data, hiring outcomes) all badly confounded by which people chose which practice.\n\n## Does the junior role come back?\n\n**Unresolved.** The hiring fall from 2023 is confounded by interest rates and a post-2021 correction, and those confounds do not disentangle with the data that exists.\n\n**What would settle it:** several years of hiring data through a period where rates and the correction are no longer moving, disaggregated by role and firm size. That means about 2028 before anybody can speak confidently, which is a long time for the people affected now.\n\n## Is training on copyrighted material lawful?\n\n**Unresolved by design: it is in front of courts.** Different statutes, different jurisdictions, rulings already pointing different ways on different facts.\n\n**What would settle it:** appellate decisions in the major jurisdictions, and legislation where it is being drafted. Years, and the answer will differ by country and possibly by use.\n\n## Can prompt injection be fixed?\n\n**Open research question.** There is no general defence today. Mitigations reduce and do not eliminate, because the model consumes instructions and data through one channel.\n\n**What would settle it:** an architecture with a real privilege separation between instruction and data, holding up under adversarial evaluation. Several approaches are being explored. None is deployed at the level a security control requires. Until then, treat it as architecture: break the trifecta.\n\n## Does generated code become unmaintainable?\n\n**Too early.** The mechanisms in module nine are plausible: more code, more duplicated concepts, no author holding the model. The oldest substantially generated codebases are around three years old, and maintenance costs become legible at five to ten.\n\n**What would settle it:** longitudinal cost data on comparable systems, which will exist around 2030 and will be confounded by everything else that changed in the meantime.\n\n## What is not open\n\nSaying clearly what *is* established matters as much, or the honesty becomes a way of avoiding every conclusion.\n\nThese are not in doubt: fluent code can be wrong, and fluency is not evidence; verification does not get cheaper at the same rate as generation, so a faster writing stage in front of a fixed review stage lengthens delivery; a model cannot know a rule that exists only in your organisation; a system consuming instructions and data in one channel will act on instructions found in data; and confidence tracks fluency rather than correctness in both the reader and the writer.\n\nEvery practical recommendation in this course rests on that list, not on the open questions. That is deliberate: advice that depends on an unsettled empirical claim is advice that expires.\n\n## How to hold this\n\nWrite down, now, what would change your mind, on the two or three questions that affect your decisions. Then when the next study arrives you will be updating rather than shopping for confirmation, and you will notice the difference.\n\n> The mark of somebody worth listening to on this subject is not what they claim. It is whether they can tell you what would make them wrong."},"check":{"question":"Why are the course's practical recommendations built on mechanisms rather than on productivity findings?","options":[{"text":"Advice resting on an unsettled empirical claim stops being correct when the claim moves.","correct":true},{"text":"Productivity studies are usually vendor-funded, so their findings cannot be trusted at all.","correct":false,"tempting":"Overstates the dismissal; the problem is that the question is genuinely unsettled, not that all evidence is corrupt."},{"text":"Mechanisms are easier to explain to a non-technical audience than statistical findings are.","correct":false,"tempting":"Pedagogical convenience, not a reason the advice would remain correct."},{"text":"Mechanisms apply universally, while productivity findings are specific to particular programming languages.","correct":false,"tempting":"Language specificity is not the main limitation of those studies; population and task are."}]},"prev":{"slug":"what-previous-automations-did","title":"What happened the last few times","url":"https://addaly.com/learn/how-software-changed/what-previous-automations-did"},"next":{"slug":"what-to-do-on-monday","title":"The next month, in order","url":"https://addaly.com/learn/how-software-changed/what-to-do-on-monday"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'What this course could not tell you', in How Software Development Changed. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/how-software-changed/the-questions-that-stay-open (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"What this course could not tell you","courseTitle":"How Software Development Changed","module":{"index":9,"slug":"what-did-not-change","title":"What did not change, and what to do now"},"answer":null,"keyIdea":"The practical recommendations in this field rest on a small set of established mechanisms, while the headline empirical questions (productivity, skill, employment, law, maintainability) remain genuinely open, so advice that depends on those is advice that expires.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"a-deliberate-inventory","runs":[{"text":"A deliberate inventory"}]},{"t":"p","runs":[{"text":"Every lesson here has flagged its own limits. This one collects them, because a course that ends by summarising its conclusions and not its ignorance is training you badly."}]},{"t":"p","runs":[{"text":"For each question: why it is unresolved, and what evidence would actually settle it. That second part is the useful one: it is how you judge the next study, the next blog post and the next person who tells you the answer with confidence."}]},{"t":"h2","id":"does-it-make-teams-faster-overall","runs":[{"text":"Does it make teams faster overall?"}]},{"t":"p","runs":[{"text":"Unresolved.","b":true},{"text":" Controlled trials on fixed tasks show effects in both directions depending on the population and the task. Large observational deployments report throughput gains that cannot be separated from selection: teams that adopt eagerly differ from teams that do not. Self-reported savings measure belief."}]},{"t":"p","runs":[{"text":"What would settle it:","b":true},{"text":" end-to-end delivery measures over a long period on comparable teams, with a defect and rework column, at organisations that did not choose their own condition. Expensive, slow, rarely funded, and each such study would speak only about the kind of work it covered."}]},{"t":"p","runs":[{"text":"What you can do meanwhile:","b":true},{"text":" measure your own delivery at the end of the pipeline. Your four numbers beat any published figure for deciding what your team should do."}]},{"t":"h2","id":"does-it-degrade-skill-over-a-career","runs":[{"text":"Does it degrade skill over a career?"}]},{"t":"p","runs":[{"text":"Unresolved, and probably unresolvable.","b":true},{"text":" The mechanism from module seven is well supported in adjacent laboratory literature. The magnitude in this setting is unknown, and getting it would require randomising people into decade-long careers."}]},{"t":"p","runs":[{"text":"What would settle it:","b":true},{"text":" nothing clean. Expect the evidence to stay indirect (cohort comparisons, assessment data, hiring outcomes) all badly confounded by which people chose which practice."}]},{"t":"h2","id":"does-the-junior-role-come-back","runs":[{"text":"Does the junior role come back?"}]},{"t":"p","runs":[{"text":"Unresolved.","b":true},{"text":" The hiring fall from 2023 is confounded by interest rates and a post-2021 correction, and those confounds do not disentangle with the data that exists."}]},{"t":"p","runs":[{"text":"What would settle it:","b":true},{"text":" several years of hiring data through a period where rates and the correction are no longer moving, disaggregated by role and firm size. That means about 2028 before anybody can speak confidently, which is a long time for the people affected now."}]},{"t":"h2","id":"is-training-on-copyrighted-material-lawful","runs":[{"text":"Is training on copyrighted material lawful?"}]},{"t":"p","runs":[{"text":"Unresolved by design: it is in front of courts.","b":true},{"text":" Different statutes, different jurisdictions, rulings already pointing different ways on different facts."}]},{"t":"p","runs":[{"text":"What would settle it:","b":true},{"text":" appellate decisions in the major jurisdictions, and legislation where it is being drafted. Years, and the answer will differ by country and possibly by use."}]},{"t":"h2","id":"can-prompt-injection-be-fixed","runs":[{"text":"Can prompt injection be fixed?"}]},{"t":"p","runs":[{"text":"Open research question.","b":true},{"text":" There is no general defence today. Mitigations reduce and do not eliminate, because the model consumes instructions and data through one channel."}]},{"t":"p","runs":[{"text":"What would settle it:","b":true},{"text":" an architecture with a real privilege separation between instruction and data, holding up under adversarial evaluation. Several approaches are being explored. None is deployed at the level a security control requires. Until then, treat it as architecture: break the trifecta."}]},{"t":"h2","id":"does-generated-code-become-unmaintainable","runs":[{"text":"Does generated code become unmaintainable?"}]},{"t":"p","runs":[{"text":"Too early.","b":true},{"text":" The mechanisms in module nine are plausible: more code, more duplicated concepts, no author holding the model. The oldest substantially generated codebases are around three years old, and maintenance costs become legible at five to ten."}]},{"t":"p","runs":[{"text":"What would settle it:","b":true},{"text":" longitudinal cost data on comparable systems, which will exist around 2030 and will be confounded by everything else that changed in the meantime."}]},{"t":"h2","id":"what-is-not-open","runs":[{"text":"What is not open"}]},{"t":"p","runs":[{"text":"Saying clearly what "},{"text":"is","i":true},{"text":" established matters as much, or the honesty becomes a way of avoiding every conclusion."}]},{"t":"p","runs":[{"text":"These are not in doubt: fluent code can be wrong, and fluency is not evidence; verification does not get cheaper at the same rate as generation, so a faster writing stage in front of a fixed review stage lengthens delivery; a model cannot know a rule that exists only in your organisation; a system consuming instructions and data in one channel will act on instructions found in data; and confidence tracks fluency rather than correctness in both the reader and the writer."}]},{"t":"p","runs":[{"text":"Every practical recommendation in this course rests on that list, not on the open questions. That is deliberate: advice that depends on an unsettled empirical claim is advice that expires."}]},{"t":"h2","id":"how-to-hold-this","runs":[{"text":"How to hold this"}]},{"t":"p","runs":[{"text":"Write down, now, what would change your mind, on the two or three questions that affect your decisions. Then when the next study arrives you will be updating rather than shopping for confirmation, and you will notice the difference."}]},{"t":"quote","runs":[{"text":"The mark of somebody worth listening to on this subject is not what they claim. It is whether they can tell you what would make them wrong."}]}],"check":{"question":[{"text":"Why are the course’s practical recommendations built on mechanisms rather than on productivity findings?"}],"options":[{"text":[{"text":"Advice resting on an unsettled empirical claim stops being correct when the claim moves."}],"correct":true,"tempting":null},{"text":[{"text":"Productivity studies are usually vendor-funded, so their findings cannot be trusted at all."}],"correct":false,"tempting":[{"text":"Overstates the dismissal; the problem is that the question is genuinely unsettled, not that all evidence is corrupt."}]},{"text":[{"text":"Mechanisms are easier to explain to a non-technical audience than statistical findings are."}],"correct":false,"tempting":[{"text":"Pedagogical convenience, not a reason the advice would remain correct."}]},{"text":[{"text":"Mechanisms apply universally, while productivity findings are specific to particular programming languages."}],"correct":false,"tempting":[{"text":"Language specificity is not the main limitation of those studies; population and task are."}]}],"order":[1,2,3,0],"why":null},"prev":{"slug":"what-previous-automations-did","title":"What happened the last few times","minutes":10},"next":{"slug":"what-to-do-on-monday","title":"The next month, in order","minutes":9}}}