{"course":{"slug":"how-software-changed","title":"How Software Development Changed","url":"https://addaly.com/learn/how-software-changed","level":"intermediate","track":"work"},"lesson":{"slug":"interviews-that-still-discriminate","title":"Hiring when everyone has the same tools open","url":"https://addaly.com/learn/how-software-changed/interviews-that-still-discriminate","position":66,"of":87,"minutes":10,"answer":"Run the exercise with the tools on and score the process around them (whether the candidate runs it, reads the error, rejects an output, checks the documentation) because what became scarce is reading and judgment.","keyIdea":"Run the exercise with the tools on and score the process around them (whether the candidate runs it, reads the error, rejects an output, checks the documentation) because what became scarce is reading and judgment, not recall of syntax."},"body":{"format":"markdown","text":"## What stopped separating candidates\n\nThe algorithmic screen is finished as a filter. Every problem on the common lists has a published solution in the training data, and a remote candidate with a second window solves it perfectly in silence. You are no longer measuring problem-solving; you are measuring whether somebody chose to cheat, which correlates with nothing you want.\n\nThe standard take-home is in similar trouble. \"Build a small REST API with tests\" is a forty-minute job with assistance and produces an indistinguishable pile of competent submissions. It was always a poor instrument (it filtered hardest on who had a free weekend, which selects against carers, second jobs and anybody in a different timezone) and now it does not even filter.\n\n> Figure: What still separates candidates. Stopped working: The algorithmic screen, because every answer is published; The two-hour take-home, which is now forty minutes; Banning tools remotely, which filters for willingness to break a rule; Detection tools, which misfire on clean second-language writing. Still works, with the tools switched on: A planted defect in a clean 120-line change; A running system, real logs, and a hypothesis to abandon; Twenty thousand unfamiliar lines and one small feature; A design meeting a new constraint introduced mid-conversation; A decision they regret, followed up for the specifics. Score the process around the tool: whether they run it, read the error, ever reject an output, check the call against the documentation. That is the job now, so that is the thing to measure.\n\n## Allow the tools and watch the process\n\nThe most direct fix: run the exercise with everything switched on, and evaluate what the candidate does around the tool. That is the actual job.\n\nThings to watch for, because they separate cleanly:\n\n- Do they **run** it, or do they read it and declare it done?\n- When it fails, do they read the error message, or re-prompt immediately? Re-prompting on the first failure is the single clearest tell.\n- Do they ever **reject** an output? A candidate who accepts everything is telling you they have no independent standard.\n- Do they check the API against documentation, or trust the call signature?\n- Can they say what they would test and why, before generating the tests?\n\nSay explicitly at the start that tools are allowed and that you are interested in how they work, not whether they can recall syntax. Candidates who have been burned by contradictory rules will otherwise spend the session guessing at what you want.\n\n## Four formats that still discriminate\n\n**The planted-bug review.** Give a 120-line pull request that looks clean and contains one real defect: an off-by-one at a boundary, a missing `await`, a transaction that does not cover both writes. Forty-five minutes: find it, explain the failure it causes, say what would have caught it. This tests reading, which is the skill that actually became scarce, and it is very hard to fake because the follow-up questions are live.\n\n**The broken system.** A running service, a failing behaviour, real logs. Watch them form a hypothesis, test it, and discard it. The best signal in the whole interview is a candidate saying \"that was wrong, so it must be...\": hypothesis abandonment under evidence is rare and highly predictive.\n\n**Extend an unfamiliar codebase.** Twenty thousand lines they have never seen, one small feature, two hours, questions allowed. This measures orientation speed, which is the thing that determines how fast they will be useful.\n\n**Design with pushback.** Not \"design a URL shortener\". Give them a real constraint mid-conversation (\"the database is now in a different region\", \"legal says this data cannot leave the country\") and see whether the design bends or shatters. Also see whether they will say \"I do not know\" instead of generating.\n\n## Ask about a decision they regret\n\nOne question, and it is worth more than most of the loop: describe a technical decision you made that turned out badly, what the symptom was, and what you do differently now.\n\nIt is hard to fabricate at depth because the follow-ups need specifics: the actual failure mode, the actual month it surfaced, what the rollback cost. And it measures the thing you most want, which is whether experience turned into calibration or just into years.\n\n## Do not do these\n\n**Do not ban tools and hope.** For remote interviews you cannot enforce it, so you filter for willingness to break a rule. For in-person ones you are testing a working condition that will never recur.\n\n**Do not extend the take-home.** The fix for \"everyone finishes it\" is not eight hours. It is a different exercise. Anything over two hours is a tax on people with less free time, and you will lose good candidates who correctly decline.\n\n**Do not screen with AI-detection tools on submissions.** They do not work, they produce false positives that fall unevenly on non-native English speakers, and a wrongly rejected candidate has no way to appeal.\n\n## The limitation, stated honestly\n\nNone of these formats is a validated selection instrument. The research on structured interviews generally (same questions, same order, scored against a rubric agreed in advance) supports structure over unstructured conversation, and that finding applies here: write the rubric first. But no study shows that a planted-bug review predicts on-the-job performance, because nobody has run one.\n\nSo treat these as reasoned designs aimed at the skill you believe is now scarce, not as measurements. Write down what you are trying to detect before the loop starts, score against it afterwards, and check a year later whether the people you rated highly are the people doing well. That last step is the only thing that turns an interview process into evidence, and almost nobody does it."},"check":{"question":"Why does a candidate's reaction to the first failing run separate candidates better than whether their final code works?","options":[{"text":"Reading an error message demonstrates familiarity with the language's runtime, which is a proxy for years of experience.","correct":false,"tempting":"Treats it as an experience proxy; the point is the behaviour, which some very experienced people also get wrong."},{"text":"Final code quality is confounded by which model the candidate happened to use, so it is not comparable across candidates.","correct":false,"tempting":"Tool variance is a nuisance, but the reason for watching the reaction is about what it reveals, not about fairness of comparison."},{"text":"Re-prompting without reading is faster, so it identifies candidates who will move quickly under deadline pressure.","correct":false,"tempting":"Inverts the signal: speed without diagnosis is the behaviour that produces expensive incidents."},{"text":"Working code is reachable by anyone now; diagnosing before re-asking shows an independent standard.","correct":true}]},"prev":{"slug":"rebuilding-the-apprenticeship","title":"Tasks that are still expensive to generate","url":"https://addaly.com/learn/how-software-changed/rebuilding-the-apprenticeship"},"next":{"slug":"what-a-portfolio-proves-now","title":"What a repository proves, and what it stopped proving","url":"https://addaly.com/learn/how-software-changed/what-a-portfolio-proves-now"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Hiring when everyone has the same tools open', in How Software Development Changed. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/how-software-changed/interviews-that-still-discriminate (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Hiring when everyone has the same tools open","courseTitle":"How Software Development Changed","module":{"index":7,"slug":"skill-and-the-missing-rung","title":"Skill, hiring and the missing rung"},"answer":null,"keyIdea":"Run the exercise with the tools on and score the process around them (whether the candidate runs it, reads the error, rejects an output, checks the documentation) because what became scarce is reading and judgment, not recall of syntax.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"left":{"title":"Stopped working","items":["The algorithmic screen, because every answer is published","The two-hour take-home, which is now forty minutes","Banning tools remotely, which filters for willingness to break a rule","Detection tools, which misfire on clean second-language writing"]},"type":"compare","right":{"title":"Still works, with the tools switched on","items":["A planted defect in a clean 120-line change","A running system, real logs, and a hypothesis to abandon","Twenty thousand unfamiliar lines and one small feature","A design meeting a new constraint introduced mid-conversation","A decision they regret, followed up for the specifics"]},"title":"What still separates candidates","caption":"Score the process around the tool: whether they run it, read the error, ever reject an output, check the call against the documentation. That is the job now, so that is the thing to measure."},"alt":"Figure: What still separates candidates. Stopped working: The algorithmic screen, because every answer is published; The two-hour take-home, which is now forty minutes; Banning tools remotely, which filters for willingness to break a rule; Detection tools, which misfire on clean second-language writing. Still works, with the tools switched on: A planted defect in a clean 120-line change; A running system, real logs, and a hypothesis to abandon; Twenty thousand unfamiliar lines and one small feature; A design meeting a new constraint introduced mid-conversation; A decision they regret, followed up for the specifics. Score the process around the tool: whether they run it, read the error, ever reject an output, check the call against the documentation. That is the job now, so that is the thing to measure."}},"blocks":[{"t":"h2","id":"what-stopped-separating-candidates","runs":[{"text":"What stopped separating candidates"}]},{"t":"p","runs":[{"text":"The algorithmic screen is finished as a filter. Every problem on the common lists has a published solution in the training data, and a remote candidate with a second window solves it perfectly in silence. You are no longer measuring problem-solving; you are measuring whether somebody chose to cheat, which correlates with nothing you want."}]},{"t":"p","runs":[{"text":"The standard take-home is in similar trouble. “Build a small REST API with tests” is a forty-minute job with assistance and produces an indistinguishable pile of competent submissions. It was always a poor instrument (it filtered hardest on who had a free weekend, which selects against carers, second jobs and anybody in a different timezone) and now it does not even filter."}]},{"t":"h2","id":"allow-the-tools-and-watch-the-process","runs":[{"text":"Allow the tools and watch the process"}]},{"t":"p","runs":[{"text":"The most direct fix: run the exercise with everything switched on, and evaluate what the candidate does around the tool. That is the actual job."}]},{"t":"p","runs":[{"text":"Things to watch for, because they separate cleanly:"}]},{"t":"ul","items":[[{"text":"Do they "},{"text":"run","b":true},{"text":" it, or do they read it and declare it done?"}],[{"text":"When it fails, do they read the error message, or re-prompt immediately? Re-prompting on the first failure is the single clearest tell."}],[{"text":"Do they ever "},{"text":"reject","b":true},{"text":" an output? A candidate who accepts everything is telling you they have no independent standard."}],[{"text":"Do they check the API against documentation, or trust the call signature?"}],[{"text":"Can they say what they would test and why, before generating the tests?"}]]},{"t":"p","runs":[{"text":"Say explicitly at the start that tools are allowed and that you are interested in how they work, not whether they can recall syntax. Candidates who have been burned by contradictory rules will otherwise spend the session guessing at what you want."}]},{"t":"h2","id":"four-formats-that-still-discriminate","runs":[{"text":"Four formats that still discriminate"}]},{"t":"p","runs":[{"text":"The planted-bug review.","b":true},{"text":" Give a 120-line pull request that looks clean and contains one real defect: an off-by-one at a boundary, a missing "},{"text":"await","code":true},{"text":", a transaction that does not cover both writes. Forty-five minutes: find it, explain the failure it causes, say what would have caught it. This tests reading, which is the skill that actually became scarce, and it is very hard to fake because the follow-up questions are live."}]},{"t":"p","runs":[{"text":"The broken system.","b":true},{"text":" A running service, a failing behaviour, real logs. Watch them form a hypothesis, test it, and discard it. The best signal in the whole interview is a candidate saying “that was wrong, so it must be...”: hypothesis abandonment under evidence is rare and highly predictive."}]},{"t":"p","runs":[{"text":"Extend an unfamiliar codebase.","b":true},{"text":" Twenty thousand lines they have never seen, one small feature, two hours, questions allowed. This measures orientation speed, which is the thing that determines how fast they will be useful."}]},{"t":"p","runs":[{"text":"Design with pushback.","b":true},{"text":" Not “design a URL shortener”. Give them a real constraint mid-conversation (“the database is now in a different region”, “legal says this data cannot leave the country”) and see whether the design bends or shatters. Also see whether they will say “I do not know” instead of generating."}]},{"t":"h2","id":"ask-about-a-decision-they-regret","runs":[{"text":"Ask about a decision they regret"}]},{"t":"p","runs":[{"text":"One question, and it is worth more than most of the loop: describe a technical decision you made that turned out badly, what the symptom was, and what you do differently now."}]},{"t":"p","runs":[{"text":"It is hard to fabricate at depth because the follow-ups need specifics: the actual failure mode, the actual month it surfaced, what the rollback cost. And it measures the thing you most want, which is whether experience turned into calibration or just into years."}]},{"t":"h2","id":"do-not-do-these","runs":[{"text":"Do not do these"}]},{"t":"p","runs":[{"text":"Do not ban tools and hope.","b":true},{"text":" For remote interviews you cannot enforce it, so you filter for willingness to break a rule. For in-person ones you are testing a working condition that will never recur."}]},{"t":"p","runs":[{"text":"Do not extend the take-home.","b":true},{"text":" The fix for “everyone finishes it” is not eight hours. It is a different exercise. Anything over two hours is a tax on people with less free time, and you will lose good candidates who correctly decline."}]},{"t":"p","runs":[{"text":"Do not screen with AI-detection tools on submissions.","b":true},{"text":" They do not work, they produce false positives that fall unevenly on non-native English speakers, and a wrongly rejected candidate has no way to appeal."}]},{"t":"h2","id":"the-limitation-stated-honestly","runs":[{"text":"The limitation, stated honestly"}],"role":"limit"},{"t":"p","runs":[{"text":"None of these formats is a validated selection instrument. The research on structured interviews generally (same questions, same order, scored against a rubric agreed in advance) supports structure over unstructured conversation, and that finding applies here: write the rubric first. But no study shows that a planted-bug review predicts on-the-job performance, because nobody has run one."}]},{"t":"p","runs":[{"text":"So treat these as reasoned designs aimed at the skill you believe is now scarce, not as measurements. Write down what you are trying to detect before the loop starts, score against it afterwards, and check a year later whether the people you rated highly are the people doing well. That last step is the only thing that turns an interview process into evidence, and almost nobody does it."}]}],"check":{"question":[{"text":"Why does a candidate’s reaction to the first failing run separate candidates better than whether their final code works?"}],"options":[{"text":[{"text":"Reading an error message demonstrates familiarity with the language’s runtime, which is a proxy for years of experience."}],"correct":false,"tempting":[{"text":"Treats it as an experience proxy; the point is the behaviour, which some very experienced people also get wrong."}]},{"text":[{"text":"Final code quality is confounded by which model the candidate happened to use, so it is not comparable across candidates."}],"correct":false,"tempting":[{"text":"Tool variance is a nuisance, but the reason for watching the reaction is about what it reveals, not about fairness of comparison."}]},{"text":[{"text":"Re-prompting without reading is faster, so it identifies candidates who will move quickly under deadline pressure."}],"correct":false,"tempting":[{"text":"Inverts the signal: speed without diagnosis is the behaviour that produces expensive incidents."}]},{"text":[{"text":"Working code is reachable by anyone now; diagnosing before re-asking shows an independent standard."}],"correct":true,"tempting":null}],"order":[0,2,3,1],"why":null},"prev":{"slug":"rebuilding-the-apprenticeship","title":"Tasks that are still expensive to generate","minutes":10},"next":{"slug":"what-a-portfolio-proves-now","title":"What a repository proves, and what it stopped proving","minutes":9}}}