Hiring when everyone has the same tools open

What stopped separating candidates

The algorithmic screen is finished as a filter. Every problem on the common lists has a published solution in the training data, and a remote candidate with a second window solves it perfectly in silence. You are no longer measuring problem-solving; you are measuring whether somebody chose to cheat, which correlates with nothing you want.

The standard take-home is in similar trouble. “Build a small REST API with tests” is a forty-minute job with assistance and produces an indistinguishable pile of competent submissions. It was always a poor instrument (it filtered hardest on who had a free weekend, which selects against carers, second jobs and anybody in a different timezone) and now it does not even filter.

Allow the tools and watch the process

The most direct fix: run the exercise with everything switched on, and evaluate what the candidate does around the tool. That is the actual job.

Things to watch for, because they separate cleanly:

• Do they run it, or do they read it and declare it done?
• When it fails, do they read the error message, or re-prompt immediately? Re-prompting on the first failure is the single clearest tell.
• Do they ever reject an output? A candidate who accepts everything is telling you they have no independent standard.
• Do they check the API against documentation, or trust the call signature?
• Can they say what they would test and why, before generating the tests?

Say explicitly at the start that tools are allowed and that you are interested in how they work, not whether they can recall syntax. Candidates who have been burned by contradictory rules will otherwise spend the session guessing at what you want.

Four formats that still discriminate

The planted-bug review. Give a 120-line pull request that looks clean and contains one real defect: an off-by-one at a boundary, a missing await, a transaction that does not cover both writes. Forty-five minutes: find it, explain the failure it causes, say what would have caught it. This tests reading, which is the skill that actually became scarce, and it is very hard to fake because the follow-up questions are live.

The broken system. A running service, a failing behaviour, real logs. Watch them form a hypothesis, test it, and discard it. The best signal in the whole interview is a candidate saying “that was wrong, so it must be...”: hypothesis abandonment under evidence is rare and highly predictive.

Extend an unfamiliar codebase. Twenty thousand lines they have never seen, one small feature, two hours, questions allowed. This measures orientation speed, which is the thing that determines how fast they will be useful.

Design with pushback. Not “design a URL shortener”. Give them a real constraint mid-conversation (“the database is now in a different region”, “legal says this data cannot leave the country”) and see whether the design bends or shatters. Also see whether they will say “I do not know” instead of generating.

Ask about a decision they regret

One question, and it is worth more than most of the loop: describe a technical decision you made that turned out badly, what the symptom was, and what you do differently now.

It is hard to fabricate at depth because the follow-ups need specifics: the actual failure mode, the actual month it surfaced, what the rollback cost. And it measures the thing you most want, which is whether experience turned into calibration or just into years.

Do not do these

Do not ban tools and hope. For remote interviews you cannot enforce it, so you filter for willingness to break a rule. For in-person ones you are testing a working condition that will never recur.

Do not extend the take-home. The fix for “everyone finishes it” is not eight hours. It is a different exercise. Anything over two hours is a tax on people with less free time, and you will lose good candidates who correctly decline.

Do not screen with AI-detection tools on submissions. They do not work, they produce false positives that fall unevenly on non-native English speakers, and a wrongly rejected candidate has no way to appeal.

The limitation, stated honestly

None of these formats is a validated selection instrument. The research on structured interviews generally (same questions, same order, scored against a rubric agreed in advance) supports structure over unstructured conversation, and that finding applies here: write the rubric first. But no study shows that a planted-bug review predicts on-the-job performance, because nobody has run one.

So treat these as reasoned designs aimed at the skill you believe is now scarce, not as measurements. Write down what you are trying to detect before the loop starts, score against it afterwards, and check a year later whether the people you rated highly are the people doing well. That last step is the only thing that turns an interview process into evidence, and almost nobody does it.