Debugging with an AI assistant, and what it gets wrong

What to paste

An assistant cannot see your files, run your code, or know which versions you have installed. It has only what you paste. So paste, in one message:

1. The whole traceback. Not the last line. The chain of calls is usually where the answer is, and the bottom line alone is often unanswerable.
2. The function that failed, complete, not a summary of it.
3. What you expected to happen, in one sentence.
4. The versions, when a library is involved: python3 --version and pip show pandas | head -2.

Missing any of these produces a confident answer to a different question. The most common failure in practice is pasting only the last line, getting a general explanation of KeyError, and being no further forward.

Ask for the cause, not just the fix

A patch that makes the error go away can be worse than the error. "Explain why this happens before suggesting a change" is a better opening than "fix this", and it costs nothing.

The reason is specific: the most common repair an assistant offers for an unclear failure is a try/except around it, which is exactly the pattern that hides bugs. If you asked why, you find out that a column arrived as a string, and you fix that instead.

The failure mode to expect: invented APIs

Language models produce plausible code, and plausible is not the same as real. They will call methods that do not exist, pass arguments a function does not take, and use a signature from a version three releases old. This is not rare and it is not a sign the tool is broken; it is what a next-token predictor does when the true answer is a detail it has seen inconsistently.

Two checks, both free and both quick:

import pandas as pd
print([m for m in dir(pd.DataFrame) if "explode" in m])
help(pd.DataFrame.merge)

dir() tells you whether the method exists in your installed version. help() tells you its real signature. If a suggested method is not in dir(), no amount of arguing will make it appear.

The other half of the check is the release date. A model's knowledge stops at some point, and libraries change after that. Anything to do with a recently changed API — and AI libraries change constantly — is worth confirming against the current documentation.

Where it is genuinely fast

• Unfamiliar error messages, especially from a library you have not used. Explaining what SettingWithCopyWarning means saves half an hour of searching.
• Boilerplate you know how to check: an argparse block, a regular expression, a plotting call.
• Reading code somebody else wrote, in a language or style you do not know well.
• Generating test cases, particularly the edge cases you would not have thought of. You can verify each one against the function's actual behaviour.
• Turning a vague symptom into candidate causes you can then test yourself.

Where it misleads

• Your own domain logic. It does not know that in your business a negative quantity means a return. It will produce something reasonable and wrong.
• Version-specific behaviour, as above.
• Anything depending on data it cannot see. Ask why a merge produced 1.4 million rows from two 40,000-row tables and you get the general answer — duplicate keys — which is right, but only your data says which key.
• Performance claims. Suggestions about what is faster are frequently folklore. Measure with timeit; the answer takes ten seconds and settles it.

What must never be pasted

• API keys, tokens, passwords, connection strings. If a key has ever been in a paste, rotate it.
• Customer data, personal details, health or financial records.
• Proprietary formulas, contracts or unpublished material belonging to an employer or client.

Before pasting a traceback, read it. Tracebacks often include argument values, and argument values are often exactly the data that should not leave the building. Replace them with placeholders.

Build the minimal reproduction anyway

Cutting the failure down to fifteen lines that still fail is the most useful thing you can do, whether or not you then ask anybody. About half the time the bug becomes obvious during the cutting-down, and the answer arrives before the question is finished. When it does not, you now have something small enough to paste in full, with no secrets in it.

The assistant is a fast, well-read colleague who has never seen your codebase and will not say when it is unsure. Treat every answer as a hypothesis to check, and the tool is excellent. Treat it as an authority and it will cost you a day.