What the one-click button is actually doing

Not an edge detector

Select Subject, Remove Background, the long-press-and-lift on your phone: all of them run a neural network that was trained on hundreds of thousands of images paired with human-drawn masks. The model has learned what a person, a car, a dog, a bottle look like as shapes. It is not measuring contrast at a boundary; it is recognising an object and predicting which pixels belong to it.

That difference explains both why it works where the wand fails and how it fails.

The common architectures, in the free world:

• U²-Net: the model behind rembg and a great many web background removers. Small, fast, runs on a CPU in a second or two, trained for salient object detection: it finds the one thing the picture is about.
• Segment Anything (SAM), from Meta, released free with open weights in 2023. Different job: you give it a click or a box and it returns a mask for whatever is there. It will segment anything, including things nobody labelled, which is why it is the basis of many newer tools.
• MODNet and BiRefNet: portrait and high-resolution matting models, better on hair than U²-Net.

Running them free, locally

pip install rembg
rembg i input.jpg output.png          # one image
rembg p ./in ./out                     # a whole folder
rembg i -m isnet-general-use in.jpg out.png   # a different model

That runs on your own machine, on a CPU, with no upload and no per-image cost. On a five-year-old laptop it takes two to five seconds per image. There is a rembg[gpu] variant if you have one.

There are also GIMP plugins that wrap the same models, and a growing number of offline Android apps.

This matters for a specific reason beyond money. Using a web background remover means uploading a client’s photograph to somebody else’s server, under terms of service you have not read, which may permit training on it. For a product shot of a jar of honey nobody cares. For an unreleased product, a medical image, a photograph of a child, or anything under a confidentiality agreement, it is a decision you should be making deliberately rather than by habit.

Where models fail, and why each failure happens

Two objects, one picked. Salient-object models return the subject. Two people standing apart and it may take one. SAM-style click-based tools do not have this problem, because you tell it what you mean.

Holes get filled. A mask predicted from learned object shapes tends to produce closed, plausible silhouettes. The gap between an arm and a torso, the space inside a handle, the mesh of a bicycle wheel: these are frequently filled in, because most training examples had solid bodies there. Always check the enclosed gaps.

Thin structures disappear. Guitar strings, spectacle arms, antennae, a dog’s whiskers. These occupy a few pixels and contribute almost nothing to the loss function the model was trained against, so the model is not penalised much for missing them.

Reflective and transparent objects are guessed. A wine glass gets a solid silhouette rather than a transparency map, because the model outputs coverage and the correct answer is not coverage.

Unusual subjects. The models are excellent on people, common animals, cars and everyday products, because that is what the training sets contain. Machinery, textiles, food in a bowl, agricultural equipment and anything culturally specific are markedly weaker. If your work is in a domain the training data under-represents, expect to finish by hand more often.

The output resolution ceiling. Many models run internally at 320×320 or 1024×1024 and upscale the mask. On a 45-megapixel file the returned edge is therefore soft and approximate regardless of how sharp the original is. Tools that handle this well run a second high-resolution refinement pass; tools that do not give you a slightly mushy outline that no amount of sharpening fixes.

The workflow that uses them properly

Do not treat the output as the answer. Treat it as ninety seconds saved.

1. Run the model. Get the mask.
2. Look at the mask alone, Alt/Option-clicking the thumbnail. Every failure above is obvious in black and white and invisible against the picture.
3. Fix the enclosed gaps with a hard black brush.
4. Fix the thin structures by hand, or accept losing them if they do not matter.
5. Check the edge at 100 per cent for the resolution ceiling; if it is soft, refine it against the original with a channel-based mask or a path along the hard sections.
6. Decontaminate the colour, which is the next lesson.

The five minutes of checking is what separates a listing photograph that looks professional from one that looks automatic, and it is the part that is skipped.

The part that is unsettled

These models were trained on large image collections scraped from the web. Whether that training was lawful, and whether the models’ outputs carry any obligation to the people whose images were used, is being argued in several jurisdictions at once and the answers differ by country. Some datasets were assembled with consent and licensing; many were not, and some have been withdrawn after audits found material that should never have been in them.

There is no settled position to give you here, and anyone who tells you there is has picked a side in an argument that courts have not finished. What is within your control is narrower and worth doing: prefer models with a stated licence and a documented training set, keep client images off third-party servers where confidentiality applies, and do not represent a model-generated result as a measurement of anything.