Which upscaler, for which material

Three families, three behaviours

Model-based upscalers are not one thing. Knowing which family you are using explains the result.

GAN-based super-resolution: ESRGAN, Real-ESRGAN, and the many community models built on them. A generator produces a larger image and a discriminator judges whether it looks real; the generator learns to produce convincing texture. The result is sharp and detailed and the detail is invented. These are the workhorses, and Real-ESRGAN in particular was trained on synthetically degraded images so it handles real-world compression and blur rather than only clean downscales.

Transformer-based: SwinIR and successors. Generally more faithful and less inclined to hallucinate texture, slower, and often slightly softer.

Diffusion-based: the newest, most capable of producing convincing detail and the most likely to produce detail that has nothing to do with your image.

And separately, face restoration models (GFPGAN, CodeFormer) which do not upscale generally but regenerate faces from a learned prior of what faces look like. They are frequently bundled with upscalers and they are the component that changes who somebody is.

Picking by material

• Skin, fabric, foliage, brickwork, general photographic texture. Real-ESRGAN at 2× is excellent. This is what it was built for.
• Anime, illustration, flat graphics. Use a model trained on that material: realesrgan-x4plus-anime and equivalents. A photographic model on line art produces mushy edges.
• Text, logos, signage. Do not. The letterforms are regenerated and come back the wrong shapes. Rebuild the text as vector.
• Faces. Use face restoration only when a plausible face is acceptable. On a family photograph most people are delighted. On anything identifying, it produces a different person, subtly.
• Heavily compressed sources. Real-ESRGAN’s degradation-aware training handles these better than most. Expect the JPEG blocks to be interpreted as content.
• Evidence of any kind. No.

The order that matters

Denoise before upscaling, not after. An upscaler trained on clean-ish inputs will interpret noise as texture and enlarge it into structured artefacts: wormy patterns, clumps, pseudo-detail. Remove the noise first and the model has signal to work from.

Do not sharpen before upscaling. The halos get enlarged too. Sharpen at the end, at the output size, exactly as in the sharpening lesson.

Crop first. Upscaling a 40-megapixel file to use a 600-pixel corner of it is minutes of computation thrown away.

2× twice is not the same as 4× once. Running a 2× model twice compounds its invented detail on top of its own previous inventions, and the second pass has no real information to work with. Use the 4× model if you need 4×.

Tiling, and the seams it causes

A large image does not fit in a graphics card’s memory, so upscalers process it in tiles and join them. Where the tiles meet, the model had different context on each side, and a seam can appear: usually a faint line of different texture or brightness.

The fix is a tile overlap setting, usually called padding or tile pad. Raising it from the default 10 pixels to 32 or 64 removes most seams at a modest cost in time. In Upscayl and in the command-line realesrgan-ncnn-vulkan this is exposed directly.

If you see a regular grid of faint lines in an upscaled image, that is tiling, and it is fixable.

Running it free

Upscayl is a free, open-source desktop application wrapping Real-ESRGAN and several other models, with a plain interface, batch processing and no upload. It runs on Windows, macOS and Linux, uses the GPU through Vulkan where one exists, and works on integrated graphics slowly.

realesrgan-ncnn-vulkan -i input.jpg -o output.png -n realesrgan-x4plus -s 4 -t 256

That is the command-line version, the same engine, scriptable for a batch.

chaiNNer is a free node-based tool for building upscaling pipelines and running community models. The OpenModelDB collection hosts hundreds of community-trained models for specific material (old film, scanned prints, compressed video, particular game textures) and one of them is often much better on your specific material than a general model.

On the paid side, Topaz Gigapixel and Adobe’s Super Resolution inside Camera Raw are convenient and, on ordinary photographic material, not dramatically better than the free options.

Checking the output

Compare at 100 per cent against a bicubic enlargement of the same file. Look specifically at:

• Faces: is this the same person?
• Text and numbers: are the characters the same characters?
• Repeated fine texture: fabric weave, distant foliage, a wire fence. Is the pattern the pattern that was there, or a convincing substitute?
• Flat areas: has the model invented texture in what should be smooth sky or wall?
• Tile boundaries: any regular grid of faint lines?

If the upscale fails any of those and the picture depends on that region, use the bicubic version, which is honest about being soft.

The limit, restated because it keeps mattering

An upscaler predicts a plausible high-resolution image consistent with the low-resolution evidence. Where the evidence is strong (a well-exposed, in-focus photograph enlarged 2×) the plausible answer is usually close to the true one and the result is useful.

Where the evidence is weak (a 200-pixel face enlarged 8×) the model is mostly drawing on its training data, and what comes out tells you about the training set rather than about your photograph. There is no warning at the boundary between those two cases, and the output looks equally confident in both.

The practical rule: 2× is usually recovery-like, 4× is interpretation, 8× is illustration. Say which one you did.