What a codec throws away

Compression is a set of decisions about what you will not notice

A raw 1080p frame is 1920 × 1080 × 3 bytes, about 6MB. At 25 frames a second that is 150MB every second. A streamed 1080p video runs at about 0.6MB a second. The compressor is throwing away roughly 99.6 per cent of the data and the result still looks acceptable.

It manages that by exploiting specific, known limits of human vision. Knowing which ones tells you exactly when a codec will fail, and the failures are predictable rather than mysterious.

Colour resolution goes first

Your retina has around 6 million cones and 120 million rods, but in daylight and in front of a lit screen the rods are saturated and the cones do all the work, brightness included. What codecs exploit is how the visual system processes what the cones report. Spatial acuity for colour is far below that for luminance: you can see a fine black-and-white grating that becomes a flat grey when the same pattern is made of two equally bright colours.

Codecs exploit this with chroma subsampling, written as three numbers.

• 4:4:4: full colour resolution. One colour sample per pixel.
• 4:2:2: colour sampled at half the horizontal resolution. Half the colour data.
• 4:2:0: colour sampled at half horizontally and half vertically. A quarter of the colour data.

Almost everything you watch is 4:2:0. Every phone recording, every YouTube stream, every Blu-ray.

The consequences are concrete. A single-pixel red line on a grey background has no colour samples of its own; its colour is averaged with its neighbours and it comes out muddy. Fine coloured text on a contrasting background gets a smeared fringe. And, from the compositing module: a green-screen edge in 4:2:0 has colour information at half resolution in both directions, so the keyer is working on data that is already blurred by two pixels before it starts.

This is the reason a 4:2:2 camera setting exists and matters for green screen, graphics and heavy grading, and does not matter for a talking head that will be watched once.

Blocks and frequencies

The image is cut into blocks (8 × 8 transforms in older codecs such as JPEG and MPEG-2; coding blocks up to 64 × 64 in HEVC and 128 × 128 in AV1, with transforms up to 32 × 32 and 64 × 64 respectively) and each block is transformed from pixels into frequency coefficients, usually by a discrete cosine transform or a close relative. The first coefficient is the block’s average brightness; later ones describe progressively finer detail within it.

The compression step is quantisation: divide each coefficient by a number and round. The numbers are larger for high-frequency coefficients, because fine detail is less noticeable than broad tone. Rounding sends many high-frequency coefficients to zero, and long runs of zeros compress to almost nothing.

Everything visible about compression artefacts follows from this:

• Blocking: when quantisation is coarse, neighbouring blocks end up with different average values and the grid becomes visible. Codecs apply a deblocking filter to smooth it, which is why heavily compressed modern video looks smeared rather than blocky.
• Ringing and mosquito noise: ripples around sharp edges, because a sharp edge needs high-frequency coefficients that have been quantised away, and the reconstruction overshoots.
• Banding: smooth gradients lose the low-amplitude detail that made them smooth.
• Detail loss in texture: grass, water, confetti, rain, film grain. These are high-frequency, unpredictable, and enormously expensive to encode. The codec’s response is to flatten them.

That last one is the practical planning point: content, not resolution, drives the bitrate you need. A static interview at 1080p needs a fraction of what a 1080p shot of rain on water needs, and no encoder setting changes that.

Why noise is expensive

A codec works by predicting and then coding the difference between prediction and reality. Noise is by definition unpredictable, so every noisy pixel costs bits. A grainy or high-ISO source can easily need twice the bitrate of a clean one for the same visible quality, and the bits are being spent on reproducing noise.

This is why a light denoise before encoding often improves apparent quality at a fixed bitrate: you are moving the bits from the noise to the picture. It is also why the grain you added in compositing partly disappears after upload, and why streaming platforms are interested in synthesising film grain at the decoder: AV1 has a grain synthesis tool for exactly this reason, transmitting grain parameters rather than grain.

The generational problem

Every encode quantises. Encode, decode, edit, encode again, and the second encoder is now trying to reproduce the first one’s artefacts, including its block edges, which are high-frequency detail it did not have to encode the first time.

Three or four generations of a lossy codec is visibly worse than one. This is the entire reason intermediate codecs exist, covered in the next lesson, and the reason you never edit from a downloaded delivery file if a master exists.

The free path

ffmpeg is the reference implementation of most of this and lets you see it directly. To watch the subsampling loss on your own footage:

ffmpeg -i source.mov -vf format=yuv420p,format=yuv444p chroma.mp4

That round-trip through 4:2:0 and back shows you exactly what the colour subsampling costs, with no encoding involved. x264, x265, libvpx and libaom are all open-source encoders and their documentation is unusually honest about what each setting does.

The limitation

None of this predicts perceived quality reliably. Codec decisions are tuned against average viewers on average content, and your content may not be average: anime, screen recordings, medical imagery and scientific data all break assumptions built for filmed footage. PSNR and SSIM, the standard automatic quality metrics, correlate only loosely with what people actually notice, which is why platforms increasingly use models like VMAF and why even those are contested.

The honest method is still to encode a representative thirty seconds at two or three settings and look at them, at full size, on the kind of screen your audience uses.