{"course":{"slug":"motion-and-audio","title":"Motion, Sound and Music","url":"https://addaly.com/learn/motion-and-audio","level":"intermediate","track":"create"},"lesson":{"slug":"what-a-codec-discards","title":"What a codec throws away","url":"https://addaly.com/learn/motion-and-audio/what-a-codec-discards","position":37,"of":72,"minutes":11,"answer":"Compression exploits specific limits of vision (colour resolution first, then fine detail within blocks) so the bitrate a clip needs is set by how unpredictable its content is rather than by its resolution.","keyIdea":"Compression exploits specific limits of vision (colour resolution first, then fine detail within blocks) so the bitrate a clip needs is set by how unpredictable its content is rather than by its resolution, and noise is expensive precisely because it cannot be predicted."},"body":{"format":"markdown","text":"## Compression is a set of decisions about what you will not notice\n\nA raw 1080p frame is 1920 × 1080 × 3 bytes, about 6MB. At 25 frames a second that is 150MB every second. A streamed 1080p video runs at about 0.6MB a second. The compressor is throwing away roughly 99.6 per cent of the data and the result still looks acceptable.\n\nIt manages that by exploiting specific, known limits of human vision. Knowing which ones tells you exactly when a codec will fail, and the failures are predictable rather than mysterious.\n\n## Colour resolution goes first\n\nYour retina has around 6 million cones and 120 million rods, but in daylight and in front of a lit screen the rods are saturated and the cones do all the work, brightness included. What codecs exploit is how the visual system processes what the cones report. Spatial acuity for colour is far below that for luminance: you can see a fine black-and-white grating that becomes a flat grey when the same pattern is made of two equally bright colours.\n\nCodecs exploit this with **chroma subsampling**, written as three numbers.\n\n- **4:4:4**: full colour resolution. One colour sample per pixel.\n- **4:2:2**: colour sampled at half the horizontal resolution. Half the colour data.\n- **4:2:0**: colour sampled at half horizontally *and* half vertically. A quarter of the colour data.\n\nAlmost everything you watch is 4:2:0. Every phone recording, every YouTube stream, every Blu-ray.\n\n> Figure: How much colour each sampling scheme keeps. Bars (%): 4:4:4, one colour sample per pixel 100; 4:2:2, half horizontally 50; 4:2:0, half in both directions 25. Almost everything you watch is 4:2:0, because the eye's spatial acuity for colour is far below its acuity for brightness. The consequences are specific rather than general: a single-pixel red line goes muddy, fine coloured text picks up a smeared fringe, and a green-screen edge is drawn from colour data already blurred by two pixels before the keyer starts.\n\nThe consequences are concrete. A single-pixel red line on a grey background has no colour samples of its own; its colour is averaged with its neighbours and it comes out muddy. Fine coloured text on a contrasting background gets a smeared fringe. And, from the compositing module: a green-screen edge in 4:2:0 has colour information at half resolution in both directions, so the keyer is working on data that is already blurred by two pixels before it starts.\n\nThis is the reason a 4:2:2 camera setting exists and matters for green screen, graphics and heavy grading, and does not matter for a talking head that will be watched once.\n\n## Blocks and frequencies\n\nThe image is cut into blocks (8 × 8 transforms in older codecs such as JPEG and MPEG-2; coding blocks up to 64 × 64 in HEVC and 128 × 128 in AV1, with transforms up to 32 × 32 and 64 × 64 respectively) and each block is transformed from pixels into frequency coefficients, usually by a discrete cosine transform or a close relative. The first coefficient is the block's average brightness; later ones describe progressively finer detail within it.\n\nThe compression step is **quantisation**: divide each coefficient by a number and round. The numbers are larger for high-frequency coefficients, because fine detail is less noticeable than broad tone. Rounding sends many high-frequency coefficients to zero, and long runs of zeros compress to almost nothing.\n\nEverything visible about compression artefacts follows from this:\n\n- **Blocking**: when quantisation is coarse, neighbouring blocks end up with different average values and the grid becomes visible. Codecs apply a deblocking filter to smooth it, which is why heavily compressed modern video looks smeared rather than blocky.\n- **Ringing** and **mosquito noise**: ripples around sharp edges, because a sharp edge needs high-frequency coefficients that have been quantised away, and the reconstruction overshoots.\n- **Banding**: smooth gradients lose the low-amplitude detail that made them smooth.\n- **Detail loss in texture**: grass, water, confetti, rain, film grain. These are high-frequency, unpredictable, and enormously expensive to encode. The codec's response is to flatten them.\n\nThat last one is the practical planning point: **content, not resolution, drives the bitrate you need.** A static interview at 1080p needs a fraction of what a 1080p shot of rain on water needs, and no encoder setting changes that.\n\n## Why noise is expensive\n\nA codec works by predicting and then coding the difference between prediction and reality. Noise is by definition unpredictable, so every noisy pixel costs bits. A grainy or high-ISO source can easily need twice the bitrate of a clean one for the same visible quality, and the bits are being spent on reproducing noise.\n\nThis is why a light denoise before encoding often improves apparent quality at a fixed bitrate: you are moving the bits from the noise to the picture. It is also why the grain you added in compositing partly disappears after upload, and why streaming platforms are interested in synthesising film grain at the decoder: AV1 has a grain synthesis tool for exactly this reason, transmitting grain parameters rather than grain.\n\n## The generational problem\n\nEvery encode quantises. Encode, decode, edit, encode again, and the second encoder is now trying to reproduce the first one's artefacts, including its block edges, which are high-frequency detail it did not have to encode the first time.\n\nThree or four generations of a lossy codec is visibly worse than one. This is the entire reason intermediate codecs exist, covered in the next lesson, and the reason you never edit from a downloaded delivery file if a master exists.\n\n## The free path\n\n**ffmpeg** is the reference implementation of most of this and lets you see it directly. To watch the subsampling loss on your own footage:\n\n```bash\nffmpeg -i source.mov -vf format=yuv420p,format=yuv444p chroma.mp4\n```\n\nThat round-trip through 4:2:0 and back shows you exactly what the colour subsampling costs, with no encoding involved. **x264**, **x265**, **libvpx** and **libaom** are all open-source encoders and their documentation is unusually honest about what each setting does.\n\n## The limitation\n\nNone of this predicts perceived quality reliably. Codec decisions are tuned against average viewers on average content, and your content may not be average: anime, screen recordings, medical imagery and scientific data all break assumptions built for filmed footage. PSNR and SSIM, the standard automatic quality metrics, correlate only loosely with what people actually notice, which is why platforms increasingly use models like VMAF and why even those are contested.\n\nThe honest method is still to encode a representative thirty seconds at two or three settings and look at them, at full size, on the kind of screen your audience uses."},"check":{"question":"Two 1080p clips are encoded at the same CRF. A static interview comes out at 2 Mbit/s; a handheld shot of rain on water comes out at 11 Mbit/s. Why does constant quality produce such different rates?","options":[{"text":"The handheld shot has more camera movement, and motion vectors are the largest component of an encoded frame at high bitrates.","correct":false,"tempting":"Motion vectors are compact; the cost is in the residual detail that prediction fails to capture."},{"text":"The rain clip is likely to have been shot at a higher ISO, and the encoder allocates bitrate in proportion to the sensor gain recorded in the file's metadata.","correct":false,"tempting":"Invents a metadata-driven allocation; encoders respond to the picture, not to recorded ISO."},{"text":"Rain and moving water are high-frequency and unpredictable, so prediction fails and the residual detail that must be coded is large.","correct":true},{"text":"The interview is mostly skin tones, which fall in the part of the colour space where chroma subsampling discards the most information, so far fewer chroma coefficients survive quantisation.","correct":false,"tempting":"Subsampling is applied uniformly across the frame regardless of hue, so it cannot explain a five-fold difference between clips."}]},"prev":{"slug":"matching-the-plate","title":"Black level, grain, focus, and the order to do them in","url":"https://addaly.com/learn/motion-and-audio/matching-the-plate"},"next":{"slug":"intra-vs-inter-frames","title":"Frames that stand alone, and frames that do not","url":"https://addaly.com/learn/motion-and-audio/intra-vs-inter-frames"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'What a codec throws away', in Motion, Sound and Music. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/motion-and-audio/what-a-codec-discards (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"What a codec throws away","courseTitle":"Motion, Sound and Music","module":{"index":5,"slug":"codecs-colour-and-delivery","title":"Codecs, colour and delivery"},"answer":null,"keyIdea":"Compression exploits specific limits of vision (colour resolution first, then fine detail within blocks) so the bitrate a clip needs is set by how unpredictable its content is rather than by its resolution, and noise is expensive precisely because it cannot be predicted.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure 5.1","figure":{"type":"bars","unit":"%","title":"How much colour each sampling scheme keeps","series":[{"label":"4:4:4, one colour sample per pixel","value":100},{"label":"4:2:2, half horizontally","value":50},{"label":"4:2:0, half in both directions","value":25,"highlight":true}],"caption":"Almost everything you watch is 4:2:0, because the eye’s spatial acuity for colour is far below its acuity for brightness. The consequences are specific rather than general: a single-pixel red line goes muddy, fine coloured text picks up a smeared fringe, and a green-screen edge is drawn from colour data already blurred by two pixels before the keyer starts."},"alt":"Figure: How much colour each sampling scheme keeps. Bars (%): 4:4:4, one colour sample per pixel 100; 4:2:2, half horizontally 50; 4:2:0, half in both directions 25. Almost everything you watch is 4:2:0, because the eye’s spatial acuity for colour is far below its acuity for brightness. The consequences are specific rather than general: a single-pixel red line goes muddy, fine coloured text picks up a smeared fringe, and a green-screen edge is drawn from colour data already blurred by two pixels before the keyer starts."}},"blocks":[{"t":"h2","id":"compression-is-a-set-of-decisions-about-what-you-will-not-no","runs":[{"text":"Compression is a set of decisions about what you will not notice"}]},{"t":"p","runs":[{"text":"A raw 1080p frame is 1920 × 1080 × 3 bytes, about 6MB. At 25 frames a second that is 150MB every second. A streamed 1080p video runs at about 0.6MB a second. The compressor is throwing away roughly 99.6 per cent of the data and the result still looks acceptable."}]},{"t":"p","runs":[{"text":"It manages that by exploiting specific, known limits of human vision. Knowing which ones tells you exactly when a codec will fail, and the failures are predictable rather than mysterious."}]},{"t":"h2","id":"colour-resolution-goes-first","runs":[{"text":"Colour resolution goes first"}]},{"t":"p","runs":[{"text":"Your retina has around 6 million cones and 120 million rods, but in daylight and in front of a lit screen the rods are saturated and the cones do all the work, brightness included. What codecs exploit is how the visual system processes what the cones report. Spatial acuity for colour is far below that for luminance: you can see a fine black-and-white grating that becomes a flat grey when the same pattern is made of two equally bright colours."}]},{"t":"p","runs":[{"text":"Codecs exploit this with "},{"text":"chroma subsampling","b":true},{"text":", written as three numbers."}]},{"t":"ul","items":[[{"text":"4:4:4","b":true},{"text":": full colour resolution. One colour sample per pixel."}],[{"text":"4:2:2","b":true},{"text":": colour sampled at half the horizontal resolution. Half the colour data."}],[{"text":"4:2:0","b":true},{"text":": colour sampled at half horizontally "},{"text":"and","i":true},{"text":" half vertically. A quarter of the colour data."}]]},{"t":"p","runs":[{"text":"Almost everything you watch is 4:2:0. Every phone recording, every YouTube stream, every Blu-ray."}]},{"t":"p","runs":[{"text":"The consequences are concrete. A single-pixel red line on a grey background has no colour samples of its own; its colour is averaged with its neighbours and it comes out muddy. Fine coloured text on a contrasting background gets a smeared fringe. And, from the compositing module: a green-screen edge in 4:2:0 has colour information at half resolution in both directions, so the keyer is working on data that is already blurred by two pixels before it starts."}]},{"t":"p","runs":[{"text":"This is the reason a 4:2:2 camera setting exists and matters for green screen, graphics and heavy grading, and does not matter for a talking head that will be watched once."}]},{"t":"h2","id":"blocks-and-frequencies","runs":[{"text":"Blocks and frequencies"}]},{"t":"p","runs":[{"text":"The image is cut into blocks (8 × 8 transforms in older codecs such as JPEG and MPEG-2; coding blocks up to 64 × 64 in HEVC and 128 × 128 in AV1, with transforms up to 32 × 32 and 64 × 64 respectively) and each block is transformed from pixels into frequency coefficients, usually by a discrete cosine transform or a close relative. The first coefficient is the block’s average brightness; later ones describe progressively finer detail within it."}]},{"t":"p","runs":[{"text":"The compression step is "},{"text":"quantisation","b":true},{"text":": divide each coefficient by a number and round. The numbers are larger for high-frequency coefficients, because fine detail is less noticeable than broad tone. Rounding sends many high-frequency coefficients to zero, and long runs of zeros compress to almost nothing."}]},{"t":"p","runs":[{"text":"Everything visible about compression artefacts follows from this:"}]},{"t":"ul","items":[[{"text":"Blocking","b":true},{"text":": when quantisation is coarse, neighbouring blocks end up with different average values and the grid becomes visible. Codecs apply a deblocking filter to smooth it, which is why heavily compressed modern video looks smeared rather than blocky."}],[{"text":"Ringing","b":true},{"text":" and "},{"text":"mosquito noise","b":true},{"text":": ripples around sharp edges, because a sharp edge needs high-frequency coefficients that have been quantised away, and the reconstruction overshoots."}],[{"text":"Banding","b":true},{"text":": smooth gradients lose the low-amplitude detail that made them smooth."}],[{"text":"Detail loss in texture","b":true},{"text":": grass, water, confetti, rain, film grain. These are high-frequency, unpredictable, and enormously expensive to encode. The codec’s response is to flatten them."}]]},{"t":"p","runs":[{"text":"That last one is the practical planning point: "},{"text":"content, not resolution, drives the bitrate you need.","b":true},{"text":" A static interview at 1080p needs a fraction of what a 1080p shot of rain on water needs, and no encoder setting changes that."}]},{"t":"h2","id":"why-noise-is-expensive","runs":[{"text":"Why noise is expensive"}]},{"t":"p","runs":[{"text":"A codec works by predicting and then coding the difference between prediction and reality. Noise is by definition unpredictable, so every noisy pixel costs bits. A grainy or high-ISO source can easily need twice the bitrate of a clean one for the same visible quality, and the bits are being spent on reproducing noise."}]},{"t":"p","runs":[{"text":"This is why a light denoise before encoding often improves apparent quality at a fixed bitrate: you are moving the bits from the noise to the picture. It is also why the grain you added in compositing partly disappears after upload, and why streaming platforms are interested in synthesising film grain at the decoder: AV1 has a grain synthesis tool for exactly this reason, transmitting grain parameters rather than grain."}]},{"t":"h2","id":"the-generational-problem","runs":[{"text":"The generational problem"}]},{"t":"p","runs":[{"text":"Every encode quantises. Encode, decode, edit, encode again, and the second encoder is now trying to reproduce the first one’s artefacts, including its block edges, which are high-frequency detail it did not have to encode the first time."}]},{"t":"p","runs":[{"text":"Three or four generations of a lossy codec is visibly worse than one. This is the entire reason intermediate codecs exist, covered in the next lesson, and the reason you never edit from a downloaded delivery file if a master exists."}]},{"t":"h2","id":"the-free-path","runs":[{"text":"The free path"}],"role":"free"},{"t":"p","runs":[{"text":"ffmpeg","b":true},{"text":" is the reference implementation of most of this and lets you see it directly. To watch the subsampling loss on your own footage:"}]},{"t":"code","lang":"bash","label":"Bash","text":"ffmpeg -i source.mov -vf format=yuv420p,format=yuv444p chroma.mp4"},{"t":"p","runs":[{"text":"That round-trip through 4:2:0 and back shows you exactly what the colour subsampling costs, with no encoding involved. "},{"text":"x264","b":true},{"text":", "},{"text":"x265","b":true},{"text":", "},{"text":"libvpx","b":true},{"text":" and "},{"text":"libaom","b":true},{"text":" are all open-source encoders and their documentation is unusually honest about what each setting does."}]},{"t":"h2","id":"the-limitation","runs":[{"text":"The limitation"}],"role":"limit"},{"t":"p","runs":[{"text":"None of this predicts perceived quality reliably. Codec decisions are tuned against average viewers on average content, and your content may not be average: anime, screen recordings, medical imagery and scientific data all break assumptions built for filmed footage. PSNR and SSIM, the standard automatic quality metrics, correlate only loosely with what people actually notice, which is why platforms increasingly use models like VMAF and why even those are contested."}]},{"t":"p","runs":[{"text":"The honest method is still to encode a representative thirty seconds at two or three settings and look at them, at full size, on the kind of screen your audience uses."}]}],"check":{"question":[{"text":"Two 1080p clips are encoded at the same CRF. A static interview comes out at 2 Mbit/s; a handheld shot of rain on water comes out at 11 Mbit/s. Why does constant quality produce such different rates?"}],"options":[{"text":[{"text":"The handheld shot has more camera movement, and motion vectors are the largest component of an encoded frame at high bitrates."}],"correct":false,"tempting":[{"text":"Motion vectors are compact; the cost is in the residual detail that prediction fails to capture."}]},{"text":[{"text":"The rain clip is likely to have been shot at a higher ISO, and the encoder allocates bitrate in proportion to the sensor gain recorded in the file’s metadata."}],"correct":false,"tempting":[{"text":"Invents a metadata-driven allocation; encoders respond to the picture, not to recorded ISO."}]},{"text":[{"text":"Rain and moving water are high-frequency and unpredictable, so prediction fails and the residual detail that must be coded is large."}],"correct":true,"tempting":null},{"text":[{"text":"The interview is mostly skin tones, which fall in the part of the colour space where chroma subsampling discards the most information, so far fewer chroma coefficients survive quantisation."}],"correct":false,"tempting":[{"text":"Subsampling is applied uniformly across the frame regardless of hue, so it cannot explain a five-fold difference between clips."}]}],"order":[3,0,2,1],"why":null},"prev":{"slug":"matching-the-plate","title":"Black level, grain, focus, and the order to do them in","minutes":11},"next":{"slug":"intra-vs-inter-frames","title":"Frames that stand alone, and frames that do not","minutes":10}}}