{"course":{"slug":"cloud-and-shipping","title":"Shipping It","url":"https://addaly.com/learn/cloud-and-shipping","level":"intermediate","track":"build"},"lesson":{"slug":"bots-scrapers-and-floods","title":"Bots, scrapers and the flood you cannot absorb","url":"https://addaly.com/learn/cloud-and-shipping/bots-scrapers-and-floods","position":62,"of":74,"minutes":9,"answer":"Volumetric floods are handled upstream by a CDN you did not write, but application-layer abuse is yours: cache the public pages, put quotas and challenges in front of the expensive ones.","keyIdea":"Volumetric floods are handled upstream by a CDN you did not write, but application-layer abuse is yours: cache the public pages, put quotas and challenges in front of the expensive ones, limit by account rather than address, and log every block because the people you wrongly refuse cannot tell you."},"body":{"format":"markdown","text":"## Most of your traffic is not people\n\nPublish anything and automated traffic arrives. Measurements of the web as a whole put the automated share somewhere near or above half, and for a small site the proportion is often higher, because you have no audience yet and the crawlers do not care.\n\nIt is worth separating the kinds, because the response differs:\n\n- **Search crawlers.** Wanted. Obey `robots.txt`, identify themselves, crawl politely.\n- **Dataset and AI crawlers.** A newer, heavier category. Some obey `robots.txt`, some do not, and several have been observed requesting the same pages repeatedly. Whether you want them is a decision, not a given.\n- **Scrapers of your content.** They want your data, and they will rotate addresses and pretend to be browsers.\n- **Credential stuffers and signup abuse.** Covered by the earlier rate-limit and session lessons.\n- **Volumetric floods.** Someone trying to make you unavailable, or a badly written client doing it accidentally.\n\n## The three layers, and which is yours to fix\n\n**Layer 3 and 4: packet floods.** Gigabits of traffic aimed at your address. You cannot absorb this on an origin server; the pipe fills before your code runs. It is handled upstream, by your provider or a CDN, and every major CDN including free tiers does it automatically. Nothing you write matters at this layer.\n\n**Layer 7: application floods.** Requests that look legitimate but hit your most expensive endpoint. Search, report generation, a model call. This is where a small site actually falls over, and this layer is yours: cache aggressively, rate-limit per account and per endpoint, and make sure your most expensive path is not the most exposed one.\n\n**Application logic abuse.** Signup floods, coupon farming, comment spam, free-tier exhaustion. Not a volume problem: a rules problem. Fixed with quotas, verification tiers and cost controls, which is why the identity ladder in this project's own design exists.\n\n> Figure: Three layers of unwanted traffic, and whose job each is. Layers, top to bottom: Application logic abuse (signup floods, coupon farming, free-tier exhaustion: a rules problem, fixed with quotas and identity tiers); Layer 7: application floods (legitimate-looking requests to search, reports or a model call: yours to fix with caching and per-account limits); Layer 3 and 4: packet floods (gigabits at your address, absorbed by the provider or CDN before your code runs). The pipe-filling floods are handled upstream by a CDN you did not write. The layer that actually takes a small site down is the one that hits your most expensive endpoint, and that layer is yours.\n\n## What actually works, ordered by effort\n\n1. **Put a CDN in front.** The single biggest step, free at the entry tier from several providers, and it removes both the volumetric layer and most of the cacheable load.\n2. **Cache the expensive public pages.** A scraper hitting a cached page costs you nothing. A scraper hitting an uncached search endpoint costs you a database query each time.\n3. **Rate-limit by account and by endpoint, not only by IP.** Addresses are cheap and shared; a single mobile network address may carry thousands of legitimate users, which is exactly why blocking by IP produces collateral damage.\n4. **Make the expensive thing require an account.** Anonymous access to a cheap, cached page; anything that costs real money behind a login and a quota.\n5. **Challenge rather than block.** A proof-of-work or interactive challenge (Cloudflare Turnstile, hCaptcha, mCaptcha) costs a real user a moment and a bulk scraper a great deal. It is also the honest place to say that these degrade accessibility for some users, so put them in front of abuse-prone actions, not in front of reading.\n\n## robots.txt is a request, not a fence\n\n`robots.txt` and `noindex` are conventions honoured by well-behaved crawlers. They stop nothing that does not want to be stopped, and they have no legal force in themselves. If content must not be taken, the control is authentication, not a text file.\n\nThat said, the file is still worth writing carefully: name the paths you do not want crawled, and add a `Crawl-delay` or explicit rules for the heavy agents. Several CDNs now offer a one-click block for known AI crawlers. Whether to use it is a product decision: a site that wants to be found may want to be in those datasets, and a site whose content is its product may not.\n\n## Blocking, and its costs\n\nEvery blocking mechanism has a false-positive rate, and the people it blocks cannot tell you, because they are blocked. Three habits keep this honest:\n\n- **Log every block** with the reason, and review a sample weekly. If you find real users, your rules are too tight.\n- **Prefer degradation to denial.** Serve a cached copy, slow the response, or require a challenge rather than returning 403.\n- **Give a way back.** A support email on the block page, an appeal route, and an unblock that a human can perform in a minute.\n\nBlocking whole countries, ranges or ISPs is tempting during an incident and is usually a mistake left in place for years. If you do it under pressure, put an expiry on the rule the same day.\n\n## During an actual flood\n\nThe order that works: turn on your CDN's attack mode or equivalent, cache everything cacheable including HTML for the duration, disable the most expensive endpoints behind a flag, raise your rate limits' strictness, and only then start analysing patterns. Diagnosis is much easier from a site that is up.\n\nAnd afterwards, the useful question is not \"who did this\" (you will rarely know), but \"which endpoint let a hundred requests a second hurt us\". That endpoint is your finding, and caching or a quota is your fix."},"check":{"question":"A site adds a strict per-IP rate limit and reports of \"too many requests\" arrive from an entire university and one mobile network. What does this reveal about the control?","options":[{"text":"The limit is set too low and simply needs a higher threshold","correct":false,"tempting":"Raising the number helps those users while weakening the limit for everyone; the flaw is in the unit of measurement, not the value."},{"text":"Many legitimate users share a single public address through NAT, so an address is a poor proxy for a person","correct":true},{"text":"The users are behind a proxy that strips the forwarded header, so all requests appear to come from the proxy","correct":false,"tempting":"A real misconfiguration to check, but the described pattern (whole institutions affected) is the expected behaviour of per-IP limits even when headers are correct."},{"text":"Rate limits should never be applied to anonymous traffic","correct":false,"tempting":"Overcorrects: anonymous traffic is exactly where abuse arrives, and the answer is a better key plus cheaper responses, not no limit."}]},"prev":{"slug":"rotating-a-key-without-downtime","title":"Changing a key while everything is using it","url":"https://addaly.com/learn/cloud-and-shipping/rotating-a-key-without-downtime"},"next":{"slug":"payments-and-what-you-must-not-store","title":"Taking money without holding card numbers","url":"https://addaly.com/learn/cloud-and-shipping/payments-and-what-you-must-not-store"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Bots, scrapers and the flood you cannot absorb', in Shipping It. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/cloud-and-shipping/bots-scrapers-and-floods (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Bots, scrapers and the flood you cannot absorb","courseTitle":"Shipping It","module":{"index":7,"slug":"security-after-the-short-list","title":"Security after the short list"},"answer":null,"keyIdea":"Volumetric floods are handled upstream by a CDN you did not write, but application-layer abuse is yours: cache the public pages, put quotas and challenges in front of the expensive ones, limit by account rather than address, and log every block because the people you wrongly refuse cannot tell you.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"stack","title":"Three layers of unwanted traffic, and whose job each is","layers":[{"label":"Application logic abuse","detail":"signup floods, coupon farming, free-tier exhaustion: a rules problem, fixed with quotas and identity tiers"},{"label":"Layer 7: application floods","detail":"legitimate-looking requests to search, reports or a model call: yours to fix with caching and per-account limits"},{"label":"Layer 3 and 4: packet floods","detail":"gigabits at your address, absorbed by the provider or CDN before your code runs"}],"caption":"The pipe-filling floods are handled upstream by a CDN you did not write. The layer that actually takes a small site down is the one that hits your most expensive endpoint, and that layer is yours."},"alt":"Figure: Three layers of unwanted traffic, and whose job each is. Layers, top to bottom: Application logic abuse (signup floods, coupon farming, free-tier exhaustion: a rules problem, fixed with quotas and identity tiers); Layer 7: application floods (legitimate-looking requests to search, reports or a model call: yours to fix with caching and per-account limits); Layer 3 and 4: packet floods (gigabits at your address, absorbed by the provider or CDN before your code runs). The pipe-filling floods are handled upstream by a CDN you did not write. The layer that actually takes a small site down is the one that hits your most expensive endpoint, and that layer is yours."}},"blocks":[{"t":"h2","id":"most-of-your-traffic-is-not-people","runs":[{"text":"Most of your traffic is not people"}]},{"t":"p","runs":[{"text":"Publish anything and automated traffic arrives. Measurements of the web as a whole put the automated share somewhere near or above half, and for a small site the proportion is often higher, because you have no audience yet and the crawlers do not care."}]},{"t":"p","runs":[{"text":"It is worth separating the kinds, because the response differs:"}]},{"t":"ul","items":[[{"text":"Search crawlers.","b":true},{"text":" Wanted. Obey "},{"text":"robots.txt","code":true},{"text":", identify themselves, crawl politely."}],[{"text":"Dataset and AI crawlers.","b":true},{"text":" A newer, heavier category. Some obey "},{"text":"robots.txt","code":true},{"text":", some do not, and several have been observed requesting the same pages repeatedly. Whether you want them is a decision, not a given."}],[{"text":"Scrapers of your content.","b":true},{"text":" They want your data, and they will rotate addresses and pretend to be browsers."}],[{"text":"Credential stuffers and signup abuse.","b":true},{"text":" Covered by the earlier rate-limit and session lessons."}],[{"text":"Volumetric floods.","b":true},{"text":" Someone trying to make you unavailable, or a badly written client doing it accidentally."}]]},{"t":"h2","id":"the-three-layers-and-which-is-yours-to-fix","runs":[{"text":"The three layers, and which is yours to fix"}]},{"t":"p","runs":[{"text":"Layer 3 and 4: packet floods.","b":true},{"text":" Gigabits of traffic aimed at your address. You cannot absorb this on an origin server; the pipe fills before your code runs. It is handled upstream, by your provider or a CDN, and every major CDN including free tiers does it automatically. Nothing you write matters at this layer."}]},{"t":"p","runs":[{"text":"Layer 7: application floods.","b":true},{"text":" Requests that look legitimate but hit your most expensive endpoint. Search, report generation, a model call. This is where a small site actually falls over, and this layer is yours: cache aggressively, rate-limit per account and per endpoint, and make sure your most expensive path is not the most exposed one."}]},{"t":"p","runs":[{"text":"Application logic abuse.","b":true},{"text":" Signup floods, coupon farming, comment spam, free-tier exhaustion. Not a volume problem: a rules problem. Fixed with quotas, verification tiers and cost controls, which is why the identity ladder in this project’s own design exists."}]},{"t":"h2","id":"what-actually-works-ordered-by-effort","runs":[{"text":"What actually works, ordered by effort"}]},{"t":"ol","start":1,"items":[[{"text":"Put a CDN in front.","b":true},{"text":" The single biggest step, free at the entry tier from several providers, and it removes both the volumetric layer and most of the cacheable load."}],[{"text":"Cache the expensive public pages.","b":true},{"text":" A scraper hitting a cached page costs you nothing. A scraper hitting an uncached search endpoint costs you a database query each time."}],[{"text":"Rate-limit by account and by endpoint, not only by IP.","b":true},{"text":" Addresses are cheap and shared; a single mobile network address may carry thousands of legitimate users, which is exactly why blocking by IP produces collateral damage."}],[{"text":"Make the expensive thing require an account.","b":true},{"text":" Anonymous access to a cheap, cached page; anything that costs real money behind a login and a quota."}],[{"text":"Challenge rather than block.","b":true},{"text":" A proof-of-work or interactive challenge (Cloudflare Turnstile, hCaptcha, mCaptcha) costs a real user a moment and a bulk scraper a great deal. It is also the honest place to say that these degrade accessibility for some users, so put them in front of abuse-prone actions, not in front of reading."}]]},{"t":"h2","id":"robots-txt-is-a-request-not-a-fence","runs":[{"text":"robots.txt is a request, not a fence"}]},{"t":"p","runs":[{"text":"robots.txt","code":true},{"text":" and "},{"text":"noindex","code":true},{"text":" are conventions honoured by well-behaved crawlers. They stop nothing that does not want to be stopped, and they have no legal force in themselves. If content must not be taken, the control is authentication, not a text file."}]},{"t":"p","runs":[{"text":"That said, the file is still worth writing carefully: name the paths you do not want crawled, and add a "},{"text":"Crawl-delay","code":true},{"text":" or explicit rules for the heavy agents. Several CDNs now offer a one-click block for known AI crawlers. Whether to use it is a product decision: a site that wants to be found may want to be in those datasets, and a site whose content is its product may not."}]},{"t":"h2","id":"blocking-and-its-costs","runs":[{"text":"Blocking, and its costs"}]},{"t":"p","runs":[{"text":"Every blocking mechanism has a false-positive rate, and the people it blocks cannot tell you, because they are blocked. Three habits keep this honest:"}]},{"t":"ul","items":[[{"text":"Log every block","b":true},{"text":" with the reason, and review a sample weekly. If you find real users, your rules are too tight."}],[{"text":"Prefer degradation to denial.","b":true},{"text":" Serve a cached copy, slow the response, or require a challenge rather than returning 403."}],[{"text":"Give a way back.","b":true},{"text":" A support email on the block page, an appeal route, and an unblock that a human can perform in a minute."}]]},{"t":"p","runs":[{"text":"Blocking whole countries, ranges or ISPs is tempting during an incident and is usually a mistake left in place for years. If you do it under pressure, put an expiry on the rule the same day."}]},{"t":"h2","id":"during-an-actual-flood","runs":[{"text":"During an actual flood"}]},{"t":"p","runs":[{"text":"The order that works: turn on your CDN’s attack mode or equivalent, cache everything cacheable including HTML for the duration, disable the most expensive endpoints behind a flag, raise your rate limits’ strictness, and only then start analysing patterns. Diagnosis is much easier from a site that is up."}]},{"t":"p","runs":[{"text":"And afterwards, the useful question is not “who did this” (you will rarely know), but “which endpoint let a hundred requests a second hurt us”. That endpoint is your finding, and caching or a quota is your fix."}]}],"check":{"question":[{"text":"A site adds a strict per-IP rate limit and reports of “too many requests” arrive from an entire university and one mobile network. What does this reveal about the control?"}],"options":[{"text":[{"text":"The limit is set too low and simply needs a higher threshold"}],"correct":false,"tempting":[{"text":"Raising the number helps those users while weakening the limit for everyone; the flaw is in the unit of measurement, not the value."}]},{"text":[{"text":"Many legitimate users share a single public address through NAT, so an address is a poor proxy for a person"}],"correct":true,"tempting":null},{"text":[{"text":"The users are behind a proxy that strips the forwarded header, so all requests appear to come from the proxy"}],"correct":false,"tempting":[{"text":"A real misconfiguration to check, but the described pattern (whole institutions affected) is the expected behaviour of per-IP limits even when headers are correct."}]},{"text":[{"text":"Rate limits should never be applied to anonymous traffic"}],"correct":false,"tempting":[{"text":"Overcorrects: anonymous traffic is exactly where abuse arrives, and the answer is a better key plus cheaper responses, not no limit."}]}],"order":[2,0,3,1],"why":null},"prev":{"slug":"rotating-a-key-without-downtime","title":"Changing a key while everything is using it","minutes":8},"next":{"slug":"payments-and-what-you-must-not-store","title":"Taking money without holding card numbers","minutes":9}}}