Bots, scrapers and the flood you cannot absorb

Most of your traffic is not people

Publish anything and automated traffic arrives. Measurements of the web as a whole put the automated share somewhere near or above half, and for a small site the proportion is often higher, because you have no audience yet and the crawlers do not care.

It is worth separating the kinds, because the response differs:

• Search crawlers. Wanted. Obey robots.txt, identify themselves, crawl politely.
• Dataset and AI crawlers. A newer, heavier category. Some obey robots.txt, some do not, and several have been observed requesting the same pages repeatedly. Whether you want them is a decision, not a given.
• Scrapers of your content. They want your data, and they will rotate addresses and pretend to be browsers.
• Credential stuffers and signup abuse. Covered by the earlier rate-limit and session lessons.
• Volumetric floods. Someone trying to make you unavailable, or a badly written client doing it accidentally.

The three layers, and which is yours to fix

Layer 3 and 4: packet floods. Gigabits of traffic aimed at your address. You cannot absorb this on an origin server; the pipe fills before your code runs. It is handled upstream, by your provider or a CDN, and every major CDN including free tiers does it automatically. Nothing you write matters at this layer.

Layer 7: application floods. Requests that look legitimate but hit your most expensive endpoint. Search, report generation, a model call. This is where a small site actually falls over, and this layer is yours: cache aggressively, rate-limit per account and per endpoint, and make sure your most expensive path is not the most exposed one.

Application logic abuse. Signup floods, coupon farming, comment spam, free-tier exhaustion. Not a volume problem: a rules problem. Fixed with quotas, verification tiers and cost controls, which is why the identity ladder in this project’s own design exists.

What actually works, ordered by effort

1. Put a CDN in front. The single biggest step, free at the entry tier from several providers, and it removes both the volumetric layer and most of the cacheable load.
2. Cache the expensive public pages. A scraper hitting a cached page costs you nothing. A scraper hitting an uncached search endpoint costs you a database query each time.
3. Rate-limit by account and by endpoint, not only by IP. Addresses are cheap and shared; a single mobile network address may carry thousands of legitimate users, which is exactly why blocking by IP produces collateral damage.
4. Make the expensive thing require an account. Anonymous access to a cheap, cached page; anything that costs real money behind a login and a quota.
5. Challenge rather than block. A proof-of-work or interactive challenge (Cloudflare Turnstile, hCaptcha, mCaptcha) costs a real user a moment and a bulk scraper a great deal. It is also the honest place to say that these degrade accessibility for some users, so put them in front of abuse-prone actions, not in front of reading.

robots.txt is a request, not a fence

robots.txt and noindex are conventions honoured by well-behaved crawlers. They stop nothing that does not want to be stopped, and they have no legal force in themselves. If content must not be taken, the control is authentication, not a text file.

That said, the file is still worth writing carefully: name the paths you do not want crawled, and add a Crawl-delay or explicit rules for the heavy agents. Several CDNs now offer a one-click block for known AI crawlers. Whether to use it is a product decision: a site that wants to be found may want to be in those datasets, and a site whose content is its product may not.

Blocking, and its costs

Every blocking mechanism has a false-positive rate, and the people it blocks cannot tell you, because they are blocked. Three habits keep this honest:

• Log every block with the reason, and review a sample weekly. If you find real users, your rules are too tight.
• Prefer degradation to denial. Serve a cached copy, slow the response, or require a challenge rather than returning 403.
• Give a way back. A support email on the block page, an appeal route, and an unblock that a human can perform in a minute.

Blocking whole countries, ranges or ISPs is tempting during an incident and is usually a mistake left in place for years. If you do it under pressure, put an expiry on the rule the same day.

During an actual flood

The order that works: turn on your CDN’s attack mode or equivalent, cache everything cacheable including HTML for the duration, disable the most expensive endpoints behind a flag, raise your rate limits’ strictness, and only then start analysing patterns. Diagnosis is much easier from a site that is up.

And afterwards, the useful question is not “who did this” (you will rarely know), but “which endpoint let a hundred requests a second hurt us”. That endpoint is your finding, and caching or a quota is your fix.