How do I stop AI bots from flooding my website with junk traffic?

Layer published crawler rules, edge rate limits, bot screening, targeted blocks, and caching so one misbehaving agent cannot overwhelm the origin server.

Start by naming allowed and disallowed paths in robots.txt for reputable crawlers, but remember that rules are requests, not a firewall. Put rate limits at the CDN or reverse proxy so one IP, account, or user agent cannot hammer expensive routes. Cloudflare users can turn on Bot Fight Mode, then inspect logs before blocking a user agent so legitimate search or accessibility tools are not caught.

Cache public API responses, add short timeouts, and protect costly endpoints with quotas or authentication. If abuse continues, block the misbehaving signature temporarily and contact the operator with timestamps and request samples. The reason for layers is simple: polite agents respect rules; broken or deceptive ones must be stopped before they reach your server.