Post #2972703

@khobochka@mastodon.social and the crawlers aren't just stupid for repeatedly crawl the same identical pages over and over, they also do completely nonsensical stuff. At my work we noticed that GPTBot had found our search button and did what looks like 100.000 requests per day to /search/amp;amp;amp; repeated 100 times and then our support phone number or a url to a random image followed by more amp;'s.