Post #4360718
2026-08-03 15:31 UTC
@WeirdWriter@caneandable.social Not all people want or can afford beefy servers. There are also some webmasters who block crawlers/scrapers out of principle.
Replies (1)
-
@ArneBab@rollenspiel.social 2026-08-03 16:40
@raulmatias@mstdn.social A solution to this is to simplify the website. My website had a many million scraper hits at the beginning of last year (the server log parser almost couldn’t handle that), and all that happened was that it switched to the reduced interface (which it needs anyway to stay online when a big site links to it: don’t want to lose all traffic in case of such a happy accident). I pay about 8€ per month for the server. It’s a plain LAMP stack, maintained by the hoster. @WeirdWriter@caneandable.social