Post #3474223

@doctormo@floss.social My current hack is I know there's some urls on my website that are dynamic enough the scrapers love searching them, but real people shouldn't be that interested. grep -E "suspicious|url|patterns" /var/log/apache2/access.log | cut -f 1 -d ' ' | asncounter --output-format tsv --top 4000 --no-asn -o /tmp/scum.tsv asncounter groups ip addresses by network so it restricts the trick of using a bunch of proxies on residential networks. Then I wrote a small program to select records out of scum.tsv and feed them to fail to ban like this, fail2ban-client set apache-noscript banip 117.188.0.0/20 1.80.0.0/13 180.72.0.0/14 ... Fail to ban will expire a ban after awhile, and I've got it set to increase the ban time for multiple hits. And then I kept doing that every morning, and some network bans kept getting increased. Just make sure to exclude your administrative ip address. And the AI companies cannot die soon enough.