Dekkia

The idea behind anubis is that a browser needs to deliver proof-of-work before accessing a website. If you’re doing it one-off with puppeteer, your “browser” will happily do just that. But if you’re scraping millions of websites, short challenges like this add up quickly and you’ll end up wasting lots of compute on them. As long as scrapers decide that those websites are not worth it anubis works.

Have you tried to buy a non-smart TV recently?

There’s a tool for that: anubis.techaro.lol Alternatively cloudflare also has scraper-protection offerings.

So it’s opt-out. Great

But that stuff is copywritten as well most of the time. Just because it’s free to look at doesn’t mean it’s free to download, modify or feed into an AI.

This is the definition of ethically sourced data from the Apertus website: […] the training corpus builds only on data which is publicly available. So they still train on Websites, Blogs and Social Media. Ethical my ass.