Post #2972675
2024-12-27 14:53 UTC
@khobochka@mastodon.social
Doing ultra-wide dumb crawls without proper domain rate limit or caching and the like just sounds like severe developer incompetence.
Replies (1)
-
@khobochka@mastodon.social 2024-12-27 15:11
@ide@masto.ai I don't think this is a question of competence: competence could be summed up as an ability to efficiently achieve goals, which involves feedback leading to anticipation of problems (e.g. use domain rate limit -> avoid getting blocked, use cache -> lower compute by using storage). However, there's neither a defined goal nor any feedback here: we can't harm an LLM crawler that causes harm, its developer won't cache an entire internet and has access to near unlimited compute. This is ethics.