@denschub@mastodon.schub.social
Post #2972701
2024-12-30 10:24 UTC
@JustinDerrick@mstdn.ca @khobochka@mastodon.social this matches my experience, yeah.
I'm also convinced those companies have dedicated crawlers for specific applications like mediawiki or phpbb.
I haven't seen the same aggressive behavior on any other web thing I host, and the coverage and approach to crawling the wiki almost makes me believe this is a mediawiki-specific crawler.
Replies (1)
-
@JustinDerrick@mstdn.ca 2024-12-31 01:05
@denschub@mastodon.schub.social @khobochka@mastodon.social This forum runs of SMF - Simple Machines Forum - but it has existed for 10+ years, and has thousands of posts and tens of thousands of replies — precisely the type of thing LLMs need to be fed.