Post #1918529

2026-05-03 09:39 UTC

@niklaskorz and I are currently investigating a Mastodon media cache size issue on rheinneckar.social (https://rheinneckar.social/@admin/116509945016262429) that seems to be caused by a sudden uptick of files being cached starting last night (see plot) and caused our VPS hard drive to fill up almost entirely. Has anyone else seen this already? #fediadmin #mastoadmin

Replies (3)

@Strzelec@mastodon.online 2026-05-04 19:47

@cyclopentane @niklaskorz Hi, would that be related? Or even same reason? https://rls.social/@cdamian/116517140389681996

Open ##2103835
@Jdm2@boriken.social 2026-05-04 21:56

@cyclopentane @niklaskorz yea. Noticed this. Happening over here too.

Open ##2103837
@galaxis@mastodon.infra.de 2026-05-04 22:03

@cyclopentane @niklaskorz Repeating myself on this, but I have seen two sources of exploding media storage this year: 1) Crawlers hitting public views of timelines, behaving like a web browser, and "viewing" individual posts in thread context. They cause the instance to load remote posts and their attachments, greatly increasing media cache usage. Disabling public access to remote timelines and tags searches in the instance admin Discovery settings helps. 2) Crawlers descending into old posts from Trends, which unfortunately can not be disabled just for unauthenticated users. This causes Mastodon to temporarily re-fetch remote media into the media proxy. Since the crawlers use huge ranges of source IPs, the proxy rate limits are rarely triggered. I'm using this patch for the rate limiter: https://github.com/mastodon/mastodon/issues/37987#issuecomment-3969147018 - though with an additional change that curbs unauthenticated access to 1 per 10 minutes with 'throttle_media_proxy_unauthenticated', limit: 1 instead of 100

Open ##2103838