Post #387494
2026-02-20 16:49 UTC
Replies (13)
-
@kostyn@niagara.social 2026-02-28 23:03
@patrick@mendeddrum.org @paul@notnull.space @b2c@wien.rocks @alex@fedi.smith.geek.nz @raymaccarthy@mastodon.ie @hulver@hulvr.com @CausticHarmony@mastodon.social @lemonlolita@calckey.world @SuperMoosie@mastodon.au @CassandraVert@indieweb.social @galaxis@mastodon.infra.de @thisismissem Thank you all for your help and suggestions and boosts on this! I'll let you know what we decide to do. :)
-
@patrick@mendeddrum.org 2026-02-20 17:03
@kostyn@niagara.social Not done so, but a thought worth thinking through is whether it would be feasible to move media storage to a (canadian) s3-bucket. That would make the task a lot easier already.
-
@paul@notnull.space 2026-02-20 17:24
@kostyn@niagara.social To attempt a little change of point of view on this - what would you do if you needed to restore from backups? As that's effectively what you're doing: taking a backup and restoring it to a different host. It's definitely a worthwhile exercise to do this if you're running a service in my opinion
-
@b2c@wien.rocks 2026-02-20 18:05
@kostyn@niagara.social Probably depends how/where you currently store your data. I moved climatejustice.social with about the same amount of media storage, plus 5x the database. Media was on some S3 storage, so I used rclone to sync everything to a new storage before the move. For postgres I did a dump+import shortly before the move, ran the import, shut down the servers on the old infra, then just replayed the WAL files that had accumulated in the meanwhile. (Make sure you have WAL archiving configured and set the correct WAL level). When everything was up and running on the new infra, I ran a final, incremental rclone to make sure I got all the media. Overall downtime easily below an hour. PS: don't forget to set you DNS TTL as low as possible. PPS: If you serve media from a separate domain, and this domain changes during the move, you WILL BREAK others instances image caches, and also have to manually update some strings in the database. I wrote down in detail what this means and how to fix it here: https://wiki.fediverse.foundation/books/fediverse-foundation-announcements/page/announcement-change-of-the-cdn-of-climatejusticesocial-2023-07-13
-
@alex@fedi.smith.geek.nz 2026-02-20 18:34
@kostyn@niagara.social rclone will be your friend for the media storage, I’ve moved my instance before (about 600GB of media) between different storage providers without issue (and need to again soon… 🤦🏼♂️) Database - dump it, compress it and you’ll find it’s probably a lot smaller, then just restore at the other end. Are you actively purging media using tootctl? If not you could prune down to 7 days (or another number that suits) before moving things.
-
@raymaccarthy@mastodon.ie 2026-02-20 18:36
@kostyn@niagara.social Save it on an HDD and post / courier it to new hosting company?
-
@hulver@hulvr.com 2026-02-20 20:07
@kostyn@niagara.social I moved media to an S3 bucket well before moving the server. Before the move to an S3 compatible service I ran the media purge at quite a low retention time to reduce the amount of media to transfer. When I moved the main host I shut down the server, then backed up the database and restored to the new host. I'd already copied the rest of the setup to the new machine, so just had to load the database and redis database and start it up after the DNS change.
-
@CausticHarmony@mastodon.social 2026-02-20 20:22
@kostyn@niagara.social If you can reach the single media files of the media storage you could also downscale images and videos
-
@lemonlolita@calckey.world 2026-02-20 21:36
@kostyn@niagara.social Good luck with the move. Have you asked the host your looking to move too?
-
@SuperMoosie@mastodon.au 2026-02-20 22:50
@kostyn@niagara.social Not sure if this helps. Used their users to help transfer data. https://community.hachyderm.io/blog/2022/12/03/leaving-the-basement/
-
@CassandraVert@indieweb.social 2026-02-20 23:00
Ask @newsmast@backend.newsmast.org
-
@galaxis@mastodon.infra.de 2026-02-21 16:19
@kostyn@niagara.social Uh, why do you have 1.6TB of media storage for six users? Did you not set up media expiry, or with excessive thresholds? Except if you're consuming a lot of relays or something, media cache should be below 100GB for an instance of that size. In general, you can just throw away everything below public/system/cache/ when moving an instance - that's all external media that can be re-fetched from elsewhere on demand.
-
@thisismissem@activitypub.space 2026-02-21 18:44
You can do it, as long as you have always had media served from a custom domain (e.g., media.myserver.example instead of like s3.digitalocean.com or something) I'd recommend doing the migration offline, just take the hit with downtime. For media, rsync running on a server is your friend, but first prune your caches (that's the likely bulk of your media storage). This query run against the database will give you media storage consumed per account (local and remote) SELECT a.username || '@' || COALESCE(a.domain, 'local') AS account_identifier, SUM(m.file_file_size) AS total_storage, '/admin/accounts/' || a.id AS admin_url FROM media_attachments m JOIN accounts a ON m.account_id = a.id GROUP BY account_identifier, admin_url HAVING SUM(m.file_file_size) > 0 ORDER BY total_storage DESC; That'll help you prune more effectively — there's some accounts that use an incredible amount of storage, and cause problems for the rest of the network (i.e., reposting the same media every week, rather than referencing the same upload across multiple posts — at a technical level, it'd be better to do content-addressed uploads where the media file key is actually a CID of the binary contents; that's how AT Protocol deduplicates blob storage). For moving the database, first spin up the new server, then setup a database and redis, but without any data. You may need the rails migrations run just to get the sql database structure. Then shutdown the mastodon processes (ALL OF THEM) on the source server. Then take a dump of the mastodon database in postgresql AND take a dump of the redis database (I assume you're using the standard setup, so timelines and sidekiq queues in one redis database). Then you can just scp those across to the target server. For redis, shutdown the redis service, and just move the rdb file to the redis location. For postgresql use the standard data restore tools. The copy with scp and postgresql restore will take some time, but in my experience it's like ~30 minutes downtime. At the same time, you can update your DNS — as you've not the mastodon processes running on the new server yet, they'll see the "we're currently down" placeholder. It's perfectly doable. But, if you're concerned about being able to actually pull this off, you can always hire me. I provide this as a service to clients.