#mastodonau

Well I gotta say, moving #MastodonAU 's database to a pure SSD storage backend has made a massive difference. Whooohoo, she's much faster! The storage servers are a pair of #Dell R640s, with 2x Gold 6150 (2.7GHz 18 Cores each) and 256GB of RAM. Each one has 8 x 1TB SSDs, in a #ZFS raidz2 pool, and they automatically replicate between themselves. I am doing my best to avoid any more visits from the #fuckup-fairy Now to finish #MastodonNZ

Sorry about that long-ish outage, but we were YET AGAIN getting smashed by AI bots. I have given up fighting them, and I'm just scaling up until #mastodonAU can handle the load. This time the #PostgreSQL database was running at a load of 100+ because it was disk IO bound. I did *try* to live migrate it, but it was just insanely slow so I made the call to shut it down (hence the 'Something went wrong' errors) and move it offline. It's moved. Now it's running on a pure SSD-only backing store, which hopefully will handle everything the #AI scrapers throw at us. I'll be doing the same thing with #MastodonNZ too, shortly (but hopefully without the outage, as that's a much more lightweight instance) I also really need to fix the status page that comes up when mau is down.