#mastodonau

Sorry #MastodonNZ but there is a good chance I'm going to have to do a complete database dump and restore, as there's still some bits of corruption left over from the crash the other day. This is probably an hour or so outage. I'm actually not even totally sure that #PostgreSQL will even restart properly when I shut it down. More information will be in this thread (so you can look at it on #MastodonAU directly via the web interface.)

Well I gotta say, moving #MastodonAU 's database to a pure SSD storage backend has made a massive difference. Whooohoo, she's much faster! The storage servers are a pair of #Dell R640s, with 2x Gold 6150 (2.7GHz 18 Cores each) and 256GB of RAM. Each one has 8 x 1TB SSDs, in a #ZFS raidz2 pool, and they automatically replicate between themselves. I am doing my best to avoid any more visits from the #fuckup-fairy Now to finish #MastodonNZ

Sorry about that long-ish outage, but we were YET AGAIN getting smashed by AI bots. I have given up fighting them, and I'm just scaling up until #mastodonAU can handle the load. This time the #PostgreSQL database was running at a load of 100+ because it was disk IO bound. I did *try* to live migrate it, but it was just insanely slow so I made the call to shut it down (hence the 'Something went wrong' errors) and move it offline. It's moved. Now it's running on a pure SSD-only backing store, which hopefully will handle everything the #AI scrapers throw at us. I'll be doing the same thing with #MastodonNZ too, shortly (but hopefully without the outage, as that's a much more lightweight instance) I also really need to fix the status page that comes up when mau is down.