Post #1804295

@phnt@fluffytail.org @i@declin.eu @lain@lain.com Now oban_jobs is down to 147 entries, TBH I've never seen a queue this short on this instance — and most of those are expiring tokens (I use it on quite a few devices and have dozens of those), polls and other such things. Neither memtest86, nor extended SMART test revealed any irregularities with the hardware. Now that the worst outcome looks averted I have a different theory what could have caused a congestion this major: due to heavy network filtering I have to rely on VPN, even fetching images from some instances might be getting filtered, so now I have configured even media proxy to make connections via Squid. And I completely forgot about it, but earlier that day VPN link went down for a couple of hours: instance was still accepting inbound connections, but wasn't capable of making outbound ones. Some jobs of course time out when it's like that, but some get rescheduled and it still causes the queue to fill up. Usually it's not a problem and it just churns through accumulated jobs when the link gets restored, but like I said, the machine is old and that day looks like I didn't give it enough time to — I started interacting with people who have hundreds of followers, posting media, etc. This likely caused the queue to clog up further, up to a point when PostgreSQL started choking. I think it was this and not Dendrite — I've set it up for a few friends, but I believe they still mostly use XMPP, so it just sits there dormant most of the time. There is also Mozilla's bookmark sync service sharing PostgreSQL, but its DB I/O is minuscule compared to that of Pleroma. That is probably why restarting Pleroma didn't do the trick, but restarting the machine did — it's given it enough time to catch breath before new inbound requests started coming. I might consider setting up a quarterhourly cron job to check the queue length and configure nginx to temporarily start rejecting requests to Pleroma if the queue is above a certain threshold, or at least notify me with an XMPP message, so I won't make it worse again :marseyemojismilemouthcoldsweat: As for queue_target and friends — I tried experimenting with these parameters earlier and it certainly helps when the storage is just slow, but in cases like this it could've made matters even worse, instead of dropping requests they would've kept piling up. I had this instance on an even slower machine before and in severe cases it could even render the frontend catatonic, Pleroma was still serving requests from other instances, but I'd have to restart it in order to use it myself :marseyitsover: Thanks everyone for your advice, but looks like the reason is same old — me hosting stuff on medieval hardware :marseylaughwith: And otherwise it was just an unfortunate coincidence.