Yeah, there’s serious load right now in a way that I’ve never seen before. I’m trying to sort things out, but I’m not making much useful progress.
I also had a huge amount of yard work to do today from a recently trimmed tree, so part of my approach was waiting and seeing if it would fix itself.
Narrator’s voice: It did not.
Everything will eventually be sorted out - either by waiting for the slow processing, or by me figuring out a way to speed things along.
You can stop reading here if you’re like “ just fix it”
My background job system uses Redis and the server that’s running that is at 100% load. I’ve never seen this load on that server before.
Part of my job system schedules work a few seconds in the future because I’ve seen some strange behavior with bulk data imports (saving lots of GPS records attached to activities) and then immediately processing that saved data (I use Sidekiq for background jobs & it’s ‘too fast’). Right now I have 6 million jobs queued up to process these GPS records, and they’re not being added to the queue for some reason - just sitting there displaying “4 hours ago” as their run-time.
I also have some code that tries to prevent duplicate jobs (e.g. someone mashing on the ‘sync now’ button) by checking for itself in the queue. I think this is causing the bulk of the problems, because it’s searching through 6m+ queued jobs.