Something happened with a stats job that runs nightly (and has run nightly for years) … it caused massive database load (disk IO), which slowed everything else down.
The main job that’s affected is the one that sets completed streets on activities - it has a 2 hour delay right now. So, you will see your completed street count roughly 2 hours after the activity is saved in CityStrides. There are a lot of other jobs affected, though, so expect some general weirdness. Sorry.
It’s not reflected in the Status page because - for whatever reason, this code is 4 years old - I have that job sent to a different queue. I compared the two queues and checked where the job is enqueued from, and I don’t see any reason to use that separate queue so I’m going to move it over.
It’s not going to spring back into action now that I’ve stopped the stats job, because the database burned through its ‘burst balance’. This basically means the database needs to chill out for a bit (you and me both buddy) so the credits can accumulate. This means not doing any (or at least way less) work, though, which isn’t exactly the situation it’s in right now.
I’ve got some deferred maintenance that I’ve forgotten to do every night for the past week, so I think I’m going to push that through now since everything is shit already.
I don’t know how notifications work in threads like this, so just in case:
@peterathburn @kf1977 @u6117853 @Scott_McClure @karenadelson88 @kevincharlespels @hackles.blue-7v @tor.ivar.hansen