Incident Report: Activity progress not calculating
Status: Resolved
What happened
When your saved activities synced into CityStrides, there was a large delay in progress calculation. While it mainly affected non-Supporters, the AMSAP Challenge League was affected as well as any ongoing city syncs.
Since progress wasn’t calculated, street lists in cities could be incorrect and Node Hunter could display nodes that haven’t been marked as complete yet.
Root cause
Something caused my database to read from disk way more than it usually does, which slowed everything down.
Resolution
Paused processing that occurs from city syncs, to prioritize new activity processing.
Added an index to help reduce disk reads.
What’s next
The backlog needs to be worked through. A good thing is that it’s working through it faster than it was before this issue.
I’ll merge all the related threads into this one.
I’ll update this post as the issue moves to be fully resolved.
I appear to be having the same (or a similar) issue with this activity: CityStrides
It seems very odd that I should have 0 streets completed and 47 streets progressed as a result of this activity. I have checked a couple of streets and they look like they should have been “completed”. Apologies if I am missing something–normally I tend to just wait and things usually sort themselves out somehow, but I haven’t seen this sort of issue before.
edit: It now appears to have been magically fixed. Thanks!
I had the same issue in Farmington Hills, Michigan tonight. I had five streets progressed even though they were 100% completed. I manually marked them complete but that feels like cheating to me and it also does nothing for my global ranking.
I have a slightly different issue where it says they are completed (say ~20 unique streets) but the number it says is 3, which is what is used to compute my total streets count. So it correctly marks them as complete, but doesn’t increment the count.
I only noticed this on one of my recent runs. It was in suburbs I hadn’t run before. Dunno if that is relevant.
I had the same issue yesterday with an activity that listed completed streets as incomplete. Eventually the processing finished, and even later I received a city completion email (the activity finished a city). All this tells me that there is a very large processing job going on in the background that is affecting all other processing jobs, and unfortunately it is not accurately reflected in the Status page, which shows that everything is running smoothly. As James has not yet chimed in to clarify, I would suggest everyone continue to report their issues but be patient, as my experience shows that processing is still happening but at an extremely delayed pace. I’m sure an explanation will be forthcoming when James has the bandwidth.
Same issue here - today’s run shows 0 completed streets and 11 in progress but when I check streets individually, at least some of them show as completed
Something happened with a stats job that runs nightly (and has run nightly for years) … it caused massive database load (disk IO), which slowed everything else down.
The main job that’s affected is the one that sets completed streets on activities - it has a 2 hour delay right now. So, you will see your completed street count roughly 2 hours after the activity is saved in CityStrides. There are a lot of other jobs affected, though, so expect some general weirdness. Sorry.
It’s not reflected in the Status page because - for whatever reason, this code is 4 years old - I have that job sent to a different queue. I compared the two queues and checked where the job is enqueued from, and I don’t see any reason to use that separate queue so I’m going to move it over.
It’s not going to spring back into action now that I’ve stopped the stats job, because the database burned through its ‘burst balance’. This basically means the database needs to chill out for a bit (you and me both buddy) so the credits can accumulate. This means not doing any (or at least way less) work, though, which isn’t exactly the situation it’s in right now.
I’ve got some deferred maintenance that I’ve forgotten to do every night for the past week, so I think I’m going to push that through now since everything is shit already.
My life map hasn’t updated my activity from yesterday after being completed for over 28 hours. I was also wondering if there was a way to make the roads (2) that I had to mark as manually complete purple on the map. It says I did not hit all the nodes but the roads in the same area of eachother both run into driveways
Unsure if related, but I’m having an error where a street that I had marked as manually completed is sorta stuck. I ran the street 3 nights ago and tried to unmark it from the manually completed list but would get 2 different pop-up messages. One would be green and saying in progress and then when trying the next day the message would be red and saying try again later.
Figure it will eventually fix itself but it’s been a few days and is low priority. Now that I see this thread going, thought I’d add to it just incase.
right now, so it’s going to take some hours to reprocess.