City progress incorrect / Completed street count decreasing

From Jan 18th-30th, the progress calculation that happens after a city update has been incorrect. This is due to a speed-up that I implemented which reduces city update lag from ~2-4 months down to ~2 weeks. This update is causing the progress calculations to occur before other data manipulation is complete, which incorrectly marks things as incomplete.

At the moment, city updates are not running & I have run per-user reprocessing for anyone affected. At this time, all progress should be correct.

I’m actively working on this issue & will update this post as I make progress.
I may not reply in this thread, but I am keeping track of it. I also do have enough example data in the thread below, so you don’t need to share your info any more.
Thanks for your patience!

2 Likes

After my local town was refreshed a couple of days ago, I have a couple of streets showing as 100% complete but in incomplete list.

It shows as a newly created street as the name has changed since I ran it, so not sure if is just a case of waiting a couple of days for a refresh or if there is a bug.

Edit: All updated now so guess was a timing issue

1 Like

Same thing is happening to me with OSM updates coming through to CityStrides today. How long did you wait for this to get resolved? I’m overly anxious to see my 100% complete again :slight_smile:

1 Like

non-:nerd_face: version: Damn. Yeah, I’m looking into this now.

:nerd_face: version: Mike’s issue :point_up: was a massive queue backlog which prompted me to focus on improving post-city-update processing … I accomplished that :trophy: :rocket: but it looks like the new processing setup I have needs some further tweaks to keep the data consistent.

5 Likes

When my run uploaded today, several streets that had been completed in the past now say incomplete. When I go in to the city and check out the incomplete streets, most of them say 100% complete. I went from 13 cities completed to 6. @JamesChevalier I’m wondering if something went wrong or :woman_shrugging:t2:

1 Like

https://community.citystrides.com/t/street-listed-as-incomplete-shows-100-no-red-nodes/22065/13?u=jameschevalier

The post-city-update processing yesterday & today has been checking street progress before the updated/created nodes were processed. I’m working on resolving that, and after I do I’ll send up some jobs to correct everything.

1 Like

I’m having the opposite issue with a run that just uploaded in the past 5 minutes. The streets I ran appear in the completed street list, but the nodes still appear red :thinking: hopefully that clears up too with whatever you’re fixing in the background. Glad I get proper credit for the streets though.

@JamesChevalier - Are you thinking that the issue is fixed at this point? I am seeing the same issue with one of my completed cities that updated yesterday and now shows 14 incomplete streets, yet all of them show 100% complete in the street details. The rest of my completed towns that also updated yesterday are fine. Here is the link to the one that is the issue:

Seems to be fixed now, thanks!

Cities updated on the 18th should be cleaned up. Cities updated on the 19th will be cleaned up sometime today. It will lag by a day until I’m able to fully resolve the issue. :sweat:

3 Likes

@JamesChevalier , thanks for working on this. Is there a chance the clean up jobs ran in hard mode. I have multiple streets still that were showing complete in cities updated on 1/18 that show incomplete now. I’m not set for hard mode and all of them are well above the 90% threshold.

Seem to be having the same issue- lost around 250 streets completed since yesterday and a few completed cities

Same here. My own city “Elmbridge” had an update yesterday. I ran 2 streets which got added (questionable industrial estate ones but hey could do them) but 3 remain including one literally 100 yards from my door and the other two close. So I have of course

Actually is OK now. Guess just takes a while ?

From about the 18th to about an hour ago, things were dicey. I’ve sorted out the underlying issues, but tomorrow I’ll have to reprocess the cities that were updated today.

These issues all stems from my efforts to improve the city update performance (I over-improved some code in a way that caused it to run before other processing was finished). :nerd_face: It’s still way too early to tell how much faster the updates will be, but I’m decently confident that they’ll be faster than every 60 days (I’m not-so-secretly-anymore hoping for something closer to every 2 weeks, but it’s probably more realistic that it’ll be just-under-monthly).

4 Likes

Thanks for the update. I have got 2 “big” cities that have almost done (single figure streets to go) but was kind of hoping for an update any day as I won’t be able to fully complete either until a few streets I found to be gated/wrong on OSM that have updated are reflected here.

It is a bit annoying with the 2 month lag at the moment as if somebody marks OSM wrongly it takes ages to see the update from OSM. For instance, I found myself trying to run through somebody’s private driveway and to the side of their gate the other day to try and collect such a street. I did but should not have been trying TBH!

If it was Street Ferret, I could request an update on demand and it will be done in an hour or so. Maybe though they have a lot less users and that one is largely personal anyway. That is a huge bonus for me though to have control. I don’t see any “fun” in waiting 2 months to see a street I can’t run being shown that I can/should…

I’ll definitely be interested in hearing whether or not your cities get an update within the next 2 weeks. I’m hopeful that it will be sooner, but it’s still too early to tell.

Yeah, and making it even worse was that things were on track to extend out to 60-90 days. I’m really glad I spent the time to improve that - even if it doesn’t reach my 2 week goal, it’ll be better than 2 months!

Yeah, things get Funky™ when over 50k people show up. That could be an interesting supporter feature, though. With my recent updates, I think the system could handle on-demand update requests.

3 Likes

The recalculation did not use hard mode rules … Can you share links to a street or two that should be marked as complete for you?

50k runners, and 173k cities. Would be interesting to know how many of these cities are actually of interest, that is, being in the run scope of any of the 50k runners… Would it be possible to have a mechanism where cities that have been run by a CS strider are prioritized in the update queue?

Right now, the homepage while logged out shows some stats
79,247 active cities
4,742,431 completed streets

That’s already in place. The system does not update cities that have no completed streets.