When new data is ingested for a city, is there a sequence of what is re-processed & shown in UI?
Ex:
Data ingested into DB
existing streets updated
new streets created
count of streets in UI updated (both city page & on user page)
Check for all users with GPS in city
If user has GPS in city, re-process each activity to check for completion
Update user’s city completion % and incomplete/complete lists
etc, etc
I ask because I see my main city received an updated today. Looks like some new streets were added while others had some pre-existing street changes.
Reviewing one pre-existing street (Amanda Road | CS | OSM | looks to have been split and part of it marked as private) I’ve previously run all now eligible nodes. It’s still listed on my incomplete list.
A “newly” created street (Hobkirks Place | CS | OSM | looks to have been a way without a name until recently) currently comes up as a zero node street in CS.
I’m perfectly content with giving it a couple hours/days/zeptoseconds (my favorite unit of time) to complete/re-process/update everything. Just curious cause I’m part and wonder about the inner workings and general time frames for things.
this queues up background jobs to process completion/calculations
update the lat/lon values for any moved nodes (rare, but it can happen)
this queues up background jobs to process completion/calculations
update the osm_order values for changed nodes (this is an internal thing I haven’t put into use yet)
update counters (number of streets in city / number of nodes per street)
That flow (the outer bullet list) can take seconds/minutes/hours, depending on the size of the city.
The background jobs could complete very quickly or very slowly, depending on how much updating is done. It’s still a bit too erratic to be worth putting on the status page. It would cause more questions than answers.
I am relying on the latter background jobs (from new/moved nodes) to do the work of calculating percentages that may have changed from the earlier node deletions.
This is a great lead - I’ve been trying to figure out which step of the process is creating these zero-node streets (whether the code was creating new streets in CityStrides without nodes or whether it was deleting streets incompletely – deleting their nodes but never the street itself). It looks like it’s on the creation side.
This zero-node issue may also pair with an incorrect-count issue. If the streets have an incorrect count of nodes, then that will affect the completion percentages. It doesn’t seem to be the case for Amanda Road, though.
In case further instances are needed to confirm, found a couple other “newly” created streets (ie look like pre-existing OSM ways that didn’t have names until recently). All lead to zero node streets in CS.
This one was pre-existing but had a typo in name corrected. Also leads to zero nodes
Marly Drive | CS | OSM
And to compare against brand new OSM way that got its name at same time as creation; looks to work fine in CS.
Citadel Ave | CS | OSM
Tl;dr - CS works like phone plans. All the brand new customers (streets) get the fancy perks and work great. All the pre-existing subscribers (streets with changes) get the shaft and are stuck with their flip phone for two more years.
This is something different, if it existed in CityStrides … was updated in OSM … and after the update now has zero nodes. This is not a case of a new street being created with zero nodes. Also helpful to confirm that there’s more than one path to a zero-node street.
To clarify what I’m looking at…
Scenario A:
Street existed in CityStrides, with nodes
Street was somehow updated in OpenStreetMap (nodes added, nodes removed, re-tagged, split - there are many possibilities)
Street now has zero nodes in CityStrides
Scenario B:
Street did not exist in CityStrides
Street was somehow updated in OpenStreetMap (named, re-tagged to fit our needs)
New street was created in CityStrides, but it has zero nodes
It’s a full query, but it includes timestamps & I use those to skip over stuff that hasn’t been updated.
(This might shine a light on my zero-node issue … I just realized that the “create new nodes” step is ignoring nodes that haven’t been updated since the last city update … but I’ve recently learned that OSM updates don’t touch related records → if you update a Way, its Nodes aren’t touched to increment their timestamps … so one of the things that is likely happening is a Way being updated so that it’ll be included in CityStrides, but none of its Nodes being updated, so my code creates the new Street but doesn’t create any new Nodes for it)
Update the last few cities I’ve seen go through the update process have come out the other side without zero-node streets … I’m hopeful that this is resolved, but I’ll want to spot-check specific cities with people before I call it
and wonder about the inner workings and general time frames for things.