I’ve got some code that will update CityStrides data from OpenStreetMap. Your edits in OSM will soon make their way into CityStrides!
I’m testing it out on cities one by one, so I can see how long it takes & how many resources it consumes.
This code also solves the issue of Some Streets contain Nodes outside of City boundary
This code does not (yet) handle updates to the city border itself.
Update: There’s a bug in my post-processing, which caused people to reach thousands% complete for some streets. The cleanup for this is to completely remove all progress in these cities & rebuild it. Luckily, this only takes a few minutes per city & I’m going one at a time so there’s as short of a lag as possible between the progress being wiped & rebuilt.
Update: OK, I’ve cleaned up the hundreds of millions of bad records. I’m just about ready to resume tests.
Update 2020-09-14: I’ve got a decent system planned out, but I’m having trouble with excessive IO (from reprocessing users after updating cities) that sometimes causes downtime. I’m slowly testing groups of cities to see how many I can update at once, how much delay I need between each group, etc. This initial test is probably going to be more difficult than regular updates will be, once I get things running continuously - I’m updating cities after a year of OSM changes. There should be way less to do once I get this to a point where I’m updating cities weekly (my goal; unsure if I’ll reach it).
Update 2020-09-18: I’ve just started the automatic system & I’ll be monitoring to make sure it doesn’t overload things. It’ll update 10 cities at a time, and it checks minutely to see if it can queue up more cities. It only updates active cities (places with at least one person running there), so that’ll help speed up the frequency of updates. It also tags the ‘last sync’ date, so it’ll re-update in a set order. Right now there are 51,162 cities left in the first update pass (applying all updates in OSM sync 2019).
Update 2020-09-28: 26,068 cities left to update … the first run of this update created some streets with 0 nodes; I think I’ve fixed that, so these 26k cities shouldn’t have that issue. The next pass through of the update code will correct the other cities. I don’t have a good view of throughput yet, I’ve seen numbers from 4 cities per minute up to 16 cities per minute. So I think it might be able to update all cities every 3-9 days. Weekly is nice, I could live with that, but I’m aiming to get it down to every other day - that would be great.
Update 2020-09-29: 25,178 cities left to update … I’ve had a few in this recent batch take double-digit hours to complete.
Update 2020-09-30: 23,589 cities left to update … looks like rate is about one update per month, so far.
Update 2020-10-02: 17,655 cities left to update … I made some improvements that speed things up a bit, but I’m still wildly far off from my every other day goal.