Filter out buildings and other non-runnable features


#41

Would be nice if you could see already flagged streets in a city and have the option to either confirm or contest the flag.


#42

Similarly, I think that manual flagging should be reviewed. If someone has run a street without manual flagging, it should warn or notify you at least, if you try to manually flag. I know it’s just as easy to fake a GPS segment, but I’m fairly opposed to the concept of manual flagging.


#43

I get where you’re going with this. But for instance, yesterday, i tagged several streets as non accessible, cos (parts of) these streets don’t exist anymore! there is a huge highway built at this moment, and lots of streets are simply gone. And in the past some runners did the old road of course when it still existed. So what you are proposing is not always realistic.

And we both know we differ in the citystrides approach. I just want to run all public streets of a specific city, not some irrelevant non-street data that was uploaded to a city. If the data imported as streets would be ‘clean’, this whole discussion wasn’t necessary.


#44

Thanks for all the great work on this. The street list is getting better over time. I’d like to offer a slightly different approach. The community approach is important and we need to have it, but there’s still a lot that can be done safely with automated rules.

Now, we know OpenStreetMap is not perfect and can have bad data. But some elements just simply cannot be confused with runnable features and can safely be removed by algorithm.

If we can set a few filtering rules that we can all agree on, it would eliminate 90 percent of the non-runnable features (at least in the cities that I frequent).

Here’s my proposal:

  1. Eliminate any feature with the tag building=yes. There is no possible chance that a feature rendered as a building on OSM should be treated as a runnable feature.

  2. Eliminate any feature with a waterway key, regardless of value. Lakes, rivers, streams, etc., are not runnable and cannot be confused with roads or paths.

  3. Eliminate all leisure=park features. In fact, you might consider eliminating anything with the leisure key with the possible exception of leisure=track. I have found that the leisure=park feature is used very commonly and is rendered as a park boundary area.

  4. Eliminate any feature with a landuse key. These are also rendered as boundaries and will never be runnable features.

By excluding obvious non-runnable features we can reduce effort but still take a conservative approach that won’t inappropriately remove streets and paths.


#45

I am thinking about extracting my data processing code out of my main codebase, in order to share it around. In the meantime, this Gist contains the code that extracts Street and Node data out of OSM files.

There’s a whole World Of Process™ that occurs around these scripts:

  • collect .poly files for Cities within Regions within a Country
  • extract .osm.pbf files out of a Region’s .osm file
  • convert the .osm.pbf file to .osm file
  • then run the two scripts linked above to create two .json files (one containing Streets and the other containing Nodes)
  • then there’s another piece of code to import the data from the .json files into the database

However, the piece that you’re discussing - ignoring certain data types within the Region’s .osm file - is all within the two scripts linked at the top of this post.

…Anyway… Follow the two links at the top of that Gist, because the person who wrote the code (I just modified it a little bit in order to collect Street and Node data) goes into more detail on the process.


#46

Okay, I see what they did. Basically this script pulls down every that has a “name” tag.

I modified this a bit to also exclude any ways that have one of a handful of “definitely not runnable” features.

See Code: https://repl.it/@BrianSperlongan/QuickwittedIndolentNonagon

Sorry, not able to test or run it, but I think you get the idea…


#47

Quick question… is the process to flag the non-runnable features AND mark them manually complete, or just flag them?

I think I mentioned somewhere else, that it would be great if we can flag/complete individual nodes… such as gated communities as one example. The street might be a runnable street, but then end prematurely due to gated community, so don’t want to manually complete the entire street, just the nodes I can’t get to.


#48

Marking them complete will screw up your percentages. Frankly, I wish the “mark manually complete” option would go away. If it’s in the system and it’s runnable, you should have to run it.


Remove 'manually completed' feature
#49

Could something like this be done?

CityStrides users/volunteers could submit a list of all streets, in a predetermined format, for their town and that list would be used as the standard for completing 100%.of the streets in that town. If a user completes all the streets on that list, it would show 100%. If a new street was created due to a new development, a user could submit a new street name to be added to that list and the users would now see a completion of 99%, signaling them that a new street has been created in that town.

Users could continue to run other features, like parks, trails, etc and have them display on their LifeMap but would not count towards % of streets complete.


#50

sounds like the prefect solution to me


#51

The ‘manual completed’ idea came from the worry that someone might be an avid runner, but not using a run tracking service … then starting both run tracking & CityStrides … and being able to tick off all the streets they know they’ve run (maybe they have a particular loop they do all the time).
That, or GPS issues during tracking that manage to miss some Nodes on a run, or something.

Thanks for sharing your perspective on this. I’m imagining what things might look like without that feature. I think the GPS issues problem is real (especially the way that I currently determine if a Node is ‘completed’ or not), but I’m not sure if that occurs frequently enough to really matter. I think the ‘new user with an untracked history’ problem is garbage, upon reconsidering it. :laughing:

I want to see what everyone thinks about this idea, so I added it to the Ideas category.


#53

@JamesChevalier, what are you using as the input for generating the streets list? Are you working from the offline copy of OSM? I’ve been playing with a subsetting tool called osmfilter, and it looks like it’s possible to filter an offline file in-place which could solve the vast majority of the problem with non-runnable features without having to change any code – the input global OSM file could just be pre-filtered before letting citystrides loose on it.

Is that a viable approach?

Also, this could considerably cut down on processing time if your starting OSM file is much smaller.


#54

Re ‘new user with untracked history’, I’m one such user, in a way. While I did start out with a Garmin watch, for a few months/years (?) I switched to some app that rewarded my runs with gift cards but is not recognized by CityStrides. Completely, i.e. I used only one tracking device at a time. My plan is to re-run whatever streets I know I already ran back then. It may be possible to find the old data and somehow import into CityStrides, but it’s more enjoyable just re-doing the physical work.


#55

I’ve thought about this also, but I realized before the advent of the GPS device, most runs were out-and-back or loop courses where the distance was determined by measuring with an odometer from a car or bike. Even if all my years of old data were uploaded, I would have to run those routes again just to access all the missing side streets that I didn’t run. So…it is much easier for a new user to start with a clean slate and enjoy the process.