Filter out buildings and other non-runnable features


#1

It seems that when importing data from OpenStreetMap, citystrides does not filter out properly features that are not runnable. For example, I have a lot of

and so on (forest, administrative boundary, quarries…)

Would it be possible to filter out all those features ? I guess that filtering on the “highway” key would do the job (tag k=“highway”) http://wiki.openstreetmap.org/wiki/Highways

Vincent


Improved Flagging
#2

+1
That would make things cleaner in citystrides!


#3

Some of these should be removed (such as rivers and interstate highways), but I would argue that things like parks and buildings should be left in…although possibly in a separate category.

I would rely on the community to manually flag the objects that are impossible to run.


#4

+1

Buildings especially throw off things like the percentages in big cities. I am in Seattle, which has thousands of buildings and many of them due to the block sizes are going to be impossible to complete because of their corners being too far away or would require going onto private property. I think buildings are also just outside the spirit of what the site is trying to be.

I like having parks, but parks can suffer from similar issues because you have to get near their borders, which can be impossible or unsafe at times.


#5

+1
Some of the features are errors from the OSM, e.g. http://citystrides.com/streets/4500364 is a point in the middle of tram tracks, far from where the actual Place Dumon is located.
I would keep parks - not necessarily their borders, but if there is a walking path you can run, why not?

However one has to be careful - some highways are normally not runnable, but you can run there during a race, e.g. the bridge between Europe and Asia which is runnable only once every year during the Istanbul Marathon. It is not that easy to distinguish them from really unrunnable highways - I am stuck at 95% in Kraainem, Belgium partly because of that - but maybe it is better to be conservative?

Krzysztof


#6

Sorry if I don’t understand your point of view. Why do you want to keep buildings or park limits ? Do you run those features ? I run on paths in parks but not on the limit of the park.


#7

Hey Vincent & everyone else in the thread - I just want to quickly poke my head in and let you all know that I’m aware of the problem & really want to get it fixed. I don’t have a ton of time right now to get into all the details and decisions that got CityStrides here, but I can at least start to explain…

Yeah, I’m in a tough situation on this one. The (somewhat) shortened version of the situation is that OSM doesn’t have perfect data, so I opted to take in more than I needed with the intention of trimming out the excess. I added ‘flag for review’ buttons on each street, kind of taking a cue from Foursquare & how their user base helps manage the site’s data (or at least they used to, by giving certain ‘super users’ more access to the moderation system).
So I’ve got the start of a moderation system, but I haven’t really been able to follow through on that completely - adding different levels of access to certain users, or automatically removing streets that are flagged a certain number of times, or something else…

If I only import ‘highway’ tags, I lose out on things that are not tagged completely. In some areas, this would make my import useless.
If I import everything (like I am) and sort things out within CityStrides, then I start with a mess and need to clean things up. That’s where I am now.

Gotta run for now, but I’ll be back when I’ve got some more time for more explanation. If anyone has any more questions/suggestions, please share them - I’ll reply everywhere I can!

Thanks for talking about this everyone, I really appreciate it!


#8

Hi James. Thank you for the explanation.
I think that if there is a problem in the data, maybe the best solution is to fix it in OSM directly. OSM is a community map. Everybody can contribute and see his/her work appearing in the map within the minute. You can contribute by drawing missing features, editing attributes or even just post a note explaining what is wrong in the area (an OSM contributor will try to fix it). Of course then Citystrides will have to face the challenge of syncing its data with OSM more often…

In the area where I run OSM seems to be reliable. In this kind of area, there is a big amount of work for runners to flag non-runnable item, and a big amount of work for you (or a reviewer) to delete them. So I’m not sure this system will work in big cities.

Could you point me an area where feature are not tagged the way they should be, sothat I can better understand the problem ?

Vincent


#9

James–it’s awesome that you have started planning a moderation system and it is probably the best way to filter out the data. Even the streets themselves are intricate…for instance there are streets within gated communities that are impossible to run…and busy streets without sidewalks that are not safe to run.

The community should probably try to reach some kind of consensus about what data should be retained. Even parks can be complicated. I personally think it is cool to be able to run parks…but maybe a moderator could move the tags onto the paths instead of on the borders of the park (would be cool, but probably tricky to implement). Then there are a few parks (in Seattle) that have named trails, which I think are also important and worth retaining since they are frequented by runners. I’m ok with buildings being deleted. I bet it would only take a few dedicated users per city to filter things out.


#10

Hi Jim,

Have you tried the overpass API or overpass Turbo to import data from OSM ?
http://overpass-turbo.eu and http://overpass-turbo.eu/
It lets you query OSM through a web service and get the data immediatly as GeoJSON, gpx, kml, etc.
You can specifiy the type of object, a location filter, a time filter…
It is very simple to use and the wizard understands natural language.
For example, If I type “Highway in Walhain” in the overpass turbo wizard, I retrieve all roads in my village.


#11

Yeah, that’s how I build my queries when adding more countries. Each country’s data isn’t necessarily at the same level (not all countries adhere to an admin level organization of ‘Region = 4’ and ‘City = 8’), so I test different queries with the Overpass API until I find the best admin level matches.

Sounds like I just haven’t figured out how to further filter out non-streets… I also don’t want to filter too much - like I mentioned that in my previous post about how some things that are ‘highway’ aren’t tagged that way.

I’m at least assuming this is possible, because it’s definitely possible for a city to exist in OSM but not be accessible for my needs. An example of this is the recently requested city Iași, Romania. You can see it in Google Maps:

But if you search for it via Nominatim, you get two results…

  1. City: http://nominatim.openstreetmap.org/details.php?place_id=319899
  2. State: http://nominatim.openstreetmap.org/details.php?place_id=127949085

I can’t use the city version, because it’s a ‘node’. If you search ‘OSM:’ on that page, you’ll see where it notes that.
If you search ‘OSM:’ on the state version, you can see how it’s a ‘relation’. That’s the only type of data I can use from OSM, because that’s the only type that holds the street data - I’m not aware of a way to search for streets within a node.

So, my logic is that if entire cities aren’t completely categorized then it’s probably pretty likely that there are streets that aren’t categorized correctly as well.

That’s why I collect ‘too much’ (streets, as well as things that might not be streets) … I figure I might be collecting data that isn’t properly categorized as a street - stuff that I wouldn’t have received if I had a tighter query that was ignoring things that are definitely not streets.

There’s a lot that I don’t know about OSM, though. Maybe there’s a better answer out there that I’m not aware of yet. :confounded:


#12

In my local areas lots of stuff is in need of removal. For example, station platforms, airport runways and similar stuff is included in the list of streets.

Two things would help

  1. Ability to flag multiple streets at once.
  2. Ability to hide/mark streets already flagged by me/another user.

#13

I agree with both of those suggestions.

More suggestions:
3. Mark on the street that it has been flagged. (I have flagged the same street multiple times because I didn’t realize it was already flagged.
4. Use the power of Wiki to allow the community to remove objects.


#14

Has there been any progress on this moderation system front? I have flagged a lot of features near me, quite a few of them several months ago, without any change so far. It would be nice to finally reach 100% in some of the places near me.

I am imagining an ever growing queue of flagged objects that only you can go through and it sounds like a hellish job.


#15

Haha! Yeah, you nailed it. I can go in there once in a while and handle a number of flagged streets but I can’t keep up.

I would like to build in some way for a community flagging system, so that any member can flag streets and view the list of flagged streets to approve/deny their flag.

So one person could flag a street and if X number of people approved the flag, it would be deleted & if X number of people denied the flag, it would be un-flagged.

Do you think anyone should be able to have a voice, or do you think there should be levels of membership that allow people to have yay/nay voting rights?


#16

This feels like something that requires different levels. I would never trust the internet in general. It would not take much to create x accounts and start flagging and removing several streets. Of course, deciding on “valuable” members can be tricky in and off itself.


#17

First I think we need to decide on a fixed set of rules for flagging rather than relying on each person’s individual opinion. I know personally that I think certain things like parks and interesting landmarks should be kept in, assuming it is possible to ‘run’ them. The nodes might have to be manually moved. Others in the community might disagree though.

Once the rules are decided, there should probably be designated ‘moderators’ that have the ability to modify/remove streets. Ideally these would be experienced users who understand the system and are willing to follow the rules that the community agrees on. The moderator should also only have permission for his/her own cities. In other online forums, a moderator is often selected after demonstrating some sort of level of participation/dedication to the site.

I also agree that it would be helpful to see if a street has been flagged and if a comment has been left on the flag. Then everyone will have more information and can provide additional comments that agree or disagree with the original flagging. After that dialogue occurs, the moderator can make the final decision that is consistent with the rules that we have agreed upon.

A more advanced feature would be to allow the community to see the decisions that the moderator has made and offer critiques. That evidence could be used if there are complaints or if things need to be reverted back to a previous version. Additionally, community members could vote yes/no on the decisions that a moderator makes…maybe a 1-week voting period after the moderator submits a change. If there are no votes, the decision would be automatically approved.

Maybe that’s way too complicated but it would certainly be fun if it was implemented.


#18

I think what might be most important for you James is something simple and also not a burden time wise at least to start. Some ideas along those lines:

  1. Limit moderation capabilities to contributors.
  2. In the UI, keep showing features that have been flagged, but separate them and don’t count them towards a percent complete of a city or leaderboard. This way its only a calculation change and less code changed to build something like a moderation system.

Another thought is to build in some obvious pattern matching to exclude things like “Best Western”, “apartment” or things with say “parking lot” in the name and flag those automatically. I think large swathes could be automatically flagged in a systemic manner.


#19

I have been doing a lot of flagging in the cities I have run. There are many commonalities that exist in bad data: “One word”, Starts or ends with the word lake, Ends with Park, contains the word building, “parking lot.” Most of these signals are not 100% but if these patterns are detected, that could fast track them for removal. Company names are common in some cities but not in others – but these are also harder to filter for.

Maybe the easiest system to have a review queue and when 1 (or 2 people) approve removal, it’s gone. This should be restricted to active users who have been on the site for a while.


#20

Oh! Right!
Thanks @brettajohnson & @wetmig … I’m taking your advice & bulk-removing Street records that match certain values, like you’ve noted. I think I’ve already removed 20k+ bad streets. This will help a lot!