Unicode encoding/decoding problems in city names / poly-files

Support
1

Hi there,

I noticed some (German) cities containing unicode characters like “ü” are missing although a *.poly file exist in https://github.com/JamesChevalier/cities.

City missing although *.poly file exists:
example: München.
poly-file: https://github.com/JamesChevalier/cities/blob/master/germany/bayern/m%C3%BCnchen_bayern.poly

City existing as city in Citystrides with 2 *.poly-files
example: Waldmünchen
poly-files:

  1. germany/bayern/waldmünchen_bayern.poly: https://github.com/JamesChevalier/cities/blob/master/germany/bayern/waldmu%CC%88nchen_bayern.poly
  2. germany/bayern/waldmünchen_bayern.poly: https://github.com/JamesChevalier/cities/blob/master/germany/bayern/waldm%C3%BCnchen_bayern.poly

Yes, these are different files as you can see looking at the git-history.
If you type “germany/bayern/waldm” in github File Finder you see both. If you continue typing “germany/bayern/waldmü” one vanishes.

Difference between the 2 poly-files of Waldmünchen:
one escapes the letter “ü” as “c3 bc” (%C3%BC in the URL), the other as “u cc 88” (u%CC%88 in the URL). “c3 bc” is the unicode letter “ü”, “u cc 88” means “u with 2 dots above it”.

My take on those issues:

  • The *.poly-file of München uses “ü” and since München is not available in CityStrides, I guess “u with 2 dots above it” is the encoding that can be interpreted by the CityStrides backend.
  • The file that is shown in github File Finder when you type “germany/bayern/waldmü” is the one with “ü”, so github File Finder seems to only be able to interpret the normal unicode letter “ü”.
  • Although Waldmünchen exists as city in Citystrides it cannot be found on https://citystrides.com/cities by typing “Waldmünchen”.
    You have to stop after “Waldm” and look through the results. So there seems to be another encoding problem as the search on https://citystrides.com/cities can’t interpret any of those encodings

I know unicode encoding is an evil beast but it would be really cool if you could address this issue so cities like München get available in CityStrides.

Greetings
Christian

p.s.: sorry that I had to enter some of the links as preformatted text instead of real links, but as a new user I am not allowed to insert more than 2 links in a post…

2 Likes