Map translation

[tbodt]

What would it take to support rendering the map in any language which has translations in OSM?

[tbodt]

I see planetiler has a default list of languages: https://github.com/onthegomap/planetiler/blob/169627dea9b024f4b64f53039c302778c1c273bf/planetiler-core/src/main/java/com/onthegomap/planetiler/Planetiler.java#L106. This unfortunately doesn't include the language I want to use. I can imagine including all the languages would mean significantly bigger tile files...

[hyperknot]

Luckily it's not related to planetiler and is very simple to achieve.

1. Download the JSON from the style you prefer, for example this JSON:
   https://tiles.openfreemap.org/styles/liberty

2. Do a search and replace for "name_en" with "name_de" or whichever you prefer. Upload the JSON somewhere and simply point your style to this JSON instead of the default ones.

A dynamic JS snippet which does this is the following:

for (const layer of style.layers) {
  if (!layer.layout) continue

  const textField = layer.layout['text-field']
  if (!textField) continue

  // highway numbers, etc.
  if (isEqual(textField, ['to-string', ['get', 'ref']])) continue

  const id = layer.id

  let separator
  if (id.includes('line') || id.includes('highway')) {
    separator = ' '
  } else {
    separator = '\n'
  }

  const parts = [
      ['get', `name_${langCode}`],
      ['get', `name:${langCode}`],
      ['get', 'name'],
  ]

  layer.layout['text-field'] = [
    'case',
    ['has', 'name:nonlatin'],
    ['concat', ['get', 'name:latin'], separator, ['get', 'name:nonlatin']],
    ['coalesce', ...parts],
  ]
}

[tbodt]

This works decently well, but there are two problems

• This way of fiddling with the style JSON seems very brittle, surely it would be best for there to be some way for the style to like include a variable which can be filled in at runtime with the correct language?
• It doesn't include every language that has data in OSM/wikidata. For example: Europe has names in 82 languages on OSM, and 343 on Wikidata, but if you download and parse the PBF file for 0/0/0 you only find 76...and somehow a bunch of these 76 are not on OSM? how did that happen

[hyperknot]

This is basically the official solution. Actually the official solution is way more basic, I've spent a lot of time polishing it till I got to the version I posted. Have a look at the official example:
https://maplibre.org/maplibre-gl-js/docs/examples/language-switch/

About fiddling with JSON, basically all Mapbox/Maplibre styles are just that, JSON. I might set up a nginx function for this, but at the end of the day it'll just be a JSON with strings.

Now, about the mismatch of between OSM and OpenMapTiles, it's outside the scope of this project. You can see what's exactly in the data by going to the inspector mode of Maputnik:
https://maputnik.github.io/editor?style=https://tiles.openfreemap.org/styles/bright

[tbodt]

Thanks for the link to maputnik, it's definitely easier to use than manually deserializing pbfs.

What I don't understand, though, is the difference between OSM and OpenMapTiles, and why they would get out of sync. AIUI this project, OpenFreeMap, runs Planetiler, and then Planetiler fetches its data from OSM to generate all the tiles. Where does OpenMapTiles come in? What actually is OpenMapTiles?

[hyperknot]

Wrote a document for debugging international names:
https://github.com/hyperknot/openfreemap/blob/main/docs/debugging_names.md

[tbodt]

Turns out the list of languages in OpenMapTiles is equivalent to the list of languages in OpenFreeMaps, so that explains where it's coming from. What I don't understand is, why does planetiler use OpenMapTiles and not OSM directly?

[tbodt]

Oh there it is. https://github.com/openmaptiles/planetiler-openmaptiles/blob/5a07ce4ca7593622207cdf8f8330d52d31de8150/src/main/java/org/openmaptiles/generated/OpenMapTilesSchema.java#L62

[tbodt]

Would it be feasible to simply include all languages with data in wikidata in the tileset, or would that cost too much disk space?

[hyperknot]

So for your questions:

Why does planetiler use OpenMapTiles and not OSM directly?

Because OSM is just a database dump, it's not usable on it's own. You need to make a schema which can later describe geometries in the vector tiles. One such schema is OpenMapTiles. Other is https://shortbread-tiles.org/

Would it be feasible to simply include all languages with data in wikidata in the tileset, or would that cost too much disk space?

I don't know the answer for this, you should ask it in the Planetiler repo.

[tbodt]

Makes sense. Going to figure out what would need to change in Planetiler.

[hyperknot]

I've opened a ticket to render a full planet on the other two OpenMapTiles, so we can compare their implementation. This way we can see if something is planetiler specific, or is present in the other two implementation as well.
#25

[tbodt]

My assessment is:

• OpenMapTiles reference implementation has a list of languages defined in openmaptiles.yaml, and a surprising amount of dependencies on that list (e.g. using it to generate a list of columns for SQL statements) that would seem to make it hard to convince it to fetch labels for all languages.
• tilemaker just doesn't fetch labels in any other language by default, and has no support for wikidata - doesn't seem very useful without significant work.
• Planetiler's default task is to run with a configuration that matches OMT reference as closely as possible, apparently implemented as a Java file autogenerated from the aforementioned openmaptiles.yaml, thus inheriting the list of languages. There's no flag that will make it include all languages. But this is implemented by first fetching all labels for all languages from OSM and Wikidata, then filtering them - so it would be easy to make it just not, modulo bikeshedding over command line flags.

I'm currently looking into the feasibility of these Planetiler changes, hopefully I'll be able to do a full planet run to test the difference in output size.

[hyperknot]

Sounds great, thank you for digging into this!

These are the command line options I'm using for planetiler on a 128 GB machine. It takes about 5 hours to run:

openfreemap/modules/tile_gen/tile_gen_lib/planetiler.py

Lines 35 to 55 in f8f46a3

	command = [
	'java',
	f'-Xmx{java_memory_gb}g',
	'-jar',
	config.planetiler_path,
	f'--area={area}',
	'--download',
	'--download-threads=10',
	'--download-chunk-size-mb=1000',
	'--fetch-wikidata',
	'--output=tiles.mbtiles',
	'--storage=mmap',
	'--force',
	]
	if area == 'planet':
	command.append('--nodemap-type=array')
	command.append('--bounds=planet')
	if area == 'monaco':
	command.append('--nodemap-type=sortedtable')

[tbodt]

Looks like I can do a run on my 8GB machine in about 20 hours. Surprisingly this is fast enough for me for the moment since I have other things to do.