Create a tile server with the ability to render different layers

[BudgieInWA]

We want to make it as easy as possible to experience the result of the work done here, so;

Let's create a tile server that can render tiles right in rust or through some other means. That way, anyone can look at it in their tool of choice.

• find a library/mechanism for rendering shapes to raster nicely
• implement rendering osm2streets data to tiles (raster or vector) on request
• set up a tileserver api (WMS, TMS, etc.) that provides access to

In future, we could glue on a "download small region" step which downloads .osm xml from osm.org, or whatever data source you want to plug in. Rendering raster will be useful in JOSM too, so we don't have to do it in Java

[dabreegster]

find a library for rendering shapes to raster nicely

I've not worked with raster tiles much before, but I did briefly experiment with some vector tile formats + renderers back in https://github.com/dabreegster/leaflet-experiments/blob/main/NOTES.md. I was trying to shove the raw geometry from A/B Street -- which has one polygon for every single dashed white line on every street! -- into mbtiles. I remember even small areas were very slow and buggy. I never dug into the issues, or figured out if the vector tile renderers are batching draw calls, using quadtrees, etc.

But anyway, maybe using raster tiles is more compatible with the existing ecosystem and could avoid some of these problems entirely.

[BudgieInWA]

I really like raster tiles as a common denominator output, but that's not to say vector tiles aren't easier to work with sometimes. Maybe vector svg tiles get rasterised by an existing svg rasteriser.

[jakecoppinger]

Sorry, sent from wrong account previously.

If I understand correctly, is this issue for creating raster (or vector) tiles from what the interface of AB Street looks like? I'd find that incredibly useful as an OSM contributor - I'm not aware of any OSM tiles that show the amount of detail.

Any tips on where I could help contribute as someone who only has a little bit of Rust experience?

I was even thinking of running something like Webdriver/Selenium, navigating by adjusting the url (eg: http://play.abstreet.org/0.3.35/osm_viewer.html?system/us/seattle/maps/montlake.bin&--cam={zoom}/{lat}/{lon}) and "screenshotting" tiles.

I suppose I could also run some Typescript running a tileserver "proxy", taking in zoom/lat/long and on the backend navigating the browser to the correct spot and sending a screenshot of the right size back as the response.

I'm sure there must be better ways though!

[dabreegster]

I'm not exactly sure what Ben has in mind for this, but I think the overall goal is to get A/B Street-style detailed rendering everywhere in the world, with low effort for the user. Furthermore, let the result be displayed in the iD editor, on web maplibre maps, in JOSM, QGIS, etc -- anything that understands standard raster or vector tiles.

One approach, tiling the world somehow and pre-importing A/B Street maps for each of the tiles, is not looking easy so far. I'm playing around in acteng/atip-data-prep#1. It also would only solve the first part of the problem, not expressing things as raster or vector tiles.

Another approach is to make the client-side do it. You can go to https://a-b-street.github.io/osm2streets/#18.5/51.50687/-0.13376, import current view, and do everything lazily. This gets osm.xml extract from Overpass, calls a JS function in osm2streets (that ends up calling Rust code in this repo through WASM), and renders the result in Leaflet.

I don't have much familiarity with OSM tileservers. I think some of them can lazily generate raster or vector tiles and cache the results? If somebody requests a missing tile, they could do the same thing (query Overpass, pipe through osm2streets, take the resulting GeoJSON polygons and encode as raster or vector tiles).

Hopefully Ben has a better direction to point you. I don't think unfamiliarity with Rust should be a barrier here; the process of tiling OSM data, or dynamically fetching Overpass and calling osm2streets, or turning GeoJSON into raster or vector tiles, could all happen in any language

[BudgieInWA]

Hi @jakecoppinger, thanks for reaching out!

The idea here is to set up a Tile Server that is capable of rendering osm2streets data. I have updated the initial comment to be a bit clearer.

First, we need an actual Tile Server, TMS, or WMS or something. I don't know much about them, I've only configured things to use them, like this Leaflet tutorial. Then we need to generate the tiles somehow, and we have lots of options.

Using A/B Street

Using A/B Street as the renderer is a neat idea. That way we could have an "A/B Street" layer in other tools that leverages that existing work.

This is the least accessible option I think, for updating the rendering. A lower-overhead and more flexible mechanism for rendering is also desired.

Render in rust

Maybe the existing A/B Street rendering code is a good starting point for an in-rust render? This is not that accessible, because it would require writing rust to improve the rendering.

Use osm2streets output in Another renderer

This is the most tempting option for me. It is also how osm2streets is intended to be used in the long term.

We can implement - in the rust side - whichever geom output you need. osm2streets is already outputting a number of geojson "layers" that the Street Explorer is using.

You could start a new project in your favourite language/environment that renders just the road features on transparent tiles.

Or just straight into integrating the osm2streets road geometry into https://github.com/enzet/map-machine or something

[jakecoppinger]

I just had a good play around with Street Explorer and I think I'm getting a better understanding of it works now.

I can see it'd be possible to pre-generate (slowly!) a vector tileset by:

• Getting an OSM dump from Overpass turbo (maybe a local overpass instance so as not to overload the public server)
• Running this though A/B street to generate GeoJson (JS bindings from Rust on show at street-explorer/www/js/main.js:186 and the lines below)
• Generate vector tileset from this huge GeoJson file using https://github.com/mapbox/tippecanoe (or Mapbox Tiling Service for a quick demo on Mapbox infrastructure)

As far as lazy loading, I don't understanding vector tile schemas enough yet to understand if requests have bounds (like {zoom}/{x}/{y}) in the same way that raster tileservers do. (Edit: Yep, turns out they do work the same: https://docs.mapbox.com/api/maps/vector-tiles/)

In that case I could write a Typescript server that takes vector tile requests and either hits a cache or runs the above steps for the requested bounding box - which will be quicker than an entire city/suburb and scale nicely.

Does this sound like the right track? I'm looking forward to seeing A/B street basemaps everywhere 😀

I'll keep looking into how vector tileservers work and if I start a separate repo I'll make sure to share it back here.

[dabreegster]

Getting an OSM dump from Overpass turbo

The osm.xml can come from anywhere. If you're doing a huge area, I'd download from http://download.geofabrik.de/europe.html and maybe clip to a smaller area with osmium. If your typescript server can access a filesystem and shell out, this could be cheaper than hitting Overpass.

Does this sound like the right track?

It does to me! (Keeping in mind I don't understand tileserver workflow)

[dabreegster]

If you do go with geofabrik + osmium for clipping, I'd recommend something like this: osmium extract -b 2.25,48.81,2.42,48.91 large_area.pbf -o paris.xml -f osm,add_metadata=false
add_metadata strips out author, timestamp, and version from the XML, reducing the file size and making it a bit faster to parse.

We will hit issues at tile boundaries, because osm2streets clips roads crossing the boundary polygon. But as a first pass, we can get started with a tile server and see how things look, then figure out a good fix for this. (Maybe just clipping with a buffer?)

[jakecoppinger]

I've got a proof of concept vector tile server working built in Typescript & Koa! https://github.com/jakecoppinger/osm2streets-vector-tileserver

It:

• takes tileserver requests (eg http://localhost:3000/tile/16/60293/39332)
• Finds the bounding box of that tile
• Downloads the OSM XML from a local overpass turbo instance (instructions included) for that bounding box
• Calls osm2streets via the NodeJS bindings
• Generates the geojson (only the main geometry at the moment, very easy to improve though)
• Caches the output in a JS variable (needs improvement)

PRs are very welcome! I'm going to continue working on it and test it with a frontend soon (I haven't verified the GeoJSON output but it looks right).

[dabreegster]

Woo, this is so awesome to see! I just published https://www.npmjs.com/package/osm2streets-js and got it working from Svelte+Vite today (https://github.com/dabreegster/svelte_playground/blob/main/src/App.svelte). I've started https://github.com/dabreegster/osm2streets-vector-tileserver/tree/use_npm to start using the NPM package and simplify the build process. Hit a few snags, but I'll hopefully have time tomorrow to try again.

[jakecoppinger]

Brilliant! A snag I hit was wasm-pack build --dev --target web ./osm2streets-js caused all sorts of issues (from memory around fetch failing) which I eventually resolved with wasm-pack build --dev --target nodejs ./osm2streets-js.

I assume the npm build would be a frontend (web) build?

[dabreegster]

Yes, it's --target web, because I couldn't get the other options to work. I'm a little lost around all these options. I noticed there's no await init or similar in the controller right now -- is that somehow unnecessary with --target nodejs? That may be the current snag I was hitting