Multiple Viewports

[bagnell]

Allow for multiple viewports/camera views in a single canvas.

• One SceneManager containing multiple Scenes
  • SceneManager contains all non-view dependent primitives.
    • Primitives updated once and rendered in each viewport
    • Option on each primitive to hide for any number of viewports
  • Each Scene contains a copy of view dependent primitives.
    • Includes Globe
      • All globes search terrain geometry and texture cache before requests
      • All globes have the same terrain and imagery providers?
      • Allow globes to have different terrain and imagery providers?
      • Allow for different celestial bodies? (not implemented)
    • Includes 3D Tilesets
      • All tilesets search a geometry and texture cache before requests
      • Allow different styling?
        • Each tileset will have its own style and batch table
    • Any other primitive where each vertex needs to be multiplied by a view dependent transform on the CPU.
• One Scene with a list of Viewports that contain a camera and viewport rectangle.
  • Non-view dependent primitives updated and rendered in each viewport.
  • View dependent primitives can keep a command list per-viewport after updating for each camera
    • Can not have different terrain or imagery provider
    • Can not style 3D Tilesets per viewport.

The first option is more flexible but more complex. The second option is simpler but has a couple of constraints.

• Z-order the viewports. Model after PrimitiveCollection.
• ScreenSpaceCameraController needs to know which viewport the input originated in.
• Viewport positioning.
  • Absolute values. App responsible for resizes
  • Custom positioning implementation
  • Position with CSS and hidden divs. Potentially expensive.
• Screen space passes for each viewport and for the final framebuffer.
• Allow the same primitive to have a different material in each viewport? (probably not)

[bagnell]

We have an overlay pass. Do we want to execute those on the entire canvas or each viewport? It's only used for the shadow map debug views and ViewportQuad which isn't used anywhere. Can we remove it?

Where do we put the credits by default?

[pjcozzi]

CC #3001

[bagnell]

Having different scene modes will be possible with the first option. Would it be possible with the second?

Are we maintaining backwards compatibility? It would be difficult with the first option, probably easy for the second option.

[bagnell]

All of the widgets created by Viewer are constructed with a Scene. Which viewport should they use? If there is more than one viewport, should we disable them by default? Should we require explicit creation with a scene/viewport reference?

[yoyomule]

What is the current progress?

[pjcozzi]

@yoyomule there is initial work in the viewports branch; however, it requires pretty significant changes to the core rendering engine so we couldn't justify the effort at the time. We'll revisit, but the timeline is TBA. You are welcome to pick up this work sooner if you are interested, see CONTRIBUTING.md.

[lilleyse]

Requested on the forum: https://groups.google.com/forum/#!topic/cesium-dev/vNyg1-fVvfw

[pjcozzi]

Partial support added in #6958.

[OmarShehata]

Requested again on the forum: https://groups.google.com/d/msg/cesium-dev/SUqz4UGrSMA/F-l1ZuRTBwAJ

There is a common need for monitoring system:
Users want to see more then one screen at the same time and each screen show a map in 3D or 2D and display diffrent images.
For example,the first shows satellite images in 3D ,the second shows road map in 2D, .....
They expect the multiple screens show the same data and synchronized.
The data maybe come from server or mouse ploting.

[DoubleYellowEgg]

What is the current progress?
Is there no follow-up to this question?

[ggetz]

Here's a use case on the forum where a user would like to use one DataSourceCollection (i.e. list of entities) across multple views: https://community.cesium.com/t/shareddatasources/35455/3

[mxfh]

Specifically one would like to control the culling of tileset a bit more flexibly.
Or have control over what tiles to (pre-)load by adding some unused camera focussing the areas of interest.

Memory is costly here, so adding a second viewer is not quite the option and not helping with the culling issue at all.

Out-of-frustum tiles get unloaded way to early and can cause faulty/flickering shadows in the scene.
See unity support for this as example:

CesiumGS/cesium-unity#421

and a discussion on Unity:
https://community.cesium.com/t/increase-frustrum-culling-tolerance/31570/5?u=mxfh

I'm not aware of a similar issue for JS, but having more control of Tileset culling and loading could certainly be welcome. This also affects Terrain Tiles to some lesser extent.

[ggetz]

Also brought up in #12553.

[pmconne]

iTwin.js solves this by having a single WebGL context. The render system object creates a HTMLCanvasElement and from it obtains the WebGL context.
Each Viewport (roughly equivalent to a CesiumJS Scene) has its own canvas, and a RenderTarget responsible for rendering the viewport's WebGL content.
The target's _beginPaint function ensures the system's canvas is at least as big as the viewport, then renders the viewport's content to the system's canvas.
The target's _endPaint function blits the resultant image from the system canvas to the viewport's canvas.

Back in 2017 when we implemented this, we observed that the blit operation could be slow in non-Chromium browsers, so we implemented an optimization that draws the WebGL content directly to the screen if only one viewport is in use, skipping the blit. We suspect (hope?) that this optimization is no longer necessary, but need to test.

All WebGL resources for things like tilesets, decoration graphics, map imagery textures, etc are allocated from the shared WebGL context. They can be used by any number of viewports. Their lifetimes are tracked independently of the viewports and they are deterministically freed when no longer in use.

[ggetz]

While I'm not sure it'll be applicable to all of the desired use cases, a popular approach appears to be simulating multiple canvases with one canvas, viewports, and scissor tests.

• Example with three.js

[kring]

A few notes on implementing 3D Tiles and terrain selection for multiple views, based on having implemented a couple of different versions of this in cesium-native...

There are two broad ways to approach it.

Select a single set of tiles for all views

One is to run the tile selection algorithm once, producing a single set of tiles to display in all of the views. With this approach, a tile that is visible in any view is not culled, and the refinement decision is based on the worst SSE in any view in which the tile is visible. We've had this version in cesium-native for awhile now, and it's simpler in many ways.

The major disadvantage is you may end up rendering too much detail in some views, because that detail is needed in other views, causing aliasing artifacts or lower performance. Make sure you avoid the silly bug we currently have in cesium-native where SSE is computed for all views, not just the ones where the tile is visible, because that can fail badly in some corner cases. For example, consider two cameras at the same location but one with a very narrow field of view and one with a very wide one. This configuration will end up loading detail suitable for the narrow FOV for the entire area visible in the wide FOV.

Select an independent set of tiles for each view

The other approach is to run the tile selection algorithm separately for each view, selecting a completely separate set of tiles for each view. This is slower, because it requires traversing the BVH multiple times (or at least, any way I've thought of to avoid this would add a lot of complexity and probably not significantly improve performance). And you'll need to make sure that the selected set of tiles is only rendered in the view to which it applies, which is tricky in Unreal and Unity, but probably less so when you have full control of the rendering engine as CesiumJS does. The total number of tiles to be loaded is also higher in this approach, too, at least to the extent you're able to skip lower levels of detail.

We've recently implemented this approach in cesium-native in CesiumGS/cesium-native#1125. The major challenge was that the tile selection algorithm previously stored a "last selection result" in the Tile, and used it to drive selection decisions in the next render frame. I'm not sure if CesiumJS has a similar situation in its 3D Tiles selection algorithm, but I know it does in the terrain selection algorithm. The obvious approach of storing a map(View -> State) on each Tile instead works just fine, but we saw about a 20% increase in selection time!

So instead we use this new class called TreeTraversalState, which mantains per-view states in an array parallel to the BVH traversal order. See the class here:
https://github.com/CesiumGS/cesium-native/blob/multiple-views/CesiumUtility/include/CesiumUtility/TreeTraversalState.h
(this link maybe change to https://github.com/CesiumGS/cesium-native/blob/main/CesiumUtility/include/CesiumUtility/TreeTraversalState.h in the near future)

There's some explanation of how it works here:
CesiumGS/cesium-native#1129

In cesium-native now, we actually support both of these broad approaches. There are multiple independent "view groups", each of which contains one or more views that select tiles together.

Ownership and Lifetime

One part of this - especially the second approach - that was hard in cesium-native was tracking ownership and lifetime of Tile instances. When can we unload a Tile's content? When can we destroy the actual instance?

Some of this is likely much easier in CesiumJS thanks to the garbage collector. But CesiumJS still unloads Tile content explicitly, so it needs to decide when to do so. In cesium-native, we wanted to allow the different view groups to tick independently. One might update at 60 FPS, while another only updates rarely. So we can't take a shortcut based on frames, such as, "if a Tile was used last frame, we need to keep it, and when unloading we should first unload the Tile that was last used in the oldest frame."

In cesium-native we solved this by reference counting. Every view group holds a reference to each Tile instance that it traverses. When that reference count goes to zero, the Tile is added to the tail of a linked list of unused tiles. When the reference count goes back to 1, it is removed from that linked list. At any time, we can unload all of the tiles in the linked list, as the ones closest to the head are the last-recently-used.

Except it's more complicated than that. cesium-native now also supports unloading entire unused external tilesets and their corresponding Tile subtrees. For external tileset content to be unloadable, all tiles in its subtree must first be unloaded, which means that all the subtree content can not be referenced.

So here's the complicated (but effective and efficient) scheme:

1. View groups add references to the Tiles they traverse. Height queries also add references to tiles that need to be loaded for height sampling.
2. Another reference is added to a Tile when its content is loaded, whether any view groups are currently using that tile or not.
3. When a tile's reference count goes from zero to one, it also adds a reference to its parent tile, which will keep the subtree from unloading. When a tile's reference count goes from one to zero, the parent reference is removed.
4. A normal, non-external tileset tile's content is unloadable when the reference count is equal to the number of children with non-zero reference counts, plus the reference for the content itself. In another words, we subtract out references added for (2) and (3), and if the count is still greater than zero it means the extra counts came from (1) and so we can't unload yet.
5. An external tileset tile's content is unloadable when the tile's reference count is 1, meaning that only the referenced added in (2) remains.

Load Prioritization

When there are multiple views, each of which want to load different tiles, how do we decide which tiles to load first?

At first, I thought we would need to come up with a prioritization metric that works across views. That would probably mean it is a function of SSE or something, not the distance-based priority metric we use in cesium-native (and CesiumJS, too, I think).

Instead, we've decided to select tiles to load using a weighted round-robin algorithm. Each view has a "weight" which is initially 1.0 and can be configured by the user. When all views have equal weight, they get an equal chance to select the next tile to load. A view with double the weight of the others would get double the number of chances.

I think this turned out to be pretty elegant and definitely recommend you use it (unless you have a better idea of course!).
https://github.com/CesiumGS/cesium-native/blob/34dbe5edf6e6238106be44be8cd134ea07d85163/Cesium3DTilesSelection/src/TilesetContentManager.cpp#L1523