Camera view poor performance: image flickering (side boundaries/size change)

[nunoguedelha]

The video stream visualized in the OpenMCT canvas is often flickering. Refer to https://github.com/ami-iit/component_ANA-Avatar-XPRIZE/issues/403. Actually we can notice that the flickering is due to the image side boundaries quickly switching, i.e. the image is alternating between a shrink and stretched size fitting the canvas:

canvasImageFlickering.mp4

Fixed by #110 and ami-iit/openmct#2.

[nunoguedelha]

By running the yarpview with the "--autosize" option,

yarpview --synch --autosize --keep-above

...and connecting it to the grabber output port /icubSim/camLeftEye,

yarp connect /icubSim/camLeftEye /yarpview/img:i

...we can see that on the OpenMCT image canvas, the displayed image is initially stretched to fit the canvas container, then regains its original aspect ratio. This behavior creates the flickering impression.

canvasImageFlickering-stretch-to-fit-issue.mp4

Analysing the resizing of the displayed frames

The resizing and other render functions are implemented in the Vue.js files like https://github.com/nasa/openmct/blob/master/src/plugins/imagery/components/ImageryTimeView.vue and probably more specifically in https://github.com/nasa/openmct/blob/master/src/plugins/imagery/components/ImageryTimeView.vue#L129:

export default {
    ...
    methods: {
        resize() {
            let clientWidth = this.getClientWidth();
            if (clientWidth !== this.width) {
                this.setDimensions();
                this.updateViewBounds();
            }
        },
        ...

Analysing...

[nunoguedelha]

Function Call Sequence Leading to the Issue

Here's the sequence of function calls and events leading to the issue occurrence, as soon as we click on the "Camera view left eye" telemetry entry

----------> New image Im[i+1] received

=> directPort.onRead(...) callback executed (the callback defined and added through RealtimeTelemetryPlugin()).
   => listener["sens.camLeftEye"](point);
      => VueComponent.imageHistory.push(image)
         - // received imaged pushed to history buffer.

ImageryView.vue/default.
  => watch.imageHistorySize(newSize, oldSize)
     - // This watcher is triggered by `VueComponent.imageHistory` size change (watchers are run by `Watcher.prototype.run`)
     - // It updates the focused image index and scroll strip possible.
  => watch.focusedImageIndex()
     => VueComponent.getImageNaturalDimensions()
        - // sets VueComponent.focusedImageNaturalAspectRatio = undefined`
  => computed.sizedImageDimensions()
     - // Supposed to set `sizedImageDimensions` width and height in order to fit the container dimensions 
       // (VueComponent.imageContainerWidth, VueComponent.imageContainerHeight) at best without changing 
       // the aspect ratio.
     - // Because of undefined `VueComponent.focusedImageNaturalAspectRatio`, `sizedImageDimensions.width` 
       // set to `VueComponent.imageContainerWidth`.

----------> Previously displayed image Im[i] is stretched and fits the canvas width.

----------> New image Im[i+1] is loaded.

  => // Async call from "load" event -> sets VueComponent.focusedImageNaturalAspectRatio to image value.

  => computed.sizedImageDimensions()
     - // Sets `sizedImageDimensions` width and height in order to fit the container dimensions (height in our case).

----------> Image Im[i+1] is displayed.

---

----------> New image received

=> directPort.onRead(...) callback executed (the callback defined and added through RealtimeTelemetryPlugin()).
   ...

[nunoguedelha]

Fix

VueComponent.focusedImageNaturalAspectRatio is initialized to undefined in https://github.com/nasa/openmct/blob/ab7e2c57470d3c6076459ad7e3eaa1883cc3b8b0/src/plugins/imagery/components/ImageryView.vue#L234

    data() {
        let timeSystem = this.openmct.time.timeSystem();
        this.metadata = {};
        this.requestCount = 0;

        return {
            ...
            focusedImageNaturalAspectRatio: undefined,
            ...
        };
    },

If we remove line https://github.com/nasa/openmct/blob/ab7e2c57470d3c6076459ad7e3eaa1883cc3b8b0/src/plugins/imagery/components/ImageryView.vue#L805, the aspect ratio this.focusedImageNaturalAspectRatio is not re-initialized to undefined before the call to computed.sizedImageDimensions(), instead, the previous image aspect ratio is used.

        getImageNaturalDimensions() {
            this.focusedImageNaturalAspectRatio = undefined; <----- TO BE REMOVED

            const img = this.$refs.focusedImage;
            if (!img) {
                return;
            }
            ...

[nunoguedelha]

Respective Issue opened in https://github.com/ami-iit/openmct: ami-iit/openmct#1.

the fix here is to point to a branch in the fork https://github.com/ami-iit/openmct tracking the changes on top of tag 1.7.8 which was the selected openmct version selected so far.

[nunoguedelha]

Fixed by #110 and ami-iit/openmct#2.