WIP: Multiscale use-case

[joshmoore]

Motivation

In imaging applications, especially interactive ones, the usability of a data array is greatly increased by having pre-computed sub-resolutions of the array. For example, an array of size (10**5, 10**5) might have halving-steps pre-computed, providing arrays of sizes 5000, 2500, 1250, 625, 312 etc. Users can quickly load a low-resolution representation to choose which regions are worth loading in higher- or even full- resolution. A few examples of this trend in imaging file formats are provided under Related Reading.

The current zarr spec has the following issues when trying to naively specify such sub-resolutions:

• Arrays of differing size can only represent the individual resolution by naming convention
  ("Reslolution_0", "Resolution_1", etc.) This issue exists in a number of existing formats.
• Storing data of differing dimensions in the same chunk is not intended.
• Even if data of differing dimensions (compression)

Generalization

In other domains, a generalization of this functionality might enable "summary data" to be stored,
where along a given dimension a function has been applied, e.g. averaging. This is usually most
beneficial when the function is sufficiently time-costly that its worth trading storage for speed.

Potential implementations

Filter / Memory-layout

Each chunk could be passed to a function which stores or reads the multiscale representation
with a given chunk. (TBD)

Array relationships

Metadata on a given array could specify one or both inheritance relationships to other arrays.
For example, if a child array link to its parent, it might store the following metadata:

{
    "summary_of": {
        "key": "Resolution_0",
        "method": "halving",
        "dimensions": [0, 1]
    }
}

One issue with only having the parent relationship defined is how one determines the lowest
resolution. The child relationships could be represented with:

{
    "summarized_by": [
        {
            "key": "Resolution_1",
            "method": "having",
            "dimensions": [0, 1]
        }, ...

    ]
}

but this would require updating source arrays when creating a summary.

An alternative would be to provide a single source of metadata on the relationships between arrays.

Related reading

• N5MultiScaleSource
• BigDatViewerN5Demo
• Imaris File Format
• http://openmicroscopy.github.io/design/OME005/ (Design discussion in Storage of pyramid data in OME-TIFF ome/design#74)

Possible synonyms / Related concepts

• Global lossy compression
• Progressive compression
• Pyramidal images
• Sub-resolutions
• Summary views

[rabernat]

This is a good idea. Also probably worth looking at Cloud Optimized Geotiff:
https://www.cogeo.org/

COGs store multiscale imagery in a cloud-optimized way.

[forman]

Very good idea!

We started developing a similar feature in our project xcube which provides a CLI command xcube level <zarr-cube-dataset>. It turns Zarr data cubes into a directory containing a spatial multi-resolution pyramid with chunking tailored for image tile processing.

There is also a xcube serve command that starts a WMTS server and a viewer that exploits the leveled datasets: xcube-viewer.

I'm happy to contribute to specs and implementations.

[rabernat]

@forman - xcube looks like a fascinating project, potentially of very broad interest. Could I convince you to give us a brief presentation about it at an upcoming Pangeo weekly call? (http://pangeo.io/meeting-notes.html)

[joshmoore]

I'd be happy to join that as well.

[forman]

@rabernat Sure, thanks! May 29 should fit. Next week we are at the ESA Living Planet Symposium, maybe someone of you guys is there too?

[jakirkham]

cc-ing @jni @sofroniewn, as this issue and xcube may be of interest. 😉

[sofroniewn]

Thanks @jakirkham yes I'm highly interested in a feature like this and have had some chats with @joshmoore about it before, so glad to see an issue was made. The xcube project looks very interesting too, so also thanks for pointing me towards it.

My current use is viewing large multi-resolution pathology images, and as @joshmoore mentioned I had to adopt my own naming conventions for each layer, and saw a large increase in the total file size relative to an optimized tiff format (I was working with data from the Cameylon16 challenge) and visualizing it with napari.

[alimanfoo]

My current use is viewing large multi-resolution pathology images, and as @joshmoore <https://github.com/joshmoore> mentioned I had to adopt my own naming conventions for each layer, and saw a large increase in the total file size relative to an optimized tiff format.

I'd be interested to know if you have any sense of why file size increased, i.e., what is tiff doing differently?

[rabernat]

@forman - thanks for agreeing! I will put you on the agenda for May 29 and follow up closer to the meeting date..

[sofroniewn]

I'd be interested to know if you have any sense of why file size increased,
i.e., what is tiff doing differently?

@alimanfoo I'm not quite sure. I tried to avoid anything foolish, like changing the dtype, but I saw one link imply some of these multiresolution tiffs use jpeg2000 compression which then doesn't quite make it a fair comparison! Also the data from the tiff looks like it has some compression artifacts in it, so they have done something to it at some point.

Here's a link to a google drive that has the tiffs in it and a README.md describing the data. The file training/tumor/tumor_001.tif is a fine example if you wanted to take a look.

I'm really not sure if they have done any clever optimization across layers of the pyramid or if they just treat them all independently and are using the JPEG2000 compression on each one. Sorry for not being more helpful!

[alimanfoo]

I'm really not sure if they have done any clever optimization across layers of the pyramid or if they just treat them all independently and are using the JPEG2000 compression on each one. Sorry for not being more helpful!

No worries, I guess that's what I was wondering. I.e., is the difference just down to different compression codecs, or is there some clever optimisation that leverages the fact that you have multiple resolutions of the same image.

[jakirkham]

This line from Wikipedia suggests JPEG2000 is optimized for handling different scales.

The codestream obtained after compression of an image with JPEG 2000 is scalable in nature, meaning that it can be decoded in a number of ways; for instance, by truncating the codestream at any point, one may obtain a representation of the image at a lower resolution, or signal-to-noise ratio – see scalable compression.

Maybe there is more to the story. That said, a good first step might be to design a codec for working with JPEG 2000 compression.

[alimanfoo]

I was just reading that too :-) From what I can glean, the idea is that you can render a lower res image from the first part of the stream, then increase resolution as you read further into the stream. At least that's part of it. I wonder if you could do something like encode chunks of an array with a JPEG2000 codec, but then have multiple "virtual" zarr arrays which all share the same chunk objects, but which read the chunks to different resolutions. Not sure if that makes sense.

…

On Tue, 14 May 2019, 16:02 jakirkham, ***@***.***> wrote: This line from Wikipedia <https://en.wikipedia.org/wiki/JPEG_2000#Aims_of_the_standard> suggests JPEG2000 is optimized for handling different scales. The codestream obtained after compression of an image with JPEG 2000 is scalable in nature, meaning that it can be decoded in a number of ways; for instance, by truncating the codestream at any point, one may obtain a representation of the image at a lower resolution, or signal-to-noise ratio – see scalable compression. Maybe there is more to the story. That said, a good first step might be to design a codec for working with JPEG 2000 compression. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#23?email_source=notifications&email_token=AAFLYQQQFAUCCO75MXVNATDPVLIANA5CNFSM4HLR5BCKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODVLYVPQ#issuecomment-492276414>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAFLYQV42GNLML6223HEXTLPVLIANANCNFSM4HLR5BCA> .