Please add support for HDR JPG gain maps

[gregbenz]

Feature request

What are you trying to achieve?

Add support for transcoding JPG gain maps. These images enable the ability to share images edited for modern displays that support HDR (high dynamic range). And compared to HDR-only encoding, using a gain map allows the image to be properly adapted to any display which does not support the full HDR encoding. That adaptation is critical to being able to safely share HDR images on any display - including SDR ("standard dynamic range") and monitors with limited HDR capabilities.

These HDR displays are already more common than generally known. Roughly 80% of viewers on Instagram already have HDR support and you can see examples of HDR vs standard images in my Instagram feed in the IG phone app (or using Chrome with an M1+ MacBook Pro).

When you searched for similar feature requests, what did you find that might be related?

There is a request for JXL support (#4263). While that is a great format, JPG (and AVIF) with a gain map is more practical. The JPG format is 100% backwards compatible and already widely supported. The AVIF gain map encoding will be very important and already has support in Adobe software and Chrome. We're probably a year away from having enough critical support to use AVIF, but there is clear and steady progress towards that point. There is little support for JXL and it is unclear if it will gain acceptance on the web (especially as Chrome removed support).

What would you expect the API to look like?

Should support simple transcoding of JPG (crop, resize, compress) while retaining the HDR. That means the base image and auxiliary gain map should be edited in a consistent manner, and the HDR metadata should be retained.

The Google-backed open-source libultrahdr library already supports all of this. All that is needed is some integration with libvips.

What alternatives have you considered?

There is a request for libvips to add support: libvips/libvips#3799

Please provide sample image(s) that help explain this feature

More info on HDR and gain maps.

Here is a sample image. The image will tell you if you are viewing just the SDR base image, or viewing as HDR (when using a supported browser and display, you should see blue text). The image is encoded with both the ISO and Android XMP standards (the ISO encoding is really all that is needed, the XMP encoding is redundant for backwards compatibility, but won't be important for too long).

Developer resources for gain maps: https://gregbenzphotography.com/hdr/#developers

[lovell]

This will be dependent on libvips/libvips#3799

[gregbenz]

@lovell libvips has added initial support for libultrahdr: libvips/libvips#4645

This is built on mainline libultrahdr which supports transcoding of JPG (there is ongoing work to add support for AVIF / HEIF to libultrahdr).

[lovell]

Proposed new API to expose JPEG gain maps (via libultrahdr) now that libvips v8.18.0 is available.

withGainMap()

Convert pixel values to HDR via .pipelineColourspace('scrgb') using the input gain map, if any, and then remove it. For JPEG output it also sets .toColourspace('scrgb') to force the creation of a new gain map.

keepGainMap()

Open input with gain map attached, process main image and gain map separately on a best-effort basis. Many operations might invalidate/remove the gain map, these will generate a warning and switch to withGainMap() behaviour.

The list of operations I'd like to support where this will "just work" are:

• resize
• extract
• trim
• rotate
• flip/flop
• auto-orient

Notes:

• For output JPEG images with a gain map (reused or regenerated), libultrahdr will ignore all JPEG output properties apart from quality. The prebuilt binaries include mozjpeg so this will be the same as mozjpeg: true, with no further control.
• libultrahdr appears to generate/attach gain maps using ISO 21496-1 format, which I think is via an APP2 MPF marker. I assume this is suitable for web browsers.
• Verify tiled output via dzsave, which has some built-in gain map logic.
• Determine which gain map properties, e.g. content boost, should be exposed via metadata().
• Determine which of the same gain map properties can be updated, if any.

[gregbenz]

@lovell Can you confirm that keepGainMap() would be the default behavior?

Many developers aren't yet familiar with gain maps, lack an HDR display for visual confirmation of results, or fail to use gain map source images in validation testing. So opting in to keepGainMap would likely result in a large number of transcodes defaulting to significant loss of quality (no HDR) without this being a deliberate choice by the developer.

I believe keeping the map should be the default when the following are true:

• source is encoded with a supported gain map format.
• requested output is a file format which supports gain maps (JPG for now, but AVIF / HEIF support is coming to libultra)
• operation is from your supported list (resize, extract, trim, rotate, flip, auto-orient), as these do not request deliberate alterations of the qualitative image content (unlike say applying a filter or some effect to change the image).