Update low-level functional transforms with value_range

[vfdev-5]

Following #5500 (comment) we may want to update low-level functional transforms with value_range argument to avoid implicit hard-coded max range definition:

• 255 for uint8
• 1.0 for float

Today this is done for

• adjust_hue
  

  vision/torchvision/transforms/functional_tensor.py

  Lines 210 to 211 in 95d4189

  	if img.dtype == torch.uint8:
  	img = img.to(dtype=torch.float32) / 255.0

• all ops using _blend:
  

  vision/torchvision/transforms/functional_tensor.py

  Lines 259 to 262 in 95d4189

  	def _blend(img1: Tensor, img2: Tensor, ratio: float) -> Tensor:
  	ratio = float(ratio)
  	bound = 1.0 if img1.is_floating_point() else 255.0
  	return (ratio * img1 + (1.0 - ratio) * img2).clamp(0, bound).to(img1.dtype)

We can introduce new argument value_range and use it explicitly for these ops.
In general we can think of value_range as a tuple (min, max) which would cover majority of imagery where channels ranges are similar. There could be however other type of images (e.g. think of non-RGB color spaces or particular imagery) where value ranges could vary per channel, thus we may need to represent value_range as a list of 2-tuples: [(min_1, max_1), (min_2, max_2), ...]

cc @bjuncek @datumbox

[datumbox]

I agree with the idea of passing value_range on our existing kernels. When None we can estimate the bound using our existing approach.

Concerning using single values vs single MinMax vs Multi MinMax, I think all boils down to what is our target. Do we see TorchVision offering native support for Transforms for more exotic non-RGB/Grayscale spaces? Perhaps that could be useful for medical or astronomical applications but we should be careful not to overcomplicate the API with features we won't use. Are transforms that use _blend even likely to be used in such domains?

I would like to get the advise of you and @pmeier on this one for how we proceed.

[NicolasHug]

Apart from making the implicit ranges more explicit, what are other problems that having value_range would address?

I agree that the way we currently handle ranges isn't nice (read: not documented enough). But from my non-informed POV, it looks like adding value_range would add a lot of extra complexity for something that might be solved with better docs / user awareness?

[datumbox]

@NicolasHug Thanks for the feedback, I think we have a description/context gap on this issue. Instead of jumping directly to the problem, we should provide info on what we try to solve and why.

The current API uses ToTensor and PILToTensor to not only convert the image to Tensor but also scale it to [0,1]. This is somehow problematic because the ToTensor does too many things. As a result of this implicit scaling, the low-level kernels try to guess the max pixel value of the image from its type.

Though we will maintain this guessing for BC, providing a range_value to the low-level kernels can free us from using ToTensor. In fact we can just maintain the image in its original space whatever it is and not have to normalize it until the very end when it's thrown to the model. This will make the pictures easier to visualize and combine. It will also remove the need for calling methods like ToTensor, ConvertImageDtype etc.

[pmeier]

Summarizing an internal discussion about this: apart from Keras, all other frameworks also make assumptions on the value range of images. From all the approaches we found, we think our approach that is also used by Albumentations is the most reasonable. We use [0, 1 if dtype.is_floating_point else torch.iinfo(dtype).max]. Thus, although we missed something we probably don't need the explicit value range.

We could provide a utility function that returns the assumed value range for a given dtype. This could also be used for convert_image_dtype and _blend from above.

[datumbox]

If you check our current code you will see a TODO for using iinfo once it becomes JIT-scriptable. Instead of that, we got some methods that try to find the max value. I think we should check if iinfo is scriptable and if it's not perhaps simplify the code.

@vfdev-5 I wonder if you can help us with the above?

[vfdev-5]

no, neither torch.iinfo nor torch.finfo is scriptable, pytorch/pytorch#41492

[datumbox]

@vfdev-5 Thanks for checking. Is it possible to simplify the _max_value() ? Is there are reason why we don't have these pre-estimated and stored in mapping for all the supported types?

[vfdev-5]

I think the reason is that torch script does not support global. We could do however something like:

def convert_image_dtype(image: torch.Tensor, dtype: torch.dtype = torch.float) -> torch.Tensor:

    # ...
    _max_values: Dict[torch.dtype, int] = {
        torch.uint8: 255,
        torch.int8: 127,
        torch.int16: int(2 ** 15),
        torch.int32: int(2 ** 31),
        torch.int64: int(2 ** 63),
        ...
    }

    input_max = _max_values[image.dtype]

@datumbox do you think it worth a change ?

EDIT: updated 256 -> 255

[datumbox]

I think that makes sense. There is typo on the first record for torch.uint8, it should be 255 right?

[pmeier]

For the integer types, there is a -1 missing:

>>> assert torch.iinfo(torch.int16).max == 2**15 - 1

[datumbox]

I think we should close this for now.