Uniformize negative padding values support between PIL and Tensor

[vfdev-5]

🚀 Feature

Currently, negative padding values for functional.pad are not supported for any kind of padding mode and data type

import torch
import torchvision
from torchvision.transforms.functional import pad, to_pil_image

print(torch.__version__, torchvision.__version__)

x = torch.randint(0, 256, size=(3, 32, 32), dtype=torch.uint8)
x_pil = to_pil_image(x)
padding = (-1, -2, -3, -4)

for m in ["constant", "edge", "reflect"]:
    try:
        pad(x, padding, padding_mode=m)
    except ValueError:
        print("Tensor: Failed with ", m)

for m in ["constant", "edge", "reflect", "symmetric"]:
    try:
        pad(x_pil, padding, padding_mode=m)
    except ValueError:
        print("PIL: Failed with ", m)

> 1.7.0.dev20200701 0.8.0.dev20200701
> PIL: Failed with  edge
> PIL: Failed with  reflect
> PIL: Failed with  symmetric

This is due to np.pad used internally for PIL input.

Motivation

It would be better to provide uniform behaviour for pad with respect of input data type and negative padding values:

• raise an error
• support it in the cases

[jamt9000]

I could try this.

Potential ways to make it work for PIL are

• Use slicing to "unpad" the ndarray for the negative values https://stackoverflow.com/a/24806313
• The same but use PIL .crop() once it has become an Image again
• Convert to tensor instead of ndarray and re-use the tensor implementation (but I suppose that would not support "symmetric"?)

Any reasons to prefer either one?

[jamt9000]

These verbose but hopefully correct changes would make it work for the slicing approach: (edit: would also need to do the same for palette mode which is handled further up in the file)

diff --git a/torchvision/transforms/functional_pil.py b/torchvision/transforms/functional_pil.py
index 693be83..acb09f2 100644
--- a/torchvision/transforms/functional_pil.py
+++ b/torchvision/transforms/functional_pil.py
@@ -339,6 +339,17 @@ def pad(img, padding, fill=0, padding_mode="constant"):
             return img
 
         img = np.asarray(img)
+
+        crop_top = -min(pad_top, 0)
+        crop_bottom = -min(pad_bottom, 0)
+        crop_left = -min(pad_left, 0)
+        crop_right = -min(pad_right, 0)
+
+        pad_top = max(pad_top, 0)
+        pad_bottom = max(pad_bottom, 0)
+        pad_left = max(pad_left, 0)
+        pad_right = max(pad_right, 0)
+
         # RGB image
         if len(img.shape) == 3:
             img = np.pad(img, ((pad_top, pad_bottom), (pad_left, pad_right), (0, 0)), padding_mode)
@@ -346,7 +357,7 @@ def pad(img, padding, fill=0, padding_mode="constant"):
         if len(img.shape) == 2:
             img = np.pad(img, ((pad_top, pad_bottom), (pad_left, pad_right)), padding_mode)
 
-        return Image.fromarray(img)
+        return Image.fromarray(img[crop_top:img.shape[0] - crop_bottom, crop_left:img.shape[1] - crop_right])
 
 
 @torch.jit.unused

(Or with .crop(): Image.fromarray(img).crop((crop_left, crop_top, img.shape[1] - crop_right, img.shape[0] - crop_bottom)))

Modified test script

import torch
import torchvision
from torchvision.transforms.functional import pad, to_pil_image
import numpy as np

print(torch.__version__, torchvision.__version__)

x = torch.randint(0, 256, size=(3, 32, 32), dtype=torch.uint8)
x_pil = to_pil_image(x)

padding_examples = [(-1, -2, -3, -4),
                    (1, 2, 3, 4),
                    (-1, 2, -3, 4),
                    (1, 2, -3, -4),
                    (1, 2, 3, -4)]

for padding in padding_examples:
    print('Test', padding)

    refs = {}

    for m in ["constant", "edge", "reflect"]:
        try:
            refs[m] = pad(x, padding, padding_mode=m)
        except ValueError:
            print("Tensor: Failed with ", m)

    for m in ["constant", "edge", "reflect", "symmetric"]:
        try:
            y = pad(x_pil, padding, padding_mode=m)
            if m in refs:
                y = np.array(y)
                z = np.array(refs[m])
                print(z.shape)
                print(y.shape)
                print(np.allclose(z.transpose(1, 2, 0), y))
        except ValueError:
            print("PIL: Failed with ", m)

[vfdev-5]

@jamt9000 thanks for the suggestions and sorry for late reply. Can we just use PIL .crop for negative values as a preprocessing before padding (if any positive value provided) for PIL input ?
Something like, in 3 steps, 1) we crop for negative values if there are, 2) set negative to 0 and 3) pad for positive values as it is currently implemented. What do you think ?

[vfdev-5]

Closed via #2749 and #2744

[fmassa]

Thanks a lot for the help @jamt9000 and @vfdev-5 !