cv::cvtColor does not properly take transparency into account

[PhilLab]

System information (version)

• OpenCV => 3.2
• Operating System / Platform => Windows 64 Bit
• Compiler => Visual Studio 2013

Detailed description

cv::cvtColor(image, outImage, cv::COLOR_BGRA2GRAY`); and cv::cvtColor(image, outImage, cv::COLOR_BGRA2BGR); don't handle transparency well: RGB pixel values determine the output even if alpha value is zero.

One could indeed argue that its because of a "wrong" image but this seems to be what many standard editing tools / conversion tools produce

Steps to reproduce

Use this test image at path: PngWithTransparency.png

cv::Mat image= cv::imread(path, cv::IMREAD_UNCHANGED);
cv::Mat noAlpha, noAlphaGray;
cv::cvtColor(image, noAlpha, cv::COLOR_BGRA2BGR);
cv::cvtColor(image, noAlphaGray, cv::COLOR_BGRA2GRAY);

This is what is inside the images, a black "bounding box" and then fully white border around. Expected would be a full black background.

BTW: cv::imshow("image", image) would produce the same result as noAlpha

[alalek]

Because there is no "transparent" color in CV_8UC1 representation.
During these conversions alpha channel is not used and assumed as 255 always.

[PhilLab]

@alalek Thanks for taking a look at the issue.

However, in CV_8UC1 there is also no "red" color and gray conversion explicitly takes care of that by computing the intensity. So why not taking alpha into account as well?

[mshabunin]

@PhilLab , what about converting to premultiplied alpha (COLOR_RGBA2mRGBA)?

with_alpha = cv2.cvtColor(img, cv2.COLOR_RGBA2mRGBA)
gray = cv2.cvtColor(with_alpha, cv2.COLOR_RGBA2GRAY)

gray_orig = cv2.cvtColor(img, cv2.COLOR_BGRA2GRAY)

[alalek]

Or extract "alpha" channel and use it as mask (if it is binary 0/255).

[PhilLab]

@mshabunin @alalek Thanks for the suggestions, I didn't know of mRGBA 👍 It actually does the job though several times slower than the custom implementation:

cv::Mat_<cv::Vec4b>(image).forEach([&outImage](const cv::Vec4b& pixel, const int pos[]) -> void {
    const float opacity = pixel[3] / 255.f;
    outImage.at<cv::Vec3b>(pos[0], pos[1]) = cv::Vec3b(opacity * pixel[0], opacity * pixel[1], opacity * pixel[2]);
});

For 100 randomly generated images of sizes 1x1 to 4000x4000, this takes around 600ms instead of 2000ms when using the double pass OpenCV.

Would there be a rationale for adding something like COLOR_RGBA2mRGB and COLOR_RGBA2mGRAY with better implementations than my naive one? I am unsure of the correct placement of the m in the name, though.

[PhilLab]

Just for the record, should we ever decide to implement it (and I understand the arguments why we wouldn't do it), here are two test images I used for testing our own implementation:

• SanitizedGradientWithTransparency.png, which has no RGB values for pixels with alpha==0. With the current OpenCV implementation, this results in a binary mask interpretation, like alalek said:
  
• GradientWithTransparency.png where all pixels have RGB values, even if alpha == 0. With the current OpenCV implementation, this results in this image:
  

The expected outcome for both PNGs would be:

[mshabunin]

@PhilLab , I think something like COLOR_BGRA2mBGR can be easily implemented for now.

But I believe that advanced transformations like BGRA2Gray should use different approaches: it can be different cvtColor interface with alpha-channels and nonlinearity in mind or maybe one can utilize G-API module to build complex operations from several basic transformations with tiling optimization.

[wangqr]

If a specific background color is needed, why not just alpha blend the image onto the solid color background (addWeighted, or multiply RGB channels with alpha if the background is black) before resizing?

[kinchungwong]

@mshabunin @alalek

I have written proof-of-concept code for this feature request.

Link: https://gist.github.com/kinchungwong/df3963f09391474e1dabeebe1031ba75

The proof-of-concept code uses the following techniques:

• CPU-based SIMD
• OpenCV universal wide intrinsics
• Strip-based parallel (multi-core) processing (code updated 2018-12-07)
• Tested with baseline SSSE3 (128-bit vectors) and AVX2 (256-bit vectors) on x86-64
• Some extra intrinsics utility functions that were found to be useful (to reduce the verbosity of higher-level code)

The proof-of-concept code does not implement, but leaves open the possibility of:

• Strip-based parallel (multi-core) processing (code updated 2018-12-07)
• Error handling consistent with OpenCV code contribution guideline

The proof-of-concept code highlight the following pain points:

• Writing mixed-precision SIMD code (in which numeric data need to be converted to a wider type for arithmetic purpose) is painful and always ends up with difficult-to-read code. The universal wide intrinsics can only prevent redoing this for each architecture, but does not ease the mixed-precision pain.

This code is intended for discussion with OpenCV's development team. It is not intended for use by end-users of OpenCV. The code does not implement error handling yet. Refer to disclaimers at the top of the proof-of-concept code.

My benchmarking shows that it is much faster than other CPU-based approaches, even though I didn't implement parallel (multi-core) processing yet. I tested split and merge, BGRA2mBGRA followed by dropping alpha channel, one-pixel-at-a-time functor by @PhilLab, and GAPI. The benchmarking code is messy so I don't intend to share it. I did not compare performance with GPU-based approaches, since I don't have a machine setup that allows me to do this reliably.

---

Edited 2018-12-07

I just learned about a pending pull request #13379 by @savuor that works on the same area ("color_rgb.cpp"). I'll need to study the code to see how to integrate both contributions.

---

Remark

The color conversion code mentioned by @mshabunin can be found in OpenCV 4.0 (C++) as cv::ColorConversionCodes::COLOR_RGBA2mRGBA. Note that it is in an inner namespace, not in the main OpenCV namespace. Example usage:

cv::Mat matInput;
cv::Mat tempMBgra;
cv::Mat outputBgr;

cv::cvtColor(matInput, tempMBgra, cv::ColorConversionCodes::COLOR_RGBA2mRGBA);
cv::cvtColor(tempMBgra, outputBgr, cv::ColorConversionCodes::COLOR_BGRA2BGR);

[savuor]

@kinchungwong In my PR I'm implementing mRGBA conversions in wide intrinsics right now (but for the only supported data type now which is 8U), hope this helps.