Skip to content
This repository was archived by the owner on Feb 25, 2025. It is now read-only.

Commit 6ef67ae

Browse files
committed
[Impeller] Implement YUV texture import and sampling for video player frames.
This patch rounds out support for importing Android Hardware Buffers as Vulkan images and sampling from them. Not Android Hardware Buffers are in formats that Vulkan (let alone Impeller) understands. Some YUV textures have formats with no equivalent `vk::Format` enum and need extra information on how to sample from them. This patch adds support for correctly importing and sampling from such formats. There are severe restrictions on how sampling from such external formats works. For one, it isn’t possible to assign a combined image sampler in the render pass. The pipeline itself needs to be rebuilt to reference a specially created immutable sampler. This immutable sampler is a combination of the usual information present in an Impeller SamplerDescriptor as well as a custom “conversion” object obtained while importing the Android Hardware Buffer. There is no way to predict what conversion object will be necessary ahead of time as this will depend on the source of the Android Hardware Buffers and is likely different for different video feeds, camera sources, and other Android Hardware Buffer texture sources. To handle this uncertainty, a new pipeline variant with a JIT determined immutable sampler will be hashed and cached before being used in a render pass. The number of pipeline variants created just-in-time will depend on the number of sampler variants used in the render pass to sample from wrapped Image. For instance, specifying a sampler with a different address mode will likely result in a new pipeline variant being created. In most cases however, there will just be one or two additional pipeline variants per application. Impellers sampler diversity is very low with most samplers being the usual NN samplers. It may be possible to preload even this pipeline by trying known conversions. As said previously, there can only be a handful of these conversions. More restrictions on sampling from such images includes being limited to `VK_FILTER_LINEAR` without additional format and extension wrangling and performance penalties. Fixes flutter/flutter#142082.
1 parent 4653bd1 commit 6ef67ae

24 files changed

+1328
-306
lines changed

ci/licenses_golden/licenses_flutter

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -35316,6 +35316,10 @@ ORIGIN: ../../../flutter/impeller/renderer/backend/vulkan/vertex_descriptor_vk.h
3531635316
ORIGIN: ../../../flutter/impeller/renderer/backend/vulkan/vk.h + ../../../flutter/LICENSE
3531735317
ORIGIN: ../../../flutter/impeller/renderer/backend/vulkan/vma.cc + ../../../flutter/LICENSE
3531835318
ORIGIN: ../../../flutter/impeller/renderer/backend/vulkan/vma.h + ../../../flutter/LICENSE
35319+
ORIGIN: ../../../flutter/impeller/renderer/backend/vulkan/yuv_conversion_library_vk.cc + ../../../flutter/LICENSE
35320+
ORIGIN: ../../../flutter/impeller/renderer/backend/vulkan/yuv_conversion_library_vk.h + ../../../flutter/LICENSE
35321+
ORIGIN: ../../../flutter/impeller/renderer/backend/vulkan/yuv_conversion_vk.cc + ../../../flutter/LICENSE
35322+
ORIGIN: ../../../flutter/impeller/renderer/backend/vulkan/yuv_conversion_vk.h + ../../../flutter/LICENSE
3531935323
ORIGIN: ../../../flutter/impeller/renderer/blit_command.cc + ../../../flutter/LICENSE
3532035324
ORIGIN: ../../../flutter/impeller/renderer/blit_command.h + ../../../flutter/LICENSE
3532135325
ORIGIN: ../../../flutter/impeller/renderer/blit_pass.cc + ../../../flutter/LICENSE
@@ -38163,6 +38167,10 @@ FILE: ../../../flutter/impeller/renderer/backend/vulkan/vertex_descriptor_vk.h
3816338167
FILE: ../../../flutter/impeller/renderer/backend/vulkan/vk.h
3816438168
FILE: ../../../flutter/impeller/renderer/backend/vulkan/vma.cc
3816538169
FILE: ../../../flutter/impeller/renderer/backend/vulkan/vma.h
38170+
FILE: ../../../flutter/impeller/renderer/backend/vulkan/yuv_conversion_library_vk.cc
38171+
FILE: ../../../flutter/impeller/renderer/backend/vulkan/yuv_conversion_library_vk.h
38172+
FILE: ../../../flutter/impeller/renderer/backend/vulkan/yuv_conversion_vk.cc
38173+
FILE: ../../../flutter/impeller/renderer/backend/vulkan/yuv_conversion_vk.h
3816638174
FILE: ../../../flutter/impeller/renderer/blit_command.cc
3816738175
FILE: ../../../flutter/impeller/renderer/blit_command.h
3816838176
FILE: ../../../flutter/impeller/renderer/blit_pass.cc

impeller/renderer/backend/vulkan/BUILD.gn

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -115,6 +115,10 @@ impeller_component("vulkan") {
115115
"vk.h",
116116
"vma.cc",
117117
"vma.h",
118+
"yuv_conversion_library_vk.cc",
119+
"yuv_conversion_library_vk.h",
120+
"yuv_conversion_vk.cc",
121+
"yuv_conversion_vk.h",
118122
]
119123

120124
public_deps = [

0 commit comments

Comments
 (0)