Skip to content
This repository was archived by the owner on Feb 25, 2025. It is now read-only.

Made Picture::toImage happen on the IO thread with no need for an onscreen surface. #9813

Merged
merged 10 commits into from
Jul 16, 2019
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion ci/licenses_golden/licenses_flutter
Original file line number Diff line number Diff line change
Expand Up @@ -401,7 +401,6 @@ FILE: ../../../flutter/lib/ui/semantics/semantics_update.cc
FILE: ../../../flutter/lib/ui/semantics/semantics_update.h
FILE: ../../../flutter/lib/ui/semantics/semantics_update_builder.cc
FILE: ../../../flutter/lib/ui/semantics/semantics_update_builder.h
FILE: ../../../flutter/lib/ui/snapshot_delegate.h
FILE: ../../../flutter/lib/ui/text.dart
FILE: ../../../flutter/lib/ui/text/asset_manager_font_provider.cc
FILE: ../../../flutter/lib/ui/text/asset_manager_font_provider.h
Expand Down
1 change: 0 additions & 1 deletion lib/ui/BUILD.gn
Original file line number Diff line number Diff line change
Expand Up @@ -76,7 +76,6 @@ source_set("ui") {
"semantics/semantics_update.h",
"semantics/semantics_update_builder.cc",
"semantics/semantics_update_builder.h",
"snapshot_delegate.h",
"text/asset_manager_font_provider.cc",
"text/asset_manager_font_provider.h",
"text/font_collection.cc",
Expand Down
85 changes: 72 additions & 13 deletions lib/ui/painting/picture.cc
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,11 @@
#include "flutter/lib/ui/painting/picture.h"

#include "flutter/fml/make_copyable.h"
#include "flutter/fml/trace_event.h"
#include "flutter/lib/ui/painting/canvas.h"
#include "flutter/lib/ui/ui_dart_state.h"
#include "third_party/skia/include/core/SkImage.h"
#include "third_party/skia/include/core/SkSurface.h"
#include "third_party/tonic/converter/dart_converter.h"
#include "third_party/tonic/dart_args.h"
#include "third_party/tonic/dart_binding_macros.h"
Expand All @@ -16,6 +18,64 @@
#include "third_party/tonic/logging/dart_invoke.h"

namespace flutter {
namespace {

sk_sp<SkSurface> MakeSnapshotSurface(const SkISize& picture_size,
fml::WeakPtr<GrContext> resource_context) {
SkImageInfo image_info = SkImageInfo::MakeN32Premul(
picture_size.width(), picture_size.height(), SkColorSpace::MakeSRGB());
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the colorspace is nullptr, it defaults to SRGB. We have stuck to this default behavior elsewhere. Please do the same here as well.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had to remove this change, it started causing tests to fail.

if (resource_context) {
return SkSurface::MakeRenderTarget(resource_context.get(), // context
SkBudgeted::kNo, // budgeted
image_info // image info
);
} else {
return SkSurface::MakeRaster(image_info);
}
}

/// Makes a RAM backed (Raster) image of a picture.
/// @param[in] picture The picture that will get converted to an image.
/// @param[in] surface The surface tha will be used to render the picture. This
/// will be CPU or GPU based.
/// @todo Currently this creates a RAM backed image regardless of what type of
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"RAM backed" is odd terminology. Even device memory is in RAM right? Can we stick to just the device/host terminology like you have done for the traces?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IMO device/host is very confusing. Even yuqian asked me to add a comment to his code to clarify them. Images in RAM vs VRAM is the model I've used when dealing with the discrete GPU model. Can you tell me exactly what you want here and I'll plop it in.

/// surface is used. In certain instances we may want a GPU backed image
/// from a GPU surface to avoid the conversion.
sk_sp<SkImage> MakeRasterSnapshot(sk_sp<SkPicture> picture,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here are some points that Chinmay and I discussed offline, and think would be nice to add to the code comment:

  1. MakeRasterSnapshot makes a CPU-backed image (instead of a GPU-backed image). It's enables peeking pixels from the Dart code, or encoding the image for storage. But it introduces extra overhead for converting GPU images to CPU images. (Raster means CPU in Skia's jargon.)

  2. Maybe it's nice to also have a MakeSnapshot API that generates a GPU-backed image without the GPU-to-CPU overhead. It doesn't have to be added in this PR. Maybe just left a comment here and add it in a future PR, potentially with some benchmarks to verify the performance gain of MakeSnapshot over MakeRasterSnapshot.

  3. Shader warm-up only needs MakeSnapshot (and as Skia pointed out, we can further modify the GrContext to skip all GPU operations other than shader compilations). We can still preserve the behavior of using MakeRasterSnapshot in this PR. But eventually we'll want to switch to a cheaper version. Maybe leave a comment in the code so we don't forget this.

  4. In some cases, the developer may just want to have a GPU-backed snapshot, and directly composite it back into a GPU surface. The MakeRasterSnapshot will be quite expensive for this case as it triggers a GPU-to-CPU copy, and a CPU-to-GPU copy. Maybe leave a comment here to suggest that we should use the future MakeSnapshot API for this case.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. I'll add an issue once this lands.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd like to add that #2 was discussed here at length:
#8835

Would it be possible to work #2 in since this section of code is related? Having a GPU-backed image and avoiding the GPU-to-CPU overhead would be beneficial.

Adding @Hixie - I will soon address the concerns brought up there about increased documentation and tests in this area.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd like to add that #2 was discussed here at length:
#8835

Would it be possible to work #2 in since this section of code is related? Having a GPU-backed image and avoiding the GPU-to-CPU overhead would be beneficial.

Adding @Hixie - I will soon address the concerns brought up there about increased documentation and tests in this area.

Noted. I think it's something we should do as well, but I think it should be prioritized against other issues and not serve as an impediment to this change.

sk_sp<SkSurface> surface) {
TRACE_EVENT0("flutter", __FUNCTION__);

if (surface == nullptr || surface->getCanvas() == nullptr) {
return nullptr;
}

surface->getCanvas()->drawPicture(picture.get());

surface->getCanvas()->flush();

// Here device could mean GPU or CPU (depending on the supplied surface) and
// host means CPU; this is different from use cases like Flutter driver tests
// where device means mobile devices and host means laptops/desktops.
sk_sp<SkImage> device_snapshot;
{
TRACE_EVENT0("flutter", "MakeDeviceSnpashot");
device_snapshot = surface->makeImageSnapshot();
}

if (device_snapshot == nullptr) {
return nullptr;
}

{
TRACE_EVENT0("flutter", "DeviceHostTransfer");
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe leave a comment here like "Here device means GPU and host means CPU; this is different from use cases like Flutter driver tests where device means mobile devices and host means laptops/desktops."

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

if (auto raster_image = device_snapshot->makeRasterImage()) {
return raster_image;
}
}

return nullptr;
}
} // namespace

IMPLEMENT_WRAPPERTYPEINFO(ui, Picture);

Expand Down Expand Up @@ -75,8 +135,8 @@ Dart_Handle Picture::RasterizeToImage(sk_sp<SkPicture> picture,
new tonic::DartPersistentValue(dart_state, raw_image_callback);
auto unref_queue = dart_state->GetSkiaUnrefQueue();
auto ui_task_runner = dart_state->GetTaskRunners().GetUITaskRunner();
auto gpu_task_runner = dart_state->GetTaskRunners().GetGPUTaskRunner();
auto snapshot_delegate = dart_state->GetSnapshotDelegate();
auto io_task_runner = dart_state->GetTaskRunners().GetIOTaskRunner();
fml::WeakPtr<GrContext> resource_context = dart_state->GetResourceContext();

// We can't create an image on this task runner because we don't have a
// graphics context. Even if we did, it would be slow anyway. Also, this
Expand Down Expand Up @@ -111,17 +171,16 @@ Dart_Handle Picture::RasterizeToImage(sk_sp<SkPicture> picture,
delete image_callback;
});

// Kick things off on the GPU.
fml::TaskRunner::RunNowOrPostTask(
gpu_task_runner,
[ui_task_runner, snapshot_delegate, picture, picture_bounds, ui_task] {
sk_sp<SkImage> raster_image =
snapshot_delegate->MakeRasterSnapshot(picture, picture_bounds);

fml::TaskRunner::RunNowOrPostTask(
ui_task_runner,
[ui_task, raster_image]() { ui_task(raster_image); });
});
fml::TaskRunner::RunNowOrPostTask(io_task_runner, [ui_task_runner, picture,
picture_bounds, ui_task,
resource_context] {
sk_sp<SkSurface> surface =
MakeSnapshotSurface(picture_bounds, resource_context);
sk_sp<SkImage> raster_image = MakeRasterSnapshot(picture, surface);

fml::TaskRunner::RunNowOrPostTask(
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

RunNowOrPostTask is only used if there is some form of latching involved. There isn't any here. Please post the task directly to the UI task runner. I realize this may have been because the previous code did the same. But there really is no reason to do this here.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had to revert this too. It was causing a deadlock for some tests.

ui_task_runner, [ui_task, raster_image]() { ui_task(raster_image); });
});

return Dart_Null();
}
Expand Down
21 changes: 0 additions & 21 deletions lib/ui/snapshot_delegate.h

This file was deleted.

6 changes: 0 additions & 6 deletions lib/ui/ui_dart_state.cc
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,6 @@ UIDartState::UIDartState(
TaskRunners task_runners,
TaskObserverAdd add_callback,
TaskObserverRemove remove_callback,
fml::WeakPtr<SnapshotDelegate> snapshot_delegate,
fml::WeakPtr<IOManager> io_manager,
fml::WeakPtr<ImageDecoder> image_decoder,
std::string advisory_script_uri,
Expand All @@ -28,7 +27,6 @@ UIDartState::UIDartState(
: task_runners_(std::move(task_runners)),
add_callback_(std::move(add_callback)),
remove_callback_(std::move(remove_callback)),
snapshot_delegate_(std::move(snapshot_delegate)),
io_manager_(std::move(io_manager)),
image_decoder_(std::move(image_decoder)),
advisory_script_uri_(std::move(advisory_script_uri)),
Expand Down Expand Up @@ -115,10 +113,6 @@ void UIDartState::AddOrRemoveTaskObserver(bool add) {
}
}

fml::WeakPtr<SnapshotDelegate> UIDartState::GetSnapshotDelegate() const {
return snapshot_delegate_;
}

fml::WeakPtr<GrContext> UIDartState::GetResourceContext() const {
if (!io_manager_) {
return {};
Expand Down
5 changes: 0 additions & 5 deletions lib/ui/ui_dart_state.h
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,6 @@
#include "flutter/lib/ui/io_manager.h"
#include "flutter/lib/ui/isolate_name_server/isolate_name_server.h"
#include "flutter/lib/ui/painting/image_decoder.h"
#include "flutter/lib/ui/snapshot_delegate.h"
#include "third_party/dart/runtime/include/dart_api.h"
#include "third_party/skia/include/gpu/GrContext.h"
#include "third_party/tonic/dart_microtask_queue.h"
Expand Down Expand Up @@ -50,8 +49,6 @@ class UIDartState : public tonic::DartState {

fml::RefPtr<flutter::SkiaUnrefQueue> GetSkiaUnrefQueue() const;

fml::WeakPtr<SnapshotDelegate> GetSnapshotDelegate() const;

fml::WeakPtr<GrContext> GetResourceContext() const;

fml::WeakPtr<ImageDecoder> GetImageDecoder() const;
Expand All @@ -78,7 +75,6 @@ class UIDartState : public tonic::DartState {
UIDartState(TaskRunners task_runners,
TaskObserverAdd add_callback,
TaskObserverRemove remove_callback,
fml::WeakPtr<SnapshotDelegate> snapshot_delegate,
fml::WeakPtr<IOManager> io_manager,
fml::WeakPtr<ImageDecoder> image_decoder,
std::string advisory_script_uri,
Expand All @@ -101,7 +97,6 @@ class UIDartState : public tonic::DartState {
const TaskRunners task_runners_;
const TaskObserverAdd add_callback_;
const TaskObserverRemove remove_callback_;
fml::WeakPtr<SnapshotDelegate> snapshot_delegate_;
fml::WeakPtr<IOManager> io_manager_;
fml::WeakPtr<ImageDecoder> image_decoder_;
const std::string advisory_script_uri_;
Expand Down
28 changes: 11 additions & 17 deletions runtime/dart_isolate.cc
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,6 @@ std::weak_ptr<DartIsolate> DartIsolate::CreateRootIsolate(
fml::RefPtr<const DartSnapshot> shared_snapshot,
TaskRunners task_runners,
std::unique_ptr<Window> window,
fml::WeakPtr<SnapshotDelegate> snapshot_delegate,
fml::WeakPtr<IOManager> io_manager,
fml::WeakPtr<ImageDecoder> image_decoder,
std::string advisory_script_uri,
Expand All @@ -57,18 +56,17 @@ std::weak_ptr<DartIsolate> DartIsolate::CreateRootIsolate(
// being prepared to run.
auto root_embedder_data = std::make_unique<std::shared_ptr<DartIsolate>>(
std::make_shared<DartIsolate>(
settings, // settings
std::move(isolate_snapshot), // isolate snapshot
std::move(shared_snapshot), // shared snapshot
task_runners, // task runners
std::move(snapshot_delegate), // snapshot delegate
std::move(io_manager), // IO manager
std::move(image_decoder), // Image Decoder
advisory_script_uri, // advisory URI
advisory_script_entrypoint, // advisory entrypoint
nullptr, // child isolate preparer
isolate_create_callback, // isolate create callback
isolate_shutdown_callback // isolate shutdown callback
settings, // settings
std::move(isolate_snapshot), // isolate snapshot
std::move(shared_snapshot), // shared snapshot
task_runners, // task runners
std::move(io_manager), // IO manager
std::move(image_decoder), // Image Decoder
advisory_script_uri, // advisory URI
advisory_script_entrypoint, // advisory entrypoint
nullptr, // child isolate preparer
isolate_create_callback, // isolate create callback
isolate_shutdown_callback // isolate shutdown callback
));

std::tie(vm_isolate, embedder_isolate) = CreateDartVMAndEmbedderObjectPair(
Expand Down Expand Up @@ -106,7 +104,6 @@ DartIsolate::DartIsolate(const Settings& settings,
fml::RefPtr<const DartSnapshot> isolate_snapshot,
fml::RefPtr<const DartSnapshot> shared_snapshot,
TaskRunners task_runners,
fml::WeakPtr<SnapshotDelegate> snapshot_delegate,
fml::WeakPtr<IOManager> io_manager,
fml::WeakPtr<ImageDecoder> image_decoder,
std::string advisory_script_uri,
Expand All @@ -117,7 +114,6 @@ DartIsolate::DartIsolate(const Settings& settings,
: UIDartState(std::move(task_runners),
settings.task_observer_add,
settings.task_observer_remove,
std::move(snapshot_delegate),
std::move(io_manager),
std::move(image_decoder),
advisory_script_uri,
Expand Down Expand Up @@ -599,7 +595,6 @@ Dart_Isolate DartIsolate::DartCreateAndStartServiceIsolate(
vm_data->GetSharedSnapshot(), // shared snapshot
null_task_runners, // task runners
nullptr, // window
{}, // snapshot delegate
{}, // IO Manager
{}, // Image Decoder
DART_VM_SERVICE_ISOLATE_NAME, // script uri
Expand Down Expand Up @@ -712,7 +707,6 @@ DartIsolate::CreateDartVMAndEmbedderObjectPair(
(*raw_embedder_isolate)->GetIsolateSnapshot(), // isolate_snapshot
(*raw_embedder_isolate)->GetSharedSnapshot(), // shared_snapshot
null_task_runners, // task_runners
fml::WeakPtr<SnapshotDelegate>{}, // snapshot_delegate
fml::WeakPtr<IOManager>{}, // io_manager
fml::WeakPtr<ImageDecoder>{}, // io_manager
advisory_script_uri, // advisory_script_uri
Expand Down
3 changes: 0 additions & 3 deletions runtime/dart_isolate.h
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,6 @@
#include "flutter/fml/macros.h"
#include "flutter/fml/mapping.h"
#include "flutter/lib/ui/io_manager.h"
#include "flutter/lib/ui/snapshot_delegate.h"
#include "flutter/lib/ui/ui_dart_state.h"
#include "flutter/lib/ui/window/window.h"
#include "flutter/runtime/dart_snapshot.h"
Expand Down Expand Up @@ -47,7 +46,6 @@ class DartIsolate : public UIDartState {
fml::RefPtr<const DartSnapshot> shared_snapshot,
TaskRunners task_runners,
std::unique_ptr<Window> window,
fml::WeakPtr<SnapshotDelegate> snapshot_delegate,
fml::WeakPtr<IOManager> io_manager,
fml::WeakPtr<ImageDecoder> image_decoder,
std::string advisory_script_uri,
Expand All @@ -60,7 +58,6 @@ class DartIsolate : public UIDartState {
fml::RefPtr<const DartSnapshot> isolate_snapshot,
fml::RefPtr<const DartSnapshot> shared_snapshot,
TaskRunners task_runners,
fml::WeakPtr<SnapshotDelegate> snapshot_delegate,
fml::WeakPtr<IOManager> io_manager,
fml::WeakPtr<ImageDecoder> image_decoder,
std::string advisory_script_uri,
Expand Down
3 changes: 0 additions & 3 deletions runtime/dart_isolate_unittests.cc
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,6 @@ TEST_F(DartIsolateTest, RootIsolateCreationAndShutdown) {
vm_data->GetSharedSnapshot(), // shared snapshot
std::move(task_runners), // task runners
nullptr, // window
{}, // snapshot delegate
{}, // io manager
{}, // image decoder
"main.dart", // advisory uri
Expand Down Expand Up @@ -75,7 +74,6 @@ TEST_F(DartIsolateTest, IsolateShutdownCallbackIsInIsolateScope) {
vm_data->GetSharedSnapshot(), // shared snapshot
std::move(task_runners), // task runners
nullptr, // window
{}, // snapshot delegate
{}, // io manager
{}, // image decoder
"main.dart", // advisory uri
Expand Down Expand Up @@ -186,7 +184,6 @@ static void RunDartCodeInIsolate(DartVMRef& vm_ref,
vm_data->GetSharedSnapshot(), // shared snapshot
std::move(task_runners), // task runners
nullptr, // window
{}, // snapshot delegate
{}, // io manager
{}, // image decoder
"main.dart", // advisory uri
Expand Down
1 change: 0 additions & 1 deletion runtime/dart_lifecycle_unittests.cc
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,6 @@ static std::shared_ptr<DartIsolate> CreateAndRunRootIsolate(
vm.GetSharedSnapshot(), // shared_snapshot
runners, // task_runners
{}, // window
{}, // snapshot_delegate
{}, // io_manager
{}, // image_decoder
"main.dart", // advisory_script_uri
Expand Down
6 changes: 0 additions & 6 deletions runtime/runtime_controller.cc
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,6 @@ RuntimeController::RuntimeController(
fml::RefPtr<const DartSnapshot> p_isolate_snapshot,
fml::RefPtr<const DartSnapshot> p_shared_snapshot,
TaskRunners p_task_runners,
fml::WeakPtr<SnapshotDelegate> p_snapshot_delegate,
fml::WeakPtr<IOManager> p_io_manager,
fml::WeakPtr<ImageDecoder> p_image_decoder,
std::string p_advisory_script_uri,
Expand All @@ -33,7 +32,6 @@ RuntimeController::RuntimeController(
std::move(p_isolate_snapshot),
std::move(p_shared_snapshot),
std::move(p_task_runners),
std::move(p_snapshot_delegate),
std::move(p_io_manager),
std::move(p_image_decoder),
std::move(p_advisory_script_uri),
Expand All @@ -49,7 +47,6 @@ RuntimeController::RuntimeController(
fml::RefPtr<const DartSnapshot> p_isolate_snapshot,
fml::RefPtr<const DartSnapshot> p_shared_snapshot,
TaskRunners p_task_runners,
fml::WeakPtr<SnapshotDelegate> p_snapshot_delegate,
fml::WeakPtr<IOManager> p_io_manager,
fml::WeakPtr<ImageDecoder> p_image_decoder,
std::string p_advisory_script_uri,
Expand All @@ -63,7 +60,6 @@ RuntimeController::RuntimeController(
isolate_snapshot_(std::move(p_isolate_snapshot)),
shared_snapshot_(std::move(p_shared_snapshot)),
task_runners_(p_task_runners),
snapshot_delegate_(p_snapshot_delegate),
io_manager_(p_io_manager),
image_decoder_(p_image_decoder),
advisory_script_uri_(p_advisory_script_uri),
Expand All @@ -81,7 +77,6 @@ RuntimeController::RuntimeController(
shared_snapshot_, //
task_runners_, //
std::make_unique<Window>(this), //
snapshot_delegate_, //
io_manager_, //
image_decoder_, //
p_advisory_script_uri, //
Expand Down Expand Up @@ -142,7 +137,6 @@ std::unique_ptr<RuntimeController> RuntimeController::Clone() const {
isolate_snapshot_, //
shared_snapshot_, //
task_runners_, //
snapshot_delegate_, //
io_manager_, //
image_decoder_, //
advisory_script_uri_, //
Expand Down
Loading