[iOS] Full keyboard access scrolling #56606

LongCatIsLooong · 2024-11-15T02:32:54Z

This PR adds basic FKA scrolling support: when the iOS focus (the focus state is maintained separately from the framework focus, see the previous PR) switches to an item in a scrollable container that is too close to the edge of the viewport, the container will scroll to make sure the next item is visible.

Previous PR for context: #55964

Screen.Recording.2024-11-18.at.1.48.19.PM.mov

Why the UIScrollView subclass in the focus hierarchy

The iOS focus system does not provide an API that allows apps to notify it of focus highlight changes. So if we were to keep using the transforms sent by the framework as-is and not introducing any UIViews in the focus hierarchy, the focus highlight will be positioned at the wrong location after scrolling (via FKA or via framework). That does not seem to be part of the public API and the focus system seems to only know how to properly highlight focusable UIViews.

Things that currently may not work

Nested scroll views (have not tried to verify)

The UIScrollViews are always subviews of the FlutterView. If there are nested scrollables the focus system may not be able to properly determine the focus hierarchy (in theory the iOS focus system should never depend on UIView.parentView but I haven't tried to verify that).

If the next item is too far below the bottom of the screen and there is a tab bar with focusable items, the focus will be transferred to tab bar instead of the next item in the list

Video demo (as you can see the scrolling is really finicky):

Screen.Recording.2024-11-18.at.2.09.09.PM.mov

I've tried doing the same thing using a UITableView with similar configurations but it seems to have the same problem. I'll try to dig a bit deeper into this and see if there's a workaround.

Pre-launch Checklist

I read the Contributor Guide and followed the process outlined there for submitting PRs.
I read the Tree Hygiene wiki page, which explains my responsibilities.
I read and followed the Flutter Style Guide and the C++, Objective-C, Java style guides.
I listed at least one issue that this PR fixes in the description above.
I added new tests to check the change I am making or feature I am adding, or the PR is test-exempt. See testing the engine for instructions on writing and running engine tests.
I updated/added relevant documentation (doc comments with ///).
I signed the CLA.
All existing and new tests are passing.

If you need help, consider asking for advice on the #hackers-new channel on Discord.

LongCatIsLooong · 2024-11-18T23:04:40Z

shell/platform/darwin/ios/framework/Source/SemanticsObject.mm

@@ -174,7 +176,9 @@ - (void)accessibilityBridgeDidFinishUpdate {
  // contentOffset is 0.0, only the scroll down action is available.
  self.scrollView.frame = self.accessibilityFrame;
  self.scrollView.contentSize = [self contentSizeInternal];
-  [self.scrollView setContentOffset:[self contentOffsetInternal] animated:NO];
+  if (!self.scrollView.isDoingSystemScrolling) {


This means if the scroll view is in a FKA scrolling animation, all framework updates will be disregarded.

This is not optimal, if the user wants to interrupt the scrolling animation via swipe gestures, the UIScrollView will be temporarily out of sync with the framework scroll offset, but once the animation is done, this UIScrollView will send one last offset update to the framework, so it's guaranteed that the framework semantics tree will need another update and the new tree will also update the accessibility bridge and the offsets maintained by the scroll view and the framework will become consistent again.

I have tried removing this flag, and the offsets echoed back from the framework seem always to be one step behind and I am not sure exactly why:

It seems Shell::OnPlatformViewDispatchSemanticsAction and Shell::OnEngineUpdateSemantics both run tasks using RunNowOrPostTask. Now that the UI thread is the same as the platform thread, there should be no delay, I was expecting:

<<<FKA setContentOffset: {0, 13.333333333333334} <<<FKA setContentOffset: {0, 13.333333333333334} >>>framework setContentOffset: {0, 13.333333333333343} <<<FKA setContentOffset: {0, 15} <<<FKA setContentOffset: {0, 15} >>>framework setContentOffset: {0, 15}

but instead it's always delayed:

<<<FKA setContentOffset: {0, 13.333333333333334} <<<FKA setContentOffset: {0, 13.333333333333334} // ... framework sending back the previous outdated value <<<FKA setContentOffset: {0, 15} <<<FKA setContentOffset: {0, 15} >>>framework setContentOffset: {0, 13.333333333333334}

full log

flutter: ▄▄▄▄▄▄▄▄ Frame 19 1m 36s 583.332ms ▄▄▄▄▄▄▄▄ <<<FKA setContentOffset: {0, 0.66666666666666663} flutter: _RenderScrollSemantics#0a6f4 relayoutBoundary=up1 => Offset(0.0, 0.7), SchedulerPhase.idle <<<FKA setContentOffset: {0, 3} flutter: ▄▄▄▄▄▄▄▄ Frame 20 1m 38s 933.334ms ▄▄▄▄▄▄▄▄ <<<FKA setContentOffset: {0, 3} >>>framework setContentOffset: {0, 0.66666666666666663} flutter: _RenderScrollSemantics#0a6f4 relayoutBoundary=up1 => Offset(0.0, 3.0), SchedulerPhase.idle flutter: _RenderScrollSemantics#0a6f4 relayoutBoundary=up1 => Offset(0.0, 3.0), SchedulerPhase.idle <<<FKA setContentOffset: {0, 5} flutter: ▄▄▄▄▄▄▄▄ Frame 21 1m 38s 949.999ms ▄▄▄▄▄▄▄▄ <<<FKA setContentOffset: {0, 5} >>>framework setContentOffset: {0, 3} flutter: _RenderScrollSemantics#0a6f4 relayoutBoundary=up1 => Offset(0.0, 5.0), SchedulerPhase.idle flutter: _RenderScrollSemantics#0a6f4 relayoutBoundary=up1 => Offset(0.0, 5.0), SchedulerPhase.idle <<<FKA setContentOffset: {0, 6.666666666666667} flutter: ▄▄▄▄▄▄▄▄ Frame 22 1m 38s 966.665ms ▄▄▄▄▄▄▄▄ <<<FKA setContentOffset: {0, 6.666666666666667} >>>framework setContentOffset: {0, 5} flutter: _RenderScrollSemantics#0a6f4 relayoutBoundary=up1 => Offset(0.0, 6.7), SchedulerPhase.idle flutter: _RenderScrollSemantics#0a6f4 relayoutBoundary=up1 => Offset(0.0, 6.7), SchedulerPhase.idle <<<FKA setContentOffset: {0, 8.3333333333333339} flutter: ▄▄▄▄▄▄▄▄ Frame 23 1m 38s 983.332ms ▄▄▄▄▄▄▄▄ <<<FKA setContentOffset: {0, 8.3333333333333339} >>>framework setContentOffset: {0, 6.666666666666667} flutter: _RenderScrollSemantics#0a6f4 relayoutBoundary=up1 => Offset(0.0, 8.3), SchedulerPhase.idle flutter: _RenderScrollSemantics#0a6f4 relayoutBoundary=up1 => Offset(0.0, 8.3), SchedulerPhase.idle <<<FKA setContentOffset: {0, 9.6666666666666661} flutter: ▄▄▄▄▄▄▄▄ Frame 24 1m 38s 999.999ms ▄▄▄▄▄▄▄▄ <<<FKA setContentOffset: {0, 9.6666666666666661} >>>framework setContentOffset: {0, 8.3333333333333339} flutter: _RenderScrollSemantics#0a6f4 relayoutBoundary=up1 => Offset(0.0, 9.7), SchedulerPhase.idle flutter: _RenderScrollSemantics#0a6f4 relayoutBoundary=up1 => Offset(0.0, 9.7), SchedulerPhase.idle <<<FKA setContentOffset: {0, 11} flutter: ▄▄▄▄▄▄▄▄ Frame 25 1m 39s 16.665ms ▄▄▄▄▄▄▄▄ <<<FKA setContentOffset: {0, 11} >>>framework setContentOffset: {0, 9.6666666666666661} flutter: _RenderScrollSemantics#0a6f4 relayoutBoundary=up1 => Offset(0.0, 11.0), SchedulerPhase.idle flutter: _RenderScrollSemantics#0a6f4 relayoutBoundary=up1 => Offset(0.0, 11.0), SchedulerPhase.idle <<<FKA setContentOffset: {0, 12.333333333333334} flutter: ▄▄▄▄▄▄▄▄ Frame 26 1m 39s 33.333ms ▄▄▄▄▄▄▄▄ <<<FKA setContentOffset: {0, 12.333333333333334} >>>framework setContentOffset: {0, 11} flutter: _RenderScrollSemantics#0a6f4 relayoutBoundary=up1 => Offset(0.0, 12.3), SchedulerPhase.idle flutter: _RenderScrollSemantics#0a6f4 relayoutBoundary=up1 => Offset(0.0, 12.3), SchedulerPhase.idle <<<FKA setContentOffset: {0, 13.333333333333334} flutter: ▄▄▄▄▄▄▄▄ Frame 27 1m 39s 49.999ms ▄▄▄▄▄▄▄▄ <<<FKA setContentOffset: {0, 13.333333333333334} >>>framework setContentOffset: {0, 12.333333333333334} flutter: _RenderScrollSemantics#0a6f4 relayoutBoundary=up1 => Offset(0.0, 13.3), SchedulerPhase.idle flutter: _RenderScrollSemantics#0a6f4 relayoutBoundary=up1 => Offset(0.0, 13.3), SchedulerPhase.idle <<<FKA setContentOffset: {0, 14.333333333333334} flutter: ▄▄▄▄▄▄▄▄ Frame 28 1m 39s 66.666ms ▄▄▄▄▄▄▄▄ <<<FKA setContentOffset: {0, 14.333333333333334} >>>framework setContentOffset: {0, 13.333333333333334} flutter: _RenderScrollSemantics#0a6f4 relayoutBoundary=up1 => Offset(0.0, 14.3), SchedulerPhase.idle flutter: _RenderScrollSemantics#0a6f4 relayoutBoundary=up1 => Offset(0.0, 14.3), SchedulerPhase.idle <<<FKA setContentOffset: {0, 15.333333333333334} flutter: ▄▄▄▄▄▄▄▄ Frame 29 1m 39s 83.333ms ▄▄▄▄▄▄▄▄ <<<FKA setContentOffset: {0, 15.333333333333334} >>>framework setContentOffset: {0, 14.333333333333334} flutter: _RenderScrollSemantics#0a6f4 relayoutBoundary=up1 => Offset(0.0, 15.3), SchedulerPhase.idle flutter: _RenderScrollSemantics#0a6f4 relayoutBoundary=up1 => Offset(0.0, 15.3), SchedulerPhase.idle <<<FKA setContentOffset: {0, 16} flutter: ▄▄▄▄▄▄▄▄ Frame 30

yes, i think this makes sense since FKA set content will change the scrollcontroller, but it will have to wait for one frame to get the layout and semantics update back

Please add a comment covering this here.

So it looks like I was looking at an older checkout, and it was changed here. The event ordering becomes the way I expected it to be if we run DispatchSemanticsAction synchronously like before. Jonah said RunNowOrPostTask semantics action dispatch is fine we'll just need to post an empty task afterwards to flush the dart futures. I think if we do that instead we could eliminate this flag.

That seems to be the better option because we don't need the flag and the scroll view will be responsive to user gestures during the FKA scroll animation, but I think I'll make the attempt a separate PR.

I am curious to see if it will work, but I think even if the action is send to framework synchronously, it may still take one frame for flutter to rebuild layout and flush semantics back

I am curious to see if it will work, but I think even if the action is send to framework synchronously, it may still take one frame for flutter to rebuild layout and flush semantics back

That's fine I think. The problem with DispatchSemanticsAction being async is that we are essentially maintaining two sources of truth so it becomes a distributed system where the communication channels only guarantee FIFO ordering. If the UIScrollView here and the framework semantics tree having different opinion about what the content offset should be, the update takes time to travel and the inconsistency can be observed.

If the communication is synchronous then the inconsistent state won't be observable as the updates would arrive instantaneously.

reidbaker

android code LGTM

chunhtai · 2024-11-20T19:17:23Z

shell/platform/darwin/ios/framework/Source/FlutterSemanticsScrollView.h

+/// This flag is set by the `FlutterSemanticsScrollView` itself, typically in
+/// one of the `UIScrollViewDelegate` methods.
+///
+/// When this flag is true, the Flutter framework should typically cease


Do we have code from the dart side to enforce this?

looks like the update will be dropped. I will probably say that in the here

The flag is only read by the engine code and the framework side does not know about this. Yeah the wording is confusing I'll change it to "When this flag is true, the SemanticsObject ignores all incoming framework updates"

(updated the comment)

shell/platform/darwin/ios/framework/Source/SemanticsObject+UIFocusSystem.mm

chunhtai · 2024-11-20T20:02:11Z

shell/platform/darwin/ios/framework/Source/SemanticsObject+UIFocusSystem.mm

@@ -48,6 +55,9 @@ - (void)didUpdateFocusInContext:(UIFocusUpdateContext*)context
 }

 - (id<UIFocusEnvironment>)parentFocusEnvironment {
+  if ([self.parent isKindOfClass:[FlutterScrollableSemanticsObject class]]) {


doesn't this need to be recursively up?

since we recursively looks up in frame calculation

This should return the parent focus container, or nil if none. Since [FlutterScrollableSemanticsObject parentFocusEnvironment] returns its parent, and FlutterSemanticsScrollView is a UIScrollView, this looks right to me.

What do you mean by recursively up? Returning the root node? This method is supposed to return the parent object in the focus hierarchy not the root (this method is part of the UIFocusEnvironment protocol and the iOS focus engine will use this method to traverse the focus hierarchy).

shell/platform/darwin/ios/framework/Source/SemanticsObject+UIFocusSystem.mm

chunhtai · 2024-11-20T20:23:53Z

shell/platform/darwin/ios/framework/Source/SemanticsObject+UIFocusSystem.mm

@@ -87,16 +145,99 @@ - (CGRect)frame {
  //
  // This method is only supposed to return items within the given


just a note that I have run into weird bug before because i returned child something outside of the rect of accessibilityContainer. I am not sure if it applies here.

Yeah I can try adding culling here but I'll have to test it separately (and it seems to be working fine w/o the culling).

chunhtai · 2024-11-20T20:29:45Z

shell/platform/darwin/ios/framework/Source/SemanticsObject+UIFocusSystem.mm

+  // to a Float64List in dart.
+  // The size of a CGFloat is architecture-dependent and it's typically the word
+  // size. The last iOS with 32-bit support was iOS 10.
+  static_assert(sizeof(CGPoint) == sizeof(CGFloat) * 2);


should we make it a soft-crash or short circuit if this condition does not hold

This is a compile-time check. We typically deliver the engine as a prebuilt binary so it's not going to a concern for most users / devs. This assert is mostly for code readers to see the assumptions we made about CGFloat here.

This feels pretty naughty. Even just for the sake of readability, I'd be tempted to create a double[], copy the values in, then encode and ship that. The extra CPU work is negligible and the one line of code will save you five lines of comment. Though I'd be tempted to still have a one line comment stating that you're encoding a Float64List for Dart.

chunhtai · 2024-11-20T20:40:15Z

shell/platform/darwin/ios/framework/Source/SemanticsObject.mm

@@ -174,7 +176,9 @@ - (void)accessibilityBridgeDidFinishUpdate {
  // contentOffset is 0.0, only the scroll down action is available.
  self.scrollView.frame = self.accessibilityFrame;
  self.scrollView.contentSize = [self contentSizeInternal];
-  [self.scrollView setContentOffset:[self contentOffsetInternal] animated:NO];
+  if (!self.scrollView.isDoingSystemScrolling) {


yes, i think this makes sense since FKA set content will change the scrollcontroller, but it will have to wait for one frame to get the layout and semantics update back

navaronbracke · 2024-11-21T07:46:34Z

lib/ui/semantics.dart

@@ -86,6 +87,17 @@ class SemanticsAction {
  /// scrollable.
  static const SemanticsAction scrollDown = SemanticsAction._(_kScrollDownIndex, 'scrollDown');

+  /// A request to scroll the scrollable container to a given scroll offset.
+  ///
+  /// The payload of this [SemanticsAction] is a flutter-stanard-encoded


Suggested change

/// The payload of this [SemanticsAction] is a flutter-stanard-encoded

/// The payload of this [SemanticsAction] is a flutter-standard-encoded

cbracken

Love it!

LGTM + codeowners approval for embedder.h change.
Some comments and nits on the iOS side. Shout when you want a second pass!

cbracken · 2024-11-20T21:28:42Z

shell/platform/darwin/ios/framework/Source/SemanticsObject+UIFocusSystem.mm

@@ -48,6 +55,9 @@ - (void)didUpdateFocusInContext:(UIFocusUpdateContext*)context
 }

 - (id<UIFocusEnvironment>)parentFocusEnvironment {
+  if ([self.parent isKindOfClass:[FlutterScrollableSemanticsObject class]]) {


This should return the parent focus container, or nil if none. Since [FlutterScrollableSemanticsObject parentFocusEnvironment] returns its parent, and FlutterSemanticsScrollView is a UIScrollView, this looks right to me.

shell/platform/darwin/ios/framework/Source/SemanticsObject+UIFocusSystem.mm

cbracken · 2024-11-21T16:11:41Z

shell/platform/darwin/ios/framework/Source/SemanticsObject+UIFocusSystem.mm

+  // to a Float64List in dart.
+  // The size of a CGFloat is architecture-dependent and it's typically the word
+  // size. The last iOS with 32-bit support was iOS 10.
+  static_assert(sizeof(CGPoint) == sizeof(CGFloat) * 2);


This feels pretty naughty. Even just for the sake of readability, I'd be tempted to create a double[], copy the values in, then encode and ship that. The extra CPU work is negligible and the one line of code will save you five lines of comment. Though I'd be tempted to still have a one line comment stating that you're encoding a Float64List for Dart.

cbracken · 2024-11-21T16:12:18Z

shell/platform/darwin/ios/framework/Source/SemanticsObject.mm

@@ -174,7 +176,9 @@ - (void)accessibilityBridgeDidFinishUpdate {
  // contentOffset is 0.0, only the scroll down action is available.
  self.scrollView.frame = self.accessibilityFrame;
  self.scrollView.contentSize = [self contentSizeInternal];
-  [self.scrollView setContentOffset:[self contentOffsetInternal] animated:NO];
+  if (!self.scrollView.isDoingSystemScrolling) {


Please add a comment covering this here.

shell/platform/darwin/ios/framework/Source/SemanticsObjectTest.mm

shell/platform/darwin/ios/framework/Source/SemanticsObject.mm

shell/platform/darwin/ios/framework/Source/SemanticsObjectTest.mm

shell/platform/darwin/ios/framework/Source/SemanticsObject+UIFocusSystem.mm

cbracken · 2024-11-21T16:32:12Z

shell/platform/darwin/ios/framework/Source/SemanticsObject+UIFocusSystem.mm

+      [reversedItems addObject:((FlutterScrollableSemanticsObject*)child).scrollView];
+    } else {
+      [reversedItems addObject:child];
+    }


Consider extracting something a static like this for readability:

static id<UIFocusItem>* GetFocusItemForNode(SemanticsObject* node) { if ([node isKindOfClass:[FlutterScrollableSemanticsObject class]]) { return ((FlutterScrollableSemanticsObject*)node).scrollView; } return node; }

That simplifies this code just doing one thing (reversing the order of the focus nodes), rather than two at the same time (reversing the order + determining the focus node).

Made this an instance method to avoid isKindOfClass:

Even better!

cbracken · 2024-11-21T16:48:23Z

shell/platform/darwin/ios/framework/Source/SemanticsObjectTest.mm

@@ -17,6 +18,11 @@
 const float kFloatCompareEpsilon = 0.001;

 @interface SemanticsObject (UIFocusSystem) <UIFocusItem, UIFocusItemContainer>
+@property(nonatomic) FlutterSemanticsScrollView* scrollView;


We're re-declaring this property in three places now.

On SemanticsObject in SemanticsObject.mm for internal use in the class.

On the category on FlutterScrollableSemanticsObject, which is a subclass of SemanticsObject.

Here, for testing.

I think it's safe for us to push the property declaration into SemanticsObject.h and remove it in the other places.

LongCatIsLooong · 2024-11-26T00:33:47Z

Reason for revert: flutter/flutter#159456

This reverts commit 7a8c1e8.

Reverts: #56606 Initiated by: LongCatIsLooong Reason for reverting: flutter/flutter#159456 Original PR Author: LongCatIsLooong Reviewed By: {chunhtai, cbracken} This change reverts the following previous change: This PR adds basic FKA scrolling support: when the iOS focus (the focus state is maintained separately from the framework focus, see the previous PR) switches to an item in a scrollable container that is too close to the edge of the viewport, the container will scroll to make sure the next item is visible. Previous PR for context: #55964 https://github.com/user-attachments/assets/84ae5153-f955-4d23-9901-ce942c0e98ac ### Why the UIScrollView subclass in the focus hierarchy The iOS focus system does not provide an API that allows apps to notify it of focus highlight changes. So if we were to keep using the transforms sent by the framework as-is and not introducing any UIViews in the focus hierarchy, the focus highlight will be positioned at the wrong location after scrolling (via FKA or via framework). That does not seem to be part of the public API and the focus system seems to only know how to properly highlight focusable UIViews. ### Things that currently may not work 1. Nested scroll views (have not tried to verify) The `UIScrollView`s are always subviews of the `FlutterView`. If there are nested scrollables the focus system may not be able to properly determine the focus hierarchy (in theory the iOS focus system should never depend on `UIView.parentView` but I haven't tried to verify that). 2. If the next item is too far below the bottom of the screen and there is a tab bar with focusable items, the focus will be transferred to tab bar instead of the next item in the list Video demo (as you can see the scrolling is really finicky): https://github.com/user-attachments/assets/51c2bfe4-d7b3-4614-aa49-4256214f8978 I've tried doing the same thing using a `UITableView` with similar configurations but it seems to have the same problem. I'll try to dig a bit deeper into this and see if there's a workaround. [C++, Objective-C, Java style guides]: https://github.com/flutter/engine/blob/main/CONTRIBUTING.md#style

…159461) flutter/engine@fe45a66...fb64399 2024-11-26 [email protected] iOS: Migrate PlatformViewsController to Objective-C (flutter/engine#56790) 2024-11-26 [email protected] Started caching HandleGLES's hash and made them immutable (flutter/engine#56800) 2024-11-26 98614782+auto-submit[bot]@users.noreply.github.com Reverts "[iOS] Full keyboard access scrolling (#56606)" (flutter/engine#56802) 2024-11-26 [email protected] Roll Skia from 12c8bd6ac1d9 to c1c8ff84997c (14 revisions) (flutter/engine#56801) 2024-11-25 [email protected] [Impeller] better handle allocation herustics of Android slide in page transition. (flutter/engine#56762) 2024-11-25 [email protected] iOS: Eliminate logging of non-zero origin platformviews (flutter/engine#56796) 2024-11-25 [email protected] [impeller] gles: started storing the number of handle deletions to avoid reallocation (flutter/engine#56799) 2024-11-25 [email protected] Roll Fuchsia Linux SDK from 9o0fWa2xVhmxV6Mtn... to 50xtjbMWWrqay_7m_... (flutter/engine#56795) 2024-11-25 [email protected] Roll Dart SDK from df716eaa6ed2 to 4b49546a1dfa (1 revision) (flutter/engine#56793) 2024-11-25 [email protected] [iOS] Full keyboard access scrolling (flutter/engine#56606) 2024-11-25 [email protected] [android] remove fml_check from surface_texture_external_texture (flutter/engine#56760) 2024-11-25 [email protected] removed unused variable for skia initialization (flutter/engine#56791) Also rolling transitive DEPS: fuchsia/sdk/core/linux-amd64 from 9o0fWa2xVhmx to 50xtjbMWWrqa If this roll has caused a breakage, revert this CL and stop the roller using the controls here: https://autoroll.skia.org/r/flutter-engine-flutter-autoroll Please CC [email protected],[email protected] on the revert to ensure that a human is aware of the problem. To file a bug in Flutter: https://github.com/flutter/flutter/issues/new/choose To report a problem with the AutoRoller itself, please file a bug: https://issues.skia.org/issues/new?component=1389291&template=1850622 Documentation for the AutoRoller is here: https://skia.googlesource.com/buildbot/+doc/main/autoroll/README.md

…visions) (#159461)" (#159498)  Reverts: #159461   Initiated by: yjbanov   Reason for reverting: engine regression #159497   Original PR Author: engine-flutter-autoroll   Reviewed By: {fluttergithubbot}   This change reverts the following previous change: flutter/engine@fe45a66...fb64399 2024-11-26 [email protected] iOS: Migrate PlatformViewsController to Objective-C (flutter/engine#56790) 2024-11-26 [email protected] Started caching HandleGLES's hash and made them immutable (flutter/engine#56800) 2024-11-26 98614782+auto-submit[bot]@users.noreply.github.com Reverts "[iOS] Full keyboard access scrolling (#56606)" (flutter/engine#56802) 2024-11-26 [email protected] Roll Skia from 12c8bd6ac1d9 to c1c8ff84997c (14 revisions) (flutter/engine#56801) 2024-11-25 [email protected] [Impeller] better handle allocation herustics of Android slide in page transition. (flutter/engine#56762) 2024-11-25 [email protected] iOS: Eliminate logging of non-zero origin platformviews (flutter/engine#56796) 2024-11-25 [email protected] [impeller] gles: started storing the number of handle deletions to avoid reallocation (flutter/engine#56799) 2024-11-25 [email protected] Roll Fuchsia Linux SDK from 9o0fWa2xVhmxV6Mtn... to 50xtjbMWWrqay_7m_... (flutter/engine#56795) 2024-11-25 [email protected] Roll Dart SDK from df716eaa6ed2 to 4b49546a1dfa (1 revision) (flutter/engine#56793) 2024-11-25 [email protected] [iOS] Full keyboard access scrolling (flutter/engine#56606) 2024-11-25 [email protected] [android] remove fml_check from surface_texture_external_texture (flutter/engine#56760) 2024-11-25 [email protected] removed unused variable for skia initialization (flutter/engine#56791) Also rolling transitive DEPS: fuchsia/sdk/core/linux-amd64 from 9o0fWa2xVhmx to 50xtjbMWWrqa If this roll has caused a breakage, revert this CL and stop the roller using the controls here: https://autoroll.skia.org/r/flutter-engine-flutter-autoroll Please CC [email protected],[email protected] on the revert to ensure that a human is aware of the problem. To file a bug in Flutter: https://github.com/flutter/flutter/issues/new/choose To report a problem with the AutoRoller itself, please file a bug: https://issues.skia.org/issues/new?component=1389291&template=1850622 Documentation for the AutoRoller is here: https://skia.googlesource.com/buildbot/+doc/main/autoroll/README.md  Co-authored-by: auto-submit[bot] <[email protected]>

…159504) flutter/engine@fe45a66...bd165dc 2024-11-26 [email protected] Revert "iOS: Migrate PlatformViewsController to Objective-C (#56790)" (flutter/engine#56817) 2024-11-26 [email protected] iOS: Rename FlutterPlatformViews_Internal.mm (flutter/engine#56816) 2024-11-26 [email protected] iOS: Eliminate global in platformviews controller (flutter/engine#56805) 2024-11-26 [email protected] Roll Skia from a276978ba7c8 to f149f852c70a (1 revision) (flutter/engine#56812) 2024-11-26 [email protected] Roll Skia from d7a267d88fd6 to a276978ba7c8 (1 revision) (flutter/engine#56811) 2024-11-26 [email protected] Roll Skia from 8d9d892657a7 to d7a267d88fd6 (1 revision) (flutter/engine#56810) 2024-11-26 [email protected] Roll Dart SDK from bdb76c714009 to ca02d403f1a8 (1 revision) (flutter/engine#56809) 2024-11-26 [email protected] Roll Skia from b697dd1b03b2 to 8d9d892657a7 (1 revision) (flutter/engine#56808) 2024-11-26 [email protected] Roll Skia from c1c8ff84997c to b697dd1b03b2 (1 revision) (flutter/engine#56807) 2024-11-26 [email protected] iOS: Delete FlutterPlatformViewsController.layerPoolSize (flutter/engine#56806) 2024-11-26 [email protected] Roll Dart SDK from 4b49546a1dfa to bdb76c714009 (1 revision) (flutter/engine#56803) 2024-11-26 [email protected] iOS: Migrate PlatformViewsController to Objective-C (flutter/engine#56790) 2024-11-26 [email protected] Started caching HandleGLES's hash and made them immutable (flutter/engine#56800) 2024-11-26 98614782+auto-submit[bot]@users.noreply.github.com Reverts "[iOS] Full keyboard access scrolling (#56606)" (flutter/engine#56802) 2024-11-26 [email protected] Roll Skia from 12c8bd6ac1d9 to c1c8ff84997c (14 revisions) (flutter/engine#56801) 2024-11-25 [email protected] [Impeller] better handle allocation herustics of Android slide in page transition. (flutter/engine#56762) 2024-11-25 [email protected] iOS: Eliminate logging of non-zero origin platformviews (flutter/engine#56796) 2024-11-25 [email protected] [impeller] gles: started storing the number of handle deletions to avoid reallocation (flutter/engine#56799) 2024-11-25 [email protected] Roll Fuchsia Linux SDK from 9o0fWa2xVhmxV6Mtn... to 50xtjbMWWrqay_7m_... (flutter/engine#56795) 2024-11-25 [email protected] Roll Dart SDK from df716eaa6ed2 to 4b49546a1dfa (1 revision) (flutter/engine#56793) 2024-11-25 [email protected] [iOS] Full keyboard access scrolling (flutter/engine#56606) 2024-11-25 [email protected] [android] remove fml_check from surface_texture_external_texture (flutter/engine#56760) 2024-11-25 [email protected] removed unused variable for skia initialization (flutter/engine#56791) Also rolling transitive DEPS: fuchsia/sdk/core/linux-amd64 from 9o0fWa2xVhmx to 50xtjbMWWrqa If this roll has caused a breakage, revert this CL and stop the roller using the controls here: https://autoroll.skia.org/r/flutter-engine-flutter-autoroll Please CC [email protected] on the revert to ensure that a human is aware of the problem. To file a bug in Flutter: https://github.com/flutter/flutter/issues/new/choose To report a problem with the AutoRoller itself, please file a bug: https://issues.skia.org/issues/new?component=1389291&template=1850622 Documentation for the AutoRoller is here: https://skia.googlesource.com/buildbot/+doc/main/autoroll/README.md

…)" This reverts commit 50174be.

Reverts #56802 flutter/flutter#159517 should address the engine roll failure. I'm not planning to land this until the coming Monday.

This PR adds basic FKA scrolling support: when the iOS focus (the focus state is maintained separately from the framework focus, see the previous PR) switches to an item in a scrollable container that is too close to the edge of the viewport, the container will scroll to make sure the next item is visible. Previous PR for context: flutter/engine#55964 https://github.com/user-attachments/assets/84ae5153-f955-4d23-9901-ce942c0e98ac ### Why the UIScrollView subclass in the focus hierarchy The iOS focus system does not provide an API that allows apps to notify it of focus highlight changes. So if we were to keep using the transforms sent by the framework as-is and not introducing any UIViews in the focus hierarchy, the focus highlight will be positioned at the wrong location after scrolling (via FKA or via framework). That does not seem to be part of the public API and the focus system seems to only know how to properly highlight focusable UIViews. ### Things that currently may not work 1. Nested scroll views (have not tried to verify) The `UIScrollView`s are always subviews of the `FlutterView`. If there are nested scrollables the focus system may not be able to properly determine the focus hierarchy (in theory the iOS focus system should never depend on `UIView.parentView` but I haven't tried to verify that). 2. If the next item is too far below the bottom of the screen and there is a tab bar with focusable items, the focus will be transferred to tab bar instead of the next item in the list Video demo (as you can see the scrolling is really finicky): https://github.com/user-attachments/assets/51c2bfe4-d7b3-4614-aa49-4256214f8978 I've tried doing the same thing using a `UITableView` with similar configurations but it seems to have the same problem. I'll try to dig a bit deeper into this and see if there's a workaround. [C++, Objective-C, Java style guides]: https://github.com/flutter/engine/blob/main/CONTRIBUTING.md#style

…er/engine#56802) Reverts: flutter/engine#56606 Initiated by: LongCatIsLooong Reason for reverting: flutter#159456 Original PR Author: LongCatIsLooong Reviewed By: {chunhtai, cbracken} This change reverts the following previous change: This PR adds basic FKA scrolling support: when the iOS focus (the focus state is maintained separately from the framework focus, see the previous PR) switches to an item in a scrollable container that is too close to the edge of the viewport, the container will scroll to make sure the next item is visible. Previous PR for context: flutter/engine#55964 https://github.com/user-attachments/assets/84ae5153-f955-4d23-9901-ce942c0e98ac ### Why the UIScrollView subclass in the focus hierarchy The iOS focus system does not provide an API that allows apps to notify it of focus highlight changes. So if we were to keep using the transforms sent by the framework as-is and not introducing any UIViews in the focus hierarchy, the focus highlight will be positioned at the wrong location after scrolling (via FKA or via framework). That does not seem to be part of the public API and the focus system seems to only know how to properly highlight focusable UIViews. ### Things that currently may not work 1. Nested scroll views (have not tried to verify) The `UIScrollView`s are always subviews of the `FlutterView`. If there are nested scrollables the focus system may not be able to properly determine the focus hierarchy (in theory the iOS focus system should never depend on `UIView.parentView` but I haven't tried to verify that). 2. If the next item is too far below the bottom of the screen and there is a tab bar with focusable items, the focus will be transferred to tab bar instead of the next item in the list Video demo (as you can see the scrolling is really finicky): https://github.com/user-attachments/assets/51c2bfe4-d7b3-4614-aa49-4256214f8978 I've tried doing the same thing using a `UITableView` with similar configurations but it seems to have the same problem. I'll try to dig a bit deeper into this and see if there's a workaround. [C++, Objective-C, Java style guides]: https://github.com/flutter/engine/blob/main/CONTRIBUTING.md#style

LongCatIsLooong added 8 commits October 29, 2024 16:17

WIP

de4715d

Basic scrolling

93f866a

FKA scrolling

24eb36d

scroll view scroll

7450322

Merge remote-tracking branch 'upstream/main' into fka-scroll

035b476

tests

a57d56e

tests

0703665

docs

63776e6

github-actions bot added platform-ios platform-fuchsia platform-web Code specifically for the web engine labels Nov 15, 2024

tests

bc501a2

LongCatIsLooong force-pushed the fka-scroll branch from 49a8d81 to bc501a2 Compare November 15, 2024 19:59

add to embedder enums

b91b878

github-actions bot added the platform-android label Nov 15, 2024

fix variable names

4196be8

LongCatIsLooong commented Nov 18, 2024

View reviewed changes

LongCatIsLooong requested review from chunhtai and hannah-hyj November 19, 2024 00:25

LongCatIsLooong marked this pull request as ready for review November 19, 2024 00:26

LongCatIsLooong requested review from cbracken and chinmaygarde as code owners November 19, 2024 00:26

reidbaker reviewed Nov 19, 2024

View reviewed changes

chunhtai reviewed Nov 20, 2024

View reviewed changes

Update FlutterSemanticsScrollView.h

b072522

navaronbracke reviewed Nov 21, 2024

View reviewed changes

cbracken approved these changes Nov 21, 2024

View reviewed changes

cbracken reviewed Nov 21, 2024

View reviewed changes

shell/platform/darwin/ios/framework/Source/SemanticsObject+UIFocusSystem.mm Show resolved Hide resolved

cbracken reviewed Nov 21, 2024

View reviewed changes

auto-submit bot merged commit 7a8c1e8 into flutter:main Nov 25, 2024
36 checks passed

LongCatIsLooong deleted the fka-scroll branch November 25, 2024 21:13

engine-flutter-autoroll mentioned this pull request Nov 25, 2024

Roll Flutter Engine from fe45a6608651 to 7a8c1e8031ff (3 revisions) flutter/flutter#159447

Closed

engine-flutter-autoroll added a commit to engine-flutter-autoroll/flutter that referenced this pull request Nov 25, 2024

7a8c1e803 [iOS] Full keyboard access scrolling (flutter/engine#56606)

580d1ab

engine-flutter-autoroll mentioned this pull request Nov 25, 2024

Roll Flutter Engine from fe45a6608651 to 5db272c977dc (4 revisions) flutter/flutter#159451

Closed

engine-flutter-autoroll added a commit to engine-flutter-autoroll/flutter that referenced this pull request Nov 25, 2024

7a8c1e803 [iOS] Full keyboard access scrolling (flutter/engine#56606)

54b3a74

engine-flutter-autoroll mentioned this pull request Nov 25, 2024

Roll Flutter Engine from fe45a6608651 to 258666b1c730 (5 revisions) flutter/flutter#159453

Closed

engine-flutter-autoroll added a commit to engine-flutter-autoroll/flutter that referenced this pull request Nov 25, 2024

7a8c1e803 [iOS] Full keyboard access scrolling (flutter/engine#56606)

90327e5

LongCatIsLooong added the revert Label used to revert changes in a closed and merged pull request. label Nov 26, 2024

auto-submit bot pushed a commit that referenced this pull request Nov 26, 2024

Revert "[iOS] Full keyboard access scrolling (#56606)"

3d6da02

This reverts commit 7a8c1e8.

auto-submit bot mentioned this pull request Nov 26, 2024

Reverts "[iOS] Full keyboard access scrolling (#56606)" #56802

Merged

8 tasks

auto-submit bot removed the revert Label used to revert changes in a closed and merged pull request. label Nov 26, 2024

engine-flutter-autoroll mentioned this pull request Nov 26, 2024

Roll Flutter Engine from fe45a6608651 to 50174be65944 (10 revisions) flutter/flutter#159459

Closed

engine-flutter-autoroll added a commit to engine-flutter-autoroll/flutter that referenced this pull request Nov 26, 2024

7a8c1e803 [iOS] Full keyboard access scrolling (flutter/engine#56606)

97580e3

engine-flutter-autoroll mentioned this pull request Nov 26, 2024

Roll Flutter Engine from fe45a6608651 to fb6439918252 (12 revisions) flutter/flutter#159461

Merged

engine-flutter-autoroll added a commit to engine-flutter-autoroll/flutter that referenced this pull request Nov 26, 2024

7a8c1e803 [iOS] Full keyboard access scrolling (flutter/engine#56606)

d73e838

auto-submit bot mentioned this pull request Nov 26, 2024

Reverts "Roll Flutter Engine from fe45a6608651 to fb6439918252 (12 revisions) (#159461)" flutter/flutter#159498

Merged

engine-flutter-autoroll mentioned this pull request Nov 26, 2024

Roll Flutter Engine from fe45a6608651 to bd165dc27166 (23 revisions) flutter/flutter#159504

Merged

engine-flutter-autoroll added a commit to engine-flutter-autoroll/flutter that referenced this pull request Nov 26, 2024

7a8c1e803 [iOS] Full keyboard access scrolling (flutter/engine#56606)

3a77d1f

LongCatIsLooong added a commit that referenced this pull request Nov 27, 2024

Revert "Reverts "[iOS] Full keyboard access scrolling (#56606)" (#56802…

b04f507

…)" This reverts commit 50174be.

auto-submit bot pushed a commit that referenced this pull request Dec 2, 2024

Reland "[iOS] Full keyboard access scrolling (#56606)" (#56842)

4b66624

Reverts #56802 flutter/flutter#159517 should address the engine roll failure. I'm not planning to land this until the coming Monday.

LongCatIsLooong mentioned this pull request Dec 16, 2024

Support Full Keyboard Access on iOS for keyboard navigation events flutter/flutter#76497

Closed

SaadSafan mentioned this pull request Apr 7, 2025

Full Keyboard Access on iOS Breaks External Keyboard Navigation in Flutter App flutter/flutter#166683

Open

		@@ -87,16 +145,99 @@ - (CGRect)frame {
		//
		// This method is only supposed to return items within the given

	/// The payload of this [SemanticsAction] is a flutter-stanard-encoded
	/// The payload of this [SemanticsAction] is a flutter-standard-encoded

[iOS] Full keyboard access scrolling #56606

[iOS] Full keyboard access scrolling #56606

Uh oh!

Conversation

LongCatIsLooong commented Nov 15, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why the UIScrollView subclass in the focus hierarchy

Things that currently may not work

Pre-launch Checklist

Uh oh!

Choose a reason for hiding this comment

Uh oh!

LongCatIsLooong Nov 19, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

reidbaker left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

LongCatIsLooong Nov 20, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cbracken Nov 21, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cbracken left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

cbracken Nov 21, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

LongCatIsLooong commented Nov 15, 2024 •

edited

Loading

LongCatIsLooong Nov 19, 2024 •

edited

Loading

LongCatIsLooong Nov 20, 2024 •

edited

Loading

cbracken Nov 21, 2024 •

edited

Loading

cbracken left a comment •

edited

Loading

cbracken Nov 21, 2024 •

edited

Loading

cbracken Nov 21, 2024 •

edited

Loading