-
Notifications
You must be signed in to change notification settings - Fork 6k
Bare-bones iOS FKA implementation #55964
Bare-bones iOS FKA implementation #55964
Conversation
…asic full keyboard access (FKA), except for scrolling which will be implemented in a different patch. Partially fixes #76497 On iOS 15 and above, FKA, if enabled, always consumes relevant key events, so the Flutter framework can't see those key events as they won't be delivered via the UIResponder chain (https://developer.apple.com/documentation/uikit/uikeycommand/3780513-wantspriorityoversystembehavior). This patch provides the basic focus-related information to the iOS focus engine, based on the information that is already available in the accessibility tree, so the iOS focus engine can navigate the UI hierarchy and invoke `accessibilityActivate` on the current focus when the user presses the space key. This at the moment seems to be the best option: - There doesn't seem to be a way to reliably prevents FKA from consuming the key events and that seems to be by design. - The user can remap the FKA keys in iOS system settings, but that key mapping isn't available to apps, so even if the framework can get the key events it won't be able to honor custom key maps. - When FKA is on, `-[FlutterView isAccessibilityElement]` is called without user interaction (presumably it's called when the view appears), so when the user interacts with the app using FKA, it's likely that the accessibility is already enabled, we don't have to worry detecting whether FKA is on (at least for now). Scrolling using FKA currently does not work despite `FlutterScrollableSemanticsObject` conforms to `UIFocusItemScrollableContainer`. `setContentOffset:` must be implemented using a new API that informs the framework of the new contentOffset in the scroll view. `accessibilityScroll` does not work because it scrolls too much in most cases.
Video looks cool! 😀 I'm curious if the a11y mode and FKA mode are both on, do a11y focus (you can still swipe to focus) and keyboard focus affect each other? |
#pragma mark - UIFocusItemContainer Conformance | ||
|
||
- (NSArray<id<UIFocusItem>>*)focusItemsInRect:(CGRect)rect { | ||
// It seems the iOS focus system rely heavily on this method (instead of |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: relies
nit: I know by "this method" you're referring to focusItemsInRect
but because of the location of this comment, "this method" can be confused as "self.childrenInHitTestOrder"
- (NSArray<id<UIFocusItem>>*)focusItemsInRect:(CGRect)rect { | ||
// It seems the iOS focus system rely heavily on this method (instead of | ||
// preferredFocusEnvironments) for directional navigation. | ||
// Whether the item order in the returned array matters is unknown. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
interesting 🫨 So if you set it to return a reversed "self.childrenInHitTestOrder", the traversing order is the same?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah I guess the iOS focus engine re-organize the returned array using the coordinates reported by frame
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Oh never mind. I turned FKA off on the simulator that's why everything started working magically.
In practice, using voiceover (I'm not familiar with accessibility features so I don't know if there's anything else could change the accessibility focus) and FKA together should be uncommon. |
There're some different a11y modes can change a11y focus
We don't know that. I think we should test the behavior using voiceover and FKA together.
FKA is a new thing to me. I don't know if it should affect a11y focus or not. I think our behavior should follow the native. |
I tried out some |
The original demo video was a lie unfortunately. I forgot to turn on FKA on that simulator so everything suddenly started working. I reverted some deleted changes. Updated the video. Scrolling is not working (as expected since I didn't implement |
[reversedItems | ||
addObject:self.childrenInHitTestOrder[self.childrenInHitTestOrder.count - 1 - i]]; | ||
} | ||
return reversedItems; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Returning a reversed array seems very unintuitive, are you doing this just to make menus and dialogs reachable?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes it's undocumented. I tested in an example app, popup menus couldn't be focused (instead the model barrier is the only thing focusable when a popup menu is present) if the items were in hit test order.
So a11y focus, FKA focus, and keyboard input focus, can be three different thing? Does the a11y focus change if you tap a widget with FKA? |
Right in flutter all three can be different. The uicontrols I tested don't request accessibility focus on [NSObject accessibilityActivate]. So most Flutter widget shouldn't do that by default either I think |
The behavior (accessibilityActivate not requiring accessibility focus) is different than I would assume, but I'm happy as long as our behavior is the same as native! |
0575939
to
6ca023f
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
…157486) flutter/engine@5d7caf7...0b56cb8 2024-10-24 [email protected] Bare-bones iOS FKA implementation (flutter/engine#55964) 2024-10-23 [email protected] [Impeller] libImpeller: Allow appending to the transformation stack. (flutter/engine#56072) 2024-10-23 [email protected] Add standalone 'Mac clangd' builder to replace 'Linux mac_clangd' orchestrator (flutter/engine#56014) 2024-10-23 [email protected] Roll Dart SDK from 75c42f30af7a to dd06a1e3002c (2 revisions) (flutter/engine#56070) 2024-10-23 [email protected] Roll Skia from 42f9070e6625 to 53c9663c3b83 (1 revision) (flutter/engine#56069) If this roll has caused a breakage, revert this CL and stop the roller using the controls here: https://autoroll.skia.org/r/flutter-engine-flutter-autoroll Please CC [email protected],[email protected] on the revert to ensure that a human is aware of the problem. To file a bug in Flutter: https://github.com/flutter/flutter/issues/new/choose To report a problem with the AutoRoller itself, please file a bug: https://issues.skia.org/issues/new?component=1389291&template=1850622 Documentation for the AutoRoller is here: https://skia.googlesource.com/buildbot/+doc/main/autoroll/README.md
…lutter#157486) flutter/engine@5d7caf7...0b56cb8 2024-10-24 [email protected] Bare-bones iOS FKA implementation (flutter/engine#55964) 2024-10-23 [email protected] [Impeller] libImpeller: Allow appending to the transformation stack. (flutter/engine#56072) 2024-10-23 [email protected] Add standalone 'Mac clangd' builder to replace 'Linux mac_clangd' orchestrator (flutter/engine#56014) 2024-10-23 [email protected] Roll Dart SDK from 75c42f30af7a to dd06a1e3002c (2 revisions) (flutter/engine#56070) 2024-10-23 [email protected] Roll Skia from 42f9070e6625 to 53c9663c3b83 (1 revision) (flutter/engine#56069) If this roll has caused a breakage, revert this CL and stop the roller using the controls here: https://autoroll.skia.org/r/flutter-engine-flutter-autoroll Please CC [email protected],[email protected] on the revert to ensure that a human is aware of the problem. To file a bug in Flutter: https://github.com/flutter/flutter/issues/new/choose To report a problem with the AutoRoller itself, please file a bug: https://issues.skia.org/issues/new?component=1389291&template=1850622 Documentation for the AutoRoller is here: https://skia.googlesource.com/buildbot/+doc/main/autoroll/README.md
This PR adds basic FKA scrolling support: when the iOS focus (the focus state is maintained separately from the framework focus, see the previous PR) switches to an item in a scrollable container that is too close to the edge of the viewport, the container will scroll to make sure the next item is visible. Previous PR for context: #55964 https://github.com/user-attachments/assets/84ae5153-f955-4d23-9901-ce942c0e98ac ### Why the UIScrollView subclass in the focus hierarchy The iOS focus system does not provide an API that allows apps to notify it of focus highlight changes. So if we were to keep using the transforms sent by the framework as-is and not introducing any UIViews in the focus hierarchy, the focus highlight will be positioned at the wrong location after scrolling (via FKA or via framework). That does not seem to be part of the public API and the focus system seems to only know how to properly highlight focusable UIViews. ### Things that currently may not work 1. Nested scroll views (have not tried to verify) The `UIScrollView`s are always subviews of the `FlutterView`. If there are nested scrollables the focus system may not be able to properly determine the focus hierarchy (in theory the iOS focus system should never depend on `UIView.parentView` but I haven't tried to verify that). 2. If the next item is too far below the bottom of the screen and there is a tab bar with focusable items, the focus will be transferred to tab bar instead of the next item in the list Video demo (as you can see the scrolling is really finicky): https://github.com/user-attachments/assets/51c2bfe4-d7b3-4614-aa49-4256214f8978 I've tried doing the same thing using a `UITableView` with similar configurations but it seems to have the same problem. I'll try to dig a bit deeper into this and see if there's a workaround. [C++, Objective-C, Java style guides]: https://github.com/flutter/engine/blob/main/CONTRIBUTING.md#style
Reverts: #56606 Initiated by: LongCatIsLooong Reason for reverting: flutter/flutter#159456 Original PR Author: LongCatIsLooong Reviewed By: {chunhtai, cbracken} This change reverts the following previous change: This PR adds basic FKA scrolling support: when the iOS focus (the focus state is maintained separately from the framework focus, see the previous PR) switches to an item in a scrollable container that is too close to the edge of the viewport, the container will scroll to make sure the next item is visible. Previous PR for context: #55964 https://github.com/user-attachments/assets/84ae5153-f955-4d23-9901-ce942c0e98ac ### Why the UIScrollView subclass in the focus hierarchy The iOS focus system does not provide an API that allows apps to notify it of focus highlight changes. So if we were to keep using the transforms sent by the framework as-is and not introducing any UIViews in the focus hierarchy, the focus highlight will be positioned at the wrong location after scrolling (via FKA or via framework). That does not seem to be part of the public API and the focus system seems to only know how to properly highlight focusable UIViews. ### Things that currently may not work 1. Nested scroll views (have not tried to verify) The `UIScrollView`s are always subviews of the `FlutterView`. If there are nested scrollables the focus system may not be able to properly determine the focus hierarchy (in theory the iOS focus system should never depend on `UIView.parentView` but I haven't tried to verify that). 2. If the next item is too far below the bottom of the screen and there is a tab bar with focusable items, the focus will be transferred to tab bar instead of the next item in the list Video demo (as you can see the scrolling is really finicky): https://github.com/user-attachments/assets/51c2bfe4-d7b3-4614-aa49-4256214f8978 I've tried doing the same thing using a `UITableView` with similar configurations but it seems to have the same problem. I'll try to dig a bit deeper into this and see if there's a workaround. [C++, Objective-C, Java style guides]: https://github.com/flutter/engine/blob/main/CONTRIBUTING.md#style
A bare-bones implementation for iOS focus engine support, to enable basic full keyboard access (FKA) (except for scrolling which will be implemented in a different patch). Partially f1xes flutter#76497 https://github.com/user-attachments/assets/427db87e-cc15-483a-85a1-56bf1c02c285 On iOS 15 and above, FKA, if enabled, always consumes relevant key events, so the Flutter framework can't see those key events as they won't be delivered via the `UIResponder` chain (https://developer.apple.com/documentation/uikit/uikeycommand/3780513-wantspriorityoversystembehavior). This patch provides the basic focus-related information to the iOS focus engine, derived from the information already available in the accessibility tree, so the iOS focus engine can navigate the UI hierarchy and invoke `accessibilityActivate` on the current focus when the user presses the space key. This at the moment seems to be the best option: - There doesn't seem to be a way to reliably prevent FKA from consuming the key events and that seems to be by design. - The user can remap the FKA keys in iOS system settings, but that key mapping isn't available to apps, so even if the framework can get the key events it won't be able to honor custom key maps. - When FKA is on, `-[FlutterView isAccessibilityElement]` is called without user interaction (presumably it's called when the view appears), so when the user interacts with the app using FKA, it's likely that the accessibility is already enabled, we don't have to worry detecting whether FKA is on (at least for now). Scrolling using FKA currently does not work despite `FlutterScrollableSemanticsObject` conforms to `UIFocusItemScrollableContainer`. `setContentOffset:` must be implemented using a new API that informs the framework of the new contentOffset in the scroll view. `accessibilityScroll` does not work because it scrolls too much in most cases, and it tells the framework "how much to scroll" instead of "where to scroll to" so it tends to be jumpy. ## What happens on iOS versions earlier than 15 I couldn't find iOS 14 runtime for simulators in xcode 16 so I couldn't test it. But since the key events will be delivered to the framework first regardless of whether FKA is enabled, the framework is supposed to handle keyboard focus navigation even when FKA is on. [C++, Objective-C, Java style guides]: https://github.com/flutter/engine/blob/main/CONTRIBUTING.md#style
This PR adds basic FKA scrolling support: when the iOS focus (the focus state is maintained separately from the framework focus, see the previous PR) switches to an item in a scrollable container that is too close to the edge of the viewport, the container will scroll to make sure the next item is visible. Previous PR for context: flutter/engine#55964 https://github.com/user-attachments/assets/84ae5153-f955-4d23-9901-ce942c0e98ac ### Why the UIScrollView subclass in the focus hierarchy The iOS focus system does not provide an API that allows apps to notify it of focus highlight changes. So if we were to keep using the transforms sent by the framework as-is and not introducing any UIViews in the focus hierarchy, the focus highlight will be positioned at the wrong location after scrolling (via FKA or via framework). That does not seem to be part of the public API and the focus system seems to only know how to properly highlight focusable UIViews. ### Things that currently may not work 1. Nested scroll views (have not tried to verify) The `UIScrollView`s are always subviews of the `FlutterView`. If there are nested scrollables the focus system may not be able to properly determine the focus hierarchy (in theory the iOS focus system should never depend on `UIView.parentView` but I haven't tried to verify that). 2. If the next item is too far below the bottom of the screen and there is a tab bar with focusable items, the focus will be transferred to tab bar instead of the next item in the list Video demo (as you can see the scrolling is really finicky): https://github.com/user-attachments/assets/51c2bfe4-d7b3-4614-aa49-4256214f8978 I've tried doing the same thing using a `UITableView` with similar configurations but it seems to have the same problem. I'll try to dig a bit deeper into this and see if there's a workaround. [C++, Objective-C, Java style guides]: https://github.com/flutter/engine/blob/main/CONTRIBUTING.md#style
…er/engine#56802) Reverts: flutter/engine#56606 Initiated by: LongCatIsLooong Reason for reverting: flutter#159456 Original PR Author: LongCatIsLooong Reviewed By: {chunhtai, cbracken} This change reverts the following previous change: This PR adds basic FKA scrolling support: when the iOS focus (the focus state is maintained separately from the framework focus, see the previous PR) switches to an item in a scrollable container that is too close to the edge of the viewport, the container will scroll to make sure the next item is visible. Previous PR for context: flutter/engine#55964 https://github.com/user-attachments/assets/84ae5153-f955-4d23-9901-ce942c0e98ac ### Why the UIScrollView subclass in the focus hierarchy The iOS focus system does not provide an API that allows apps to notify it of focus highlight changes. So if we were to keep using the transforms sent by the framework as-is and not introducing any UIViews in the focus hierarchy, the focus highlight will be positioned at the wrong location after scrolling (via FKA or via framework). That does not seem to be part of the public API and the focus system seems to only know how to properly highlight focusable UIViews. ### Things that currently may not work 1. Nested scroll views (have not tried to verify) The `UIScrollView`s are always subviews of the `FlutterView`. If there are nested scrollables the focus system may not be able to properly determine the focus hierarchy (in theory the iOS focus system should never depend on `UIView.parentView` but I haven't tried to verify that). 2. If the next item is too far below the bottom of the screen and there is a tab bar with focusable items, the focus will be transferred to tab bar instead of the next item in the list Video demo (as you can see the scrolling is really finicky): https://github.com/user-attachments/assets/51c2bfe4-d7b3-4614-aa49-4256214f8978 I've tried doing the same thing using a `UITableView` with similar configurations but it seems to have the same problem. I'll try to dig a bit deeper into this and see if there's a workaround. [C++, Objective-C, Java style guides]: https://github.com/flutter/engine/blob/main/CONTRIBUTING.md#style
A bare-bones implementation for iOS focus engine support, to enable basic full keyboard access (FKA) (except for scrolling which will be implemented in a different patch).
Partially f1xes flutter/flutter#76497
Screen.Recording.2024-10-21.at.7.11.10.PM.mov
On iOS 15 and above, FKA, if enabled, always consumes relevant key events, so the Flutter framework can't see those key events as they won't be delivered via the
UIResponder
chain (https://developer.apple.com/documentation/uikit/uikeycommand/3780513-wantspriorityoversystembehavior). This patch provides the basic focus-related information to the iOS focus engine, derived from the information already available in the accessibility tree, so the iOS focus engine can navigate the UI hierarchy and invokeaccessibilityActivate
on the current focus when the user presses the space key.This at the moment seems to be the best option:
-[FlutterView isAccessibilityElement]
is called without user interaction (presumably it's called when the view appears), so when the user interacts with the app using FKA, it's likely that the accessibility is already enabled, we don't have to worry detecting whether FKA is on (at least for now).Scrolling using FKA currently does not work despite
FlutterScrollableSemanticsObject
conforms toUIFocusItemScrollableContainer
.setContentOffset:
must be implemented using a new API that informs the framework of the new contentOffset in the scroll view.accessibilityScroll
does not work because it scrolls too much in most cases, and it tells the framework "how much to scroll" instead of "where to scroll to" so it tends to be jumpy.What happens on iOS versions earlier than 15
I couldn't find iOS 14 runtime for simulators in xcode 16 so I couldn't test it. But since the key events will be delivered to the framework first regardless of whether FKA is enabled, the framework is supposed to handle keyboard focus navigation even when FKA is on.
Pre-launch Checklist
///
).If you need help, consider asking for advice on the #hackers-new channel on Discord.