[SIP-64] Migrate filter_box to Dashboard Native Filter Component

[graceguo-supercat]

[SIP] Proposal for migrating filter_box viz to dashboard native filter component

Motivation

This is the last step to complete the dashboard filter components. Please see discussion about dashboard filter components.

With the dashboard native filter development become stable, Superset should migrate from filter_box to dashboard native filter components, and prepare to expire filter_box charts. As one of the key component for Superset dashboard in the last 4+ years, and given the big varieties of use cases for different users, this migration will need combination of preview, automatic scripts and manual adjustment, and DB migration.

Note

• The goal of this migration plan is to migrate filter_box used in dashboard to native_filter_configuration dashboard metadata, and saved into dashboard.
• This migration plan do not change the dashboard layout. After dashboard starts to use native filter only, dashboard owner could remove filter_box from their dashboard manually, and reorganize dashboard layout.

Proposed Change

Precondition

Superset is actively used by at least hundreds of users in many companies as their data analytics tool. In order to make sure that majorities of Superset users' work are not lost, damaged by this filter migration, I feel it is very important to make sure we complete the following Precondition check:

• Feature parity: filter components should offer all functionalities that filter_box can do

  • Filter component should support Sort Metric: feat(native-filters): add optional sort metric to select filter #14346 feat(native-filters): add sort metric to select #14590
  • Filter component should allow user add adhoc filter: feat(native-filters): Implement adhoc filters and time picker in Range and Select native filters #14313
  • Filter component should have Required or Optional option: fix(native-filters): allowClear only when required not checked #14353
  • Select filter component should have time range and time column control to limit options: [native filters] Select filter should allow user pick time column #14744
  • Show each filter field (in the filter bar) only when the field is applicable to any chart in the current visible tab.
  • When user pick a filter from filter bar, highlight all visible charts that will be impacted by this filter component.

• Performance: When user start to migrate, dashboard performance should not be impacted by using filter components

  • filter component(s) generate a lot of new queries and blocks chart start to load: [native-filter][performance] Dashboard load slower with native-filter components #14421 [native filter] Instant filter component will generate 2x more fetch #14706

• Extra feature requests: These feature are not available in filter_box, but filter component should provide. They are not blocker to prevent us from moving to Transition Mode.

  • Range filter for numbers
  • Fuzzy filter for text: user can filter text options like prefix, surfix, substring etc.
  • Filterset: allow user to share a dashboard, which is populated with a given filterset.

• Fix filter components P0 and P1 bugs

  • can not set filter scope: JS errors: [native filter] can not set filter scope: JS errors #14422

Transition Mode

Once we resolved the feature parity and performance downgrade issue, it is safe to start migrate.

I group users into 2 type and migration solution is a little different:

airbnb user

Not enabled native filter feature flag at the beginning of migration. These users do not have native_filter_configuration in the dashboard metadata. I propose this migration process:

1. When Dashboard owner open their dashboard, we offer a function in the front-end (JS only), which read config from each filter_box and automatically convert them into the data compatible with native_filter_configuration. Dashboard will seamlessly display filter components from converted filter config. At the same time, dashboard will disable filter_box chart and dashboardFilter related front-end logic to make sure didn't make unnecessary requests.

2. Now Dashboard owner should see how filter components work without much manual work. They should check if filter component generate correct filter value, if filter has right scope, etc,. This is to verify the logic of front-end migration valid and works well.

3. If Dashboard owner find some issues for the converted version, or any UI/UX feedback, concerns, they can report issue. Dashboard will keep intact no change.

4. If Dashboard owner do not want to verify right now, I also offer a snooze mode, which allow user set a period of time and Dashboard will not be converted when user open it. After the snooze time, dashboard will be automatically converted again.

5. If Dashboard owner like the converted dashboard, they can choose SAVE dashboard before they leave. After that this dashboard will have good native_filter_configuration in dashboard metadata. And when owner opens dashboard next time, we don't need to run front-end migration logic.

Note: There is a tricky part: when Dashboard owner is reviewing JS-converted filters, he/she probably can not save other changes without save native_filter_configuration. We should show clear confirm message to let owners know what content get saved.

6. Dashboard viewers (do not have can_edit permission for the dashboard), will not see JS-converted dashboard during transition mode. After dashboard owners saved the change, viewers can see and use filter components.

advanced users

Already enabled native filter feature flag at the beginning of migration. These users already manually created native filter in their dashboard so that they had native_filter_configuration metadata.

1. For these users, they already had filter components in the dashboard, but may also have some old filter_box. So they already knew how filter component work, and there is no data or functionality lost concerns. These users are like above user after Step 5.

2. Given they already had created valuable native_filter_configuration, we should not add automatic converted content into this metatdata.

3. Dashboard owners have to manually create filter components to replace their filter_box. In Transition mode, all filter_box will stop working in the dashboard (as long as dashboard has native_filter_configuration metadata).

DB Migration

After large number of dashboard filter_box are safely migrated to native filter components, we can move to DB Migration step.

During DB migration, there will be a python function that walk through all dashboards and change its metadata. It will do complete following jobs:

• Find dashboard without native_filter_configuration: find its filter_box config and convert them into native_filter_configuration, and saved into dashboard metadata.
• From dashboard_slices table, remove the all relationship entries for filter_box and dashboard.

Then we can remove all filter_box viz from slices table, remove front-end JS convert logic, and etc.

New or Changed Public Interfaces

• Convert filter_box config properties into dashboard native filter metadata.
• Dashboard will only offer interactive filtering functionalities by filter components.
• Remove filter_box charts from all dashboard.
• Expire filter_box viz_type from Superset system.

New dependencies

See Precondition section above for more details.

Migration Plan and Compatibility

Based on your dashboard complexity and filter component usage, corporation/company can optionally kickoff migration mode after this PR merged.

Migration mode is optional. In the end all filter_box used in the dashboard, will be converted into filter component by DB migration script (this will be created later by separated PR). And from then on Superset will not expire filter_box viz type in our codebase.

Rejected Alternatives

N/A

[amitmiran137]

I think that we should have it as an optional migration tool to run by demand, like a CLI command or a rest endpoint.
the reason for that is that dashboard owners need to decide on how to design their dashboard after the removal of a filter box.
the location of the filter box on the dashboard matters for them, so moving away from filter box changes how they design the look and feel of the dashboard

I can make sure that everyone migrates to the native-filters manually bc we don't have tons of dashboards yet!

[graceguo-supercat]

I think that we should have it as an optional migration tool to run by demand, like a CLI command or a rest endpoint.

@amitmiran137 I assume your request is to let each dashboard owner run the CLI command? What state is the dashboard currently in, and what state they want to achieve?
If they already had good native filter configuration, they can re-arrange dashboard layout any time. My proposal does not cover re-arrange dashboard layout solution, and i assume it must be done manually.

[michael-s-molina]

@graceguo-supercat If in the end, we’ll have a back-end migration script that converts a filter box configuration into a native filter configuration, why don’t we skip the part of creating the same logic on the front-end and just do it on the back-end? We can add to each automatically converted filter the status of approved or rejected (which will hide the filter) and offer an option to the owner to set this status on the interface. Something like this:

This way we can monitor rejected filters, migration completion, and the owner can save his dashboard without worrying about side effects. Only accepted filters will appear to non-owner users. This workflow also works for users that already have native filters created.

The owner can have both filter boxes and native filters at the same time for validation, and when it is validated, he can remove the filter box from the dashboard or we can do it on a pre-defined date.

[graceguo-supercat]

If in the end, we’ll have a back-end migration script that converts a filter box configuration into a native filter configuration, why don’t we skip the part of creating the same logic on the front-end and just do it on the back-end?

In airbnb our DS use Superset do critical jobs. In case any automatically convert didn't work as expected, or after convert to filter components, dashboard is too slow to use etc, we need to unblock our users ASAP so that at least, they can still use old filter_box to complete their work.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. For admin, please label this issue .pinned to prevent stale bot from closing the issue.

[jhult]

Should this stay open?