chore: Extensions architecture POC #31934

michael-s-molina · 2025-01-20T19:24:35Z

SUMMARY

This PR introduces new packages and functionality for introducing extensions as proposed in SIP #34162. Example extensions that were used for validating the system have been moved to the following draft PR: #34726

This PR introduces the following packages:

superset-frontend/packages/superset-core
Our package with UI components, utility functions, types, and APIs used by extensions.

superset-core
The host APIs used by backend extensions.

superset-cli
CLI commands to build, bundle and hot-rebuild extensions.

superset-extensions build     Builds extension assets.

superset-extensions bundle    Packages the extension into a .supx file.

superset-extensions dev       Provides hot-rebuilding of assets.

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

The Dataset references extension displays relevant information for SQL authors in SQL Lab:

Datasets referenced in a SQL Query: This helps users discover the origin of the data, especially for long SQL queries.
Owners of each referenced dataset: This is crucial for dealing with access permissions or quickly finding out who to contact in case of problems.
Latest available partition: This is important for understanding data completeness issues.
Estimated row count for each dataset: This helps users grasp the implications of joins.

Screen.Recording.2025-06-04.at.10.48.32.mov

TESTING INSTRUCTIONS

1 - Set the ENABLE_EXTENSIONS feature flag to True.

2 - Run npm ci on superset-frontend.

3 - Install the development dependencies by running pip install -r requirements/development.txt. This will pull in the new cli package that contains the superset-extensions command.

4 - Bundle an extension by going into its root folder and executing superset-extensions bundle. This will build the extension and generate a .supx bundle of the extension.

5 - Upload the .supx file to Superset using Swagger and the extensions endpoint.

6 - Go to SQL -> SQL Lab and check the extension.

For development mode:

1 - Execute steps 1, 2 and 3 above.

2 - Configure Superset's LOCAL_EXTENSIONS variable like:

LOCAL_EXTENSIONS = [
    "./extensions/dataset_references",
]

3 - Run in development mode by going into the extension root folder and executing superset-extensions dev. This will trigger hot-rebuilding of your frontend assets. Backend assets are currently reloaded by the default reloading mechanisms of Superset using superset run --reload.

4 - Go to SQL -> SQL Lab and check the extension.

ADDITIONAL INFORMATION

Has associated issue:
Required feature flags:
Changes UI
Includes DB Migration (follow approval process in SIP-59)
- Migration is atomic, supports rollback & is backwards-compatible
- Confirm DB migration upgrade and downgrade tested
- Runtime estimates and downtime expectations provided
Introduces new feature or API
Removes existing feature or API

korbit-ai · 2025-01-20T19:24:38Z

Based on your review schedule, I'll hold off on reviewing this PR until it's marked as ready for review. If you'd like me to take a look now, comment /korbit-review.

Your admin can change your review schedule in the Korbit Console

codecov · 2025-01-20T19:29:18Z

Codecov Report

❌ Patch coverage is 18.33741% with 334 lines in your changes missing coverage. Please review.
✅ Project coverage is 72.42%. Comparing base (76d897e) to head (286cc15).
⚠️ Report is 2389 commits behind head on master.

Files with missing lines	Patch %	Lines
superset/extensions/utils.py	0.00%	121 Missing ⚠️
superset/extensions/api.py	0.00%	57 Missing ⚠️
superset/extensions/local_extensions_watcher.py	26.31%	42 Missing ⚠️
superset/extensions/discovery.py	0.00%	30 Missing ⚠️
superset/initialization/__init__.py	25.00%	22 Missing and 2 partials ⚠️
superset/core/api/types/models.py	51.72%	14 Missing ⚠️
superset/extensions/exceptions.py	0.00%	14 Missing ⚠️
superset/extensions/types.py	0.00%	14 Missing ⚠️
superset/daos/base.py	35.71%	9 Missing ⚠️
superset/core/api/types/rest_api.py	61.53%	5 Missing ⚠️
... and 3 more

Additional details and impacted files

@@             Coverage Diff             @@
##           master   #31934       +/-   ##
===========================================
+ Coverage   60.48%   72.42%   +11.93%     
===========================================
  Files        1931      584     -1347     
  Lines       76236    42428    -33808     
  Branches     8568     4475     -4093     
===========================================
- Hits        46114    30728    -15386     
+ Misses      28017    10516    -17501     
+ Partials     2105     1184      -921

Flag	Coverage Δ
hive	`46.89% <18.33%> (-2.27%)`	⬇️
javascript	`?`
mysql	`71.45% <18.33%> (?)`
postgres	`71.50% <18.33%> (?)`
presto	`50.58% <18.33%> (-3.22%)`	⬇️
python	`72.38% <18.33%> (+8.88%)`	⬆️
sqlite	`71.07% <18.33%> (?)`
unit	`100.00% <ø> (+42.36%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

superset/migrations/versions/2025-03-07_08-35_d4d5115bdeb0_add_extensions_table.py

superset/extensions/api.py

korbit-ai · 2025-08-22T17:29:41Z

Korbit doesn't automatically review large (3000+ lines changed) pull requests such as this one. If you want me to review anyway, use /korbit-review.

michael-s-molina · 2025-08-22T19:14:08Z

mistercrunch · 2025-08-22T20:01:32Z

love the idea of general organization this way

superset/
  {feature_or_subject_area}/models.py
  {feature_or_subject_area}/daos.py
  {feature_or_subject_area}/api.py
  {feature_or_subject_area}/views.py

Super great but super disruptive and merge conflicting. Some things are kind of going that way area, you could go one-per-PR.

We'd need to identify the set of feature_or_subject_area, some might be pretty tangled up with one-another ... In practice might be hard to segment

villebro · 2025-08-22T20:43:53Z

love the idea of general organization this way
superset/
  {feature_or_subject_area}/models.py
  {feature_or_subject_area}/daos.py
  {feature_or_subject_area}/api.py
  {feature_or_subject_area}/views.py
Super great but super disruptive and merge conflicting. Some things are kind of going that way area, you could go one-per-PR.

We'd need to identify the set of feature_or_subject_area, some might be pretty tangled up with one-another ... In practice might be hard to segment

Totally, I've also been pushing for reorganizing the backend like this. Commands should also go directly under the main feature/subject area. I believe we actually already had commands and DAOs under the main feature before, but it was moved in under a centralized package at some point. I propose doing this massive refactor next time the breaking window opens to clean up the backend, as it will make things much more modular.

michael-s-molina · 2025-08-23T00:33:48Z

mistercrunch · 2025-08-30T01:07:04Z

superset-frontend/package.json

    "last 3 edge versions"
  ],
  "dependencies": {
+    "@apache-superset/core": "file:packages/superset-core",


I think this is creating some troubles for me. UI won't boot up...

What I know so far...

nuking node_modules, npm run prune, restarting docker, ...

after this PR, running npm install changes the package-lock.json

I can only get the UI to boot up on SHAs before this PR

building before/after this in master, works before, fails after

npm install changes the package-lock.json

@mistercrunch

npm install may change the lock file because it installs new or updated dependencies, keeping the project up to date during development.

npm ci never changes the lock file, ensuring that exactly what’s specified in the lock file is installed for guaranteed consistency, especially in CI environments.

@mistercrunch I opened #34924 to update the referenced version of @apache-superset/core. I tested both dev and built modes:

superset init npm ci npm run build (serving from port 8088 OK) npm run dev-server (serving from port 9000 OK)

superset-frontend/tsconfig.json

Co-authored-by: Ville Brofeldt <[email protected]> Co-authored-by: Ville Brofeldt <[email protected]> Co-authored-by: Ville Brofeldt <[email protected]> (cherry picked from commit a8be5a5)

pull-request-size bot added the size/L label Jan 20, 2025

michael-s-molina added hold! On hold extensions SIPs related to the extensions project review:draft labels Jan 20, 2025

michael-s-molina changed the title ~~chore: Plugins architecture POC~~ chore: Extensions architecture POC Feb 17, 2025

michael-s-molina force-pushed the plugin-architecture-poc branch from 904fbd9 to 871787f Compare February 17, 2025 12:34

pull-request-size bot added size/XS and removed size/L labels Feb 20, 2025

github-actions bot added the api Related to the REST API label Feb 20, 2025

pull-request-size bot added size/L and removed size/XS labels Feb 20, 2025

pull-request-size bot added size/XXL and removed size/L labels Mar 6, 2025

github-actions bot added dependencies:npm packages labels Mar 6, 2025

michael-s-molina force-pushed the plugin-architecture-poc branch from 3cde2af to f766cc4 Compare March 7, 2025 16:14

github-actions bot added risk:db-migration PRs that require a DB migration plugins labels Mar 8, 2025

villebro force-pushed the plugin-architecture-poc branch from 496c566 to 76a4e83 Compare March 11, 2025 19:57

michael-s-molina commented Mar 12, 2025

View reviewed changes

superset/migrations/versions/2025-03-07_08-35_d4d5115bdeb0_add_extensions_table.py Outdated Show resolved Hide resolved

michael-s-molina force-pushed the plugin-architecture-poc branch 3 times, most recently from 870c262 to 9f0fbc0 Compare March 18, 2025 13:18

github-actions bot removed the plugins label Mar 18, 2025

michael-s-molina commented Mar 20, 2025

View reviewed changes

superset/extensions/api.py Outdated Show resolved Hide resolved

michael-s-molina force-pushed the plugin-architecture-poc branch 2 times, most recently from 812f875 to e34b695 Compare March 28, 2025 14:18

michael-s-molina mentioned this pull request Apr 2, 2025

feat(charts): chart publishing UI #32969

Open

7 tasks

michael-s-molina requested review from eschutho, rusackas, betodealmeida, nytai, mistercrunch, craig-rueda, kgabryje and dpgaspar as code owners August 22, 2025 17:29

Ville Brofeldt and others added 8 commits August 22, 2025 10:51

Merge branch 'master' into plugin-architecture-poc

7c2370f

remove checksum

38480ba

fix test

746d843

fix gha workflow

238b221

fix CODEOWNERS

18387cb

unpin python version

38bc5be

add docstrings to core api

366984d

Use tab history for id

94201ac

Ville Brofeldt added 2 commits August 22, 2025 13:08

remove extensions

affbb54

add python matrix to cli test workflow

286cc15

villebro approved these changes Aug 22, 2025

View reviewed changes

rusackas approved these changes Aug 23, 2025

View reviewed changes

michael-s-molina merged commit a8be5a5 into apache:master Aug 23, 2025
63 checks passed

mistercrunch reviewed Aug 30, 2025

View reviewed changes

superset-frontend/tsconfig.json Show resolved Hide resolved

msyavuz removed the review:draft label Sep 19, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

chore: Extensions architecture POC #31934

chore: Extensions architecture POC #31934

Uh oh!

michael-s-molina commented Jan 20, 2025 •

edited by villebro

Loading

Uh oh!

korbit-ai bot commented Jan 20, 2025

Uh oh!

codecov bot commented Jan 20, 2025 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

korbit-ai bot commented Aug 22, 2025

Uh oh!

michael-s-molina commented Aug 22, 2025

Uh oh!

mistercrunch commented Aug 22, 2025

Uh oh!

villebro commented Aug 22, 2025

Uh oh!

Uh oh!

michael-s-molina commented Aug 23, 2025

Uh oh!

mistercrunch Aug 30, 2025

Uh oh!

mistercrunch Aug 30, 2025 •

edited

Loading

Uh oh!

michael-s-molina Sep 1, 2025

Uh oh!

michael-s-molina Sep 1, 2025

Uh oh!

Uh oh!

Uh oh!

chore: Extensions architecture POC #31934

chore: Extensions architecture POC #31934

Uh oh!

Conversation

michael-s-molina commented Jan 20, 2025 • edited by villebro Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

SUMMARY

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

TESTING INSTRUCTIONS

ADDITIONAL INFORMATION

Uh oh!

korbit-ai bot commented Jan 20, 2025

Uh oh!

codecov bot commented Jan 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

korbit-ai bot commented Aug 22, 2025

Uh oh!

michael-s-molina commented Aug 22, 2025

Uh oh!

mistercrunch commented Aug 22, 2025

Uh oh!

villebro commented Aug 22, 2025

Uh oh!

Uh oh!

michael-s-molina commented Aug 23, 2025

Uh oh!

mistercrunch Aug 30, 2025

Choose a reason for hiding this comment

Uh oh!

mistercrunch Aug 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

michael-s-molina Sep 1, 2025

Choose a reason for hiding this comment

Uh oh!

michael-s-molina Sep 1, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

michael-s-molina commented Jan 20, 2025 •

edited by villebro

Loading

codecov bot commented Jan 20, 2025 •

edited

Loading

mistercrunch Aug 30, 2025 •

edited

Loading