Skip to content

Conversation

@sh-rp
Copy link
Collaborator

@sh-rp sh-rp commented Aug 5, 2025

Description

This PR improves the dashboard preview. It includes:

  • Forwarding the correct pipeline_dir to the marimo app from the dlt cli
  • Disables state syncing, promotes pipeline overview to top, shows all schemas and show remote state and remote schemas
  • Adds support for multiple schemas on one pipeline via dropdown that switches both schema explorer and data explorer
  • Adds two buttons to open pipeline folder and local data folder (for duckdb and local filesystems) in the filebrowser (explorer, finder ...)
  • Allows to start dashboard for so far unknown pipelines without errors and a helpful error message
  • Add refresh buttons in useful locations
  • Use raw unnormaliized queries in data explorer
  • Automatically reload page and content if update to current pipeline trace file is detected.
  • Show exception traces in pipeline overview and last trace section if last run failed
  • Display pipeline state as yaml instead of json
  • Adds resource and source state view to data panel
  • Connect to duckdb in read only mode when opening the ibis panel

ToDos

  • add basic e2e tests for all new features
  • add unit tests for new helper functions
  • add doc page
  • make marimo app the default dashboard app
  • review strings for clarity and up-to-dateness

@netlify
Copy link

netlify bot commented Aug 5, 2025

Deploy Preview for dlt-hub-docs ready!

Name Link
🔨 Latest commit ed21eee
🔍 Latest deploy log https://app.netlify.com/projects/dlt-hub-docs/deploys/689f47efd2938b0008ee607f
😎 Deploy Preview https://deploy-preview-2965--dlt-hub-docs.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

@sh-rp sh-rp force-pushed the feature/improve_dashboard branch from 80deb2a to a46c910 Compare August 6, 2025 16:18
@sh-rp sh-rp linked an issue Aug 11, 2025 that may be closed by this pull request
6 tasks
sh-rp added 26 commits August 12, 2025 14:28
add multi schema pipeline to fixtures
add buttons to open pipeline folder and local data folder if present
show pipeline error screen also when manually chnaing the pipeline name in the url
@sh-rp sh-rp force-pushed the feature/improve_dashboard branch from dc14b66 to 6c2a2e5 Compare August 12, 2025 12:29
@sh-rp sh-rp self-assigned this Aug 12, 2025
@sh-rp sh-rp marked this pull request as ready for review August 13, 2025 09:43
@djudjuu djudjuu self-requested a review August 15, 2025 13:36
Copy link
Collaborator

@zilto zilto left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few minor changes and notes for follow-up tickets. Excited to see this go live!

@@ -1,5 +1,5 @@
---
title: View data with Streamlit
title: The legacy streamlit app
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What about View data with Streamlit (legacy) ?

"""Returns a connected ibis backend for the dataset. Not implemented for all destinations.
Args:
read_only (bool): Whether to open the connection in read only mode. Only used for duckdb.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The read_only flag seems like a good addition, but why / how is it only for duckdb ?

Let's open a follow-up ticket to apply read_only to other backends. (If the backend doesn't support read only, we log a warning)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.


:::info
Make sure you have `streamlit` installed: `pip install streamlit`
Make sure you have `marimo` installed: `pip install marimo`
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should have a dlt[dashboard] extra in pyproject.toml. Let's open an issue with this follow-up task + documentation update

Copy link
Contributor

@djudjuu djudjuu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lg now

Copy link
Collaborator

@zilto zilto left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the fix!

@djudjuu djudjuu merged commit b75e4aa into devel Aug 15, 2025
37 checks passed
@zilto zilto added enhancement New feature or request release-highlight Changes to highlight in release notes labels Aug 15, 2025
@zilto zilto deleted the feature/improve_dashboard branch August 15, 2025 15:03
zilto pushed a commit that referenced this pull request Aug 26, 2025
* remove uneeded file

* fix forwarding of pipelines dir to marimo app

* disable state sync and display all schemas and remote state and schemas in pipeline overview

* add support for multiple schemas

* fix e2e tests, further updates pending

* use dropdown instead of multiselect for schema selection
add multi schema pipeline to fixtures

* add last run info in pipeline overview
add buttons to open pipeline folder and local data folder if present

* fix loads browser to select correct schema

* allow to start dashboard for a pipeline that is not there yet and add helpful error message in this case

* nicer last run time formatting
show pipeline error screen also when manually chnaing the pipeline name in the url

* move buttons to top, add refresh buttons to sections

* use raw query when constructing queries

* lazy load remote state tab

* fix traces and trace typing (mostly)

* add exception traces to ui

* add file watcher

* remove test code

* add source and resource state viewer to data panel

* update existing unit tests

* add unit test for new utils

* make marimo dashboard the default app for pipeline show

* update docs

* update existing e2e tests for new yaml based rendering of state

* move streamlit app down in sidebar

* grammar fixes for dashboard strings

* open duckdb in readme mode in datapanel in dashboard

* remove old tests
re-enable dashboard main command

* add missing args to dashboard command

* small fixes to e2e tests

* add tests for exceptions

* re-organize e2e tests into invidual tests

* add basic schema selection checks

* improve dashboard help and dashboard docs page

* short some strings in testing to make selecting predictable

* merge devel

* typo

---------

Co-authored-by: djudjuu <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request release-highlight Changes to highlight in release notes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Marimo Pipeline Dashboard App Beta Release Checklist

4 participants