Skip to content

Conversation

@anuunchin
Copy link
Contributor

@anuunchin anuunchin commented Oct 24, 2025

This PR introduces a new pipeline run section to the dashboard app which uses a simple html snippet.

Screenshot 2025-11-03 at 12 04 30

@anuunchin anuunchin force-pushed the feat/pipeline-run-dashboard branch from 8054e89 to 21c3dd6 Compare October 24, 2025 14:39
@cloudflare-workers-and-pages
Copy link

cloudflare-workers-and-pages bot commented Oct 24, 2025

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Preview URL Updated (UTC)
✅ Deployment successful!
View logs
docs e1a163c Commit Preview URL

Branch Preview URL
Nov 14 2025, 06:25 PM

@anuunchin anuunchin self-assigned this Oct 24, 2025
@anuunchin anuunchin force-pushed the feat/pipeline-run-dashboard branch 3 times, most recently from 65991c7 to b029cfa Compare October 28, 2025 14:45
@rudolfix
Copy link
Collaborator

terminology (from https://www.notion.so/dlthub/dlt-Entities-their-Lifecycles-and-Relations-2679fb8e23cf80169921c4ea069bfff8?source=copy_link#2679fb8e23cf8109b9c3eee928a2a377). definitions are pretty clear and you can easily refer to code

  • pipeline run succeeded or failed (correct your status)

Idea display status of load package(s) as boxes like the "COMPLETED"

  • pipeline created load packages: collect packages from all steps to get load id and package
  • display color coded states of all packages (with load ids on it): pending | loaded | aborted
  • package can be "partially loaded" (I think in pending and aborted states). there's a method on package info. I hope we have it in trace. if you see it - it should be a red color ie: "load_id: partially loaded in aborted"

schema indicators

  • look for "applied schema update" if it is found in the trace - display "migration" box

for later

  • detect state sync (loaded job with state table name)
  • detect dev_mode (I do not think we have it in trace but we could use dataset name)
  • detect full refresh type (I do not think we hace it in trace)

feedback to current UI
experiment: if we have trace just put it on top, without "Pipeline Run" and switch. Also you do not need to repeat pipeline name, it is already on top

@anuunchin anuunchin marked this pull request as ready for review November 3, 2025 10:53
@anuunchin anuunchin requested a review from rudolfix November 3, 2025 10:53
@anuunchin anuunchin force-pushed the feat/pipeline-run-dashboard branch from 878eba9 to c9e6bda Compare November 3, 2025 11:03
@anuunchin anuunchin force-pushed the feat/pipeline-run-dashboard branch from c9e6bda to 60307b4 Compare November 4, 2025 07:54
Copy link
Collaborator

@rudolfix rudolfix left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. all the comments are minor. visually is really good.

you could add example pipeline with partially loaded package:

@dlt.destination
    def wait_forever(item, schema):
        # never exit
        from time import sleep

        # this one does not wake up on signal
        while not _done:
            sleep(1)

        # make the job fail if it gets here
        raise KeyboardInterrupt()

    pipeline = dlt.pipeline(
        "signal_waiter", destination=wait_forever(), dataset_name="_data", progress=DictCollector()
    )

you can modify this destination so it passes for one table but raises for another triggering "partially loaded" state.

failed: bool


def _format_duration(ms: float) -> str:
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we have humanize library. it should be able to generate friendly strings. ie humanize.precisedelta pls take a look

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

humanize doesn't seem to be generating friendly enough strings - I think it's better to leave it the way it is currently 👀

@anuunchin anuunchin force-pushed the feat/pipeline-run-dashboard branch 3 times, most recently from f6e713a to e06d81b Compare November 10, 2025 10:06
@anuunchin anuunchin force-pushed the feat/pipeline-run-dashboard branch from e06d81b to 6550200 Compare November 10, 2025 10:27
Copy link
Collaborator

@rudolfix rudolfix left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@rudolfix rudolfix merged commit e7e54b2 into devel Nov 14, 2025
70 of 72 checks passed
@rudolfix rudolfix deleted the feat/pipeline-run-dashboard branch November 14, 2025 18:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants