adds extract and normalize traces #839
Conversation
✅ Deploy Preview for dlt-hub-docs canceled.
|
dlt/common/pipeline.py
Outdated
| """ids of the loaded packages""" | ||
| load_packages: List[LoadPackageInfo] | ||
| """Information on loaded packages""" | ||
| started_at: datetime.datetime |
There was a problem hiding this comment.
This should have a finished_at entry, no? currently I am getting a value here for the load step I think, but not for the others.
There was a problem hiding this comment.
OK I will add it! for extract and normalize it is easy because those steps are atomic so I can just use now() as finished at
| "job_metrics": {job.job_id(): metrics for job, metrics in job_metrics.items()}, | ||
| "table_metrics": table_metrics, | ||
| "resource_metrics": resource_metrics, | ||
| "dag": source.resources.selected_dag, |
There was a problem hiding this comment.
so from this we can see which resources where selected during pipeline run, right? But the info which table stems from which resource is not in here, is it?
There was a problem hiding this comment.
you can get this information from the schema (there's resource tag for each table). also it won't be too hard to add this info.
I'd merge the stuff as it is. or make a clear request that you want it :) @sh-rp
|
@sh-rp I'll be adding more tests mostly for the shapes of the data to this PR |
rudolfix
force-pushed
the
rfix/adds-extract-normalize-traces
branch
from
ebff184 to
4a240ca
Compare
Description
Implements #754
Implements #757