fix(assets): use manifest parent_map for accurate lineage by fdelbrayelle · Pull Request #240 · kestra-io/plugin-dbt

fdelbrayelle · 2026-03-04T14:19:13Z

Summary

Use the manifest-level parent_map (the canonical DAG representation) to resolve asset dependencies instead of node-level depends_on.nodes, which can include extra edges not reflecting the real dbt DAG structure
Falls back to depends_on.nodes for older manifests that don't include parent_map
Adds parentMap field to the Manifest model

Test plan

Existing tests updated with parent_map data and still pass
New test (shouldUseParentMapForLineage) verifies parent_map takes precedence over depends_on.nodes when they differ

Fixes #239

Use the manifest-level parent_map (the canonical DAG) to resolve asset dependencies instead of node-level depends_on.nodes which can include extra edges not reflecting the real dbt DAG structure.

github-actions · 2026-03-04T14:25:55Z

📦 Artifacts

Name	Size	Updated	Expiration
	208.48 KB	Mar 4, 26, 4:19:30 PM UTC	Mar 11, 26, 4:19:31 PM UTC

🧪 Java Unit Tests

	Tests	Passed ✅	Skipped ⚠️	Failed	Time ⏱
Java Tests Report	26 ran	25 ✅	1 ⚠️	0 ❌	5m 12s 734ms

🔁 Unreleased Commits

✅ No unreleased commits found.

Ben8t · 2026-03-04T14:32:15Z

I think you need to double-check, doesn't seem to fully solve

fdelbrayelle · 2026-03-04T15:39:45Z

Here is what I have in the DB for assets:

[
  {
    "key": "fde_memory.main.int_customer_orders",
    "value": {
      "id": "memory.main.int_customer_orders",
      "name": "int_customer_orders",
      "type": "io.kestra.plugin.ee.assets.Table",
      "schema": "main",
      "system": "duckdb",
      "created": "2026-03-04T14:55:24.792885Z",
      "deleted": false,
      "updated": "2026-03-04T14:55:24.962814Z",
      "database": "memory",
      "metadata": {
        "name": "int_customer_orders",
        "schema": "main",
        "system": "duckdb",
        "database": "memory"
      },
      "tenantId": "fde"
    },
    "tenant_id": "fde",
    "namespace": null,
    "id": "memory.main.int_customer_orders",
    "type": "io.kestra.plugin.ee.assets.Table",
    "display_name": null,
    "created": "2026-03-04 14:55:24.792885 +00:00",
    "updated": "2026-03-04 14:55:24.962814 +00:00",
    "deleted": false,
    "fulltext": null
  },
  {
    "key": "fde_memory.main.my_second_dbt_model",
    "value": {
      "id": "memory.main.my_second_dbt_model",
      "name": "my_second_dbt_model",
      "type": "io.kestra.plugin.ee.assets.Table",
      "schema": "main",
      "system": "duckdb",
      "created": "2026-03-04T14:55:24.792922Z",
      "deleted": false,
      "updated": "2026-03-04T14:55:24.985039Z",
      "database": "memory",
      "metadata": {
        "name": "my_second_dbt_model",
        "schema": "main",
        "system": "duckdb",
        "database": "memory"
      },
      "tenantId": "fde"
    },
    "tenant_id": "fde",
    "namespace": null,
    "id": "memory.main.my_second_dbt_model",
    "type": "io.kestra.plugin.ee.assets.Table",
    "display_name": null,
    "created": "2026-03-04 14:55:24.792922 +00:00",
    "updated": "2026-03-04 14:55:24.985039 +00:00",
    "deleted": false,
    "fulltext": null
  },
  {
    "key": "fde_memory.main.stg_customers",
    "value": {
      "id": "memory.main.stg_customers",
      "name": "stg_customers",
      "type": "io.kestra.plugin.ee.assets.Table",
      "schema": "main",
      "system": "duckdb",
      "created": "2026-03-04T14:55:24.792930Z",
      "deleted": false,
      "updated": "2026-03-04T14:55:25.004978Z",
      "database": "memory",
      "metadata": {
        "name": "stg_customers",
        "schema": "main",
        "system": "duckdb",
        "database": "memory"
      },
      "tenantId": "fde"
    },
    "tenant_id": "fde",
    "namespace": null,
    "id": "memory.main.stg_customers",
    "type": "io.kestra.plugin.ee.assets.Table",
    "display_name": null,
    "created": "2026-03-04 14:55:24.792930 +00:00",
    "updated": "2026-03-04 14:55:25.004978 +00:00",
    "deleted": false,
    "fulltext": null
  },
  {
    "key": "fde_memory.main.int_order_payments",
    "value": {
      "id": "memory.main.int_order_payments",
      "name": "int_order_payments",
      "type": "io.kestra.plugin.ee.assets.Table",
      "schema": "main",
      "system": "duckdb",
      "created": "2026-03-04T14:55:24.792950Z",
      "deleted": false,
      "updated": "2026-03-04T14:55:25.025884Z",
      "database": "memory",
      "metadata": {
        "name": "int_order_payments",
        "schema": "main",
        "system": "duckdb",
        "database": "memory"
      },
      "tenantId": "fde"
    },
    "tenant_id": "fde",
    "namespace": null,
    "id": "memory.main.int_order_payments",
    "type": "io.kestra.plugin.ee.assets.Table",
    "display_name": null,
    "created": "2026-03-04 14:55:24.792950 +00:00",
    "updated": "2026-03-04 14:55:25.025884 +00:00",
    "deleted": false,
    "fulltext": null
  },
  {
    "key": "fde_memory.main.fct_revenue_by_customer",
    "value": {
      "id": "memory.main.fct_revenue_by_customer",
      "name": "fct_revenue_by_customer",
      "type": "io.kestra.plugin.ee.assets.Table",
      "schema": "main",
      "system": "duckdb",
      "created": "2026-03-04T14:55:24.792968Z",
      "deleted": false,
      "updated": "2026-03-04T14:55:25.044774Z",
      "database": "memory",
      "metadata": {
        "name": "fct_revenue_by_customer",
        "schema": "main",
        "system": "duckdb",
        "database": "memory"
      },
      "tenantId": "fde"
    },
    "tenant_id": "fde",
    "namespace": null,
    "id": "memory.main.fct_revenue_by_customer",
    "type": "io.kestra.plugin.ee.assets.Table",
    "display_name": null,
    "created": "2026-03-04 14:55:24.792968 +00:00",
    "updated": "2026-03-04 14:55:25.044774 +00:00",
    "deleted": false,
    "fulltext": null
  },
  {
    "key": "fde_memory.main.stg_payments",
    "value": {
      "id": "memory.main.stg_payments",
      "name": "stg_payments",
      "type": "io.kestra.plugin.ee.assets.Table",
      "schema": "main",
      "system": "duckdb",
      "created": "2026-03-04T14:55:24.793104Z",
      "deleted": false,
      "updated": "2026-03-04T14:55:25.066339Z",
      "database": "memory",
      "metadata": {
        "name": "stg_payments",
        "schema": "main",
        "system": "duckdb",
        "database": "memory"
      },
      "tenantId": "fde"
    },
    "tenant_id": "fde",
    "namespace": null,
    "id": "memory.main.stg_payments",
    "type": "io.kestra.plugin.ee.assets.Table",
    "display_name": null,
    "created": "2026-03-04 14:55:24.793104 +00:00",
    "updated": "2026-03-04 14:55:25.066339 +00:00",
    "deleted": false,
    "fulltext": null
  },
  {
    "key": "fde_memory.main.fct_customer_summary",
    "value": {
      "id": "memory.main.fct_customer_summary",
      "name": "fct_customer_summary",
      "type": "io.kestra.plugin.ee.assets.Table",
      "schema": "main",
      "system": "duckdb",
      "created": "2026-03-04T14:55:24.793126Z",
      "deleted": false,
      "updated": "2026-03-04T14:55:25.091688Z",
      "database": "memory",
      "metadata": {
        "name": "fct_customer_summary",
        "schema": "main",
        "system": "duckdb",
        "database": "memory"
      },
      "tenantId": "fde"
    },
    "tenant_id": "fde",
    "namespace": null,
    "id": "memory.main.fct_customer_summary",
    "type": "io.kestra.plugin.ee.assets.Table",
    "display_name": null,
    "created": "2026-03-04 14:55:24.793126 +00:00",
    "updated": "2026-03-04 14:55:25.091688 +00:00",
    "deleted": false,
    "fulltext": null
  },
  {
    "key": "fde_memory.main.int_daily_revenue",
    "value": {
      "id": "memory.main.int_daily_revenue",
      "name": "int_daily_revenue",
      "type": "io.kestra.plugin.ee.assets.Table",
      "schema": "main",
      "system": "duckdb",
      "created": "2026-03-04T14:55:24.793142Z",
      "deleted": false,
      "updated": "2026-03-04T14:55:25.115897Z",
      "database": "memory",
      "metadata": {
        "name": "int_daily_revenue",
        "schema": "main",
        "system": "duckdb",
        "database": "memory"
      },
      "tenantId": "fde"
    },
    "tenant_id": "fde",
    "namespace": null,
    "id": "memory.main.int_daily_revenue",
    "type": "io.kestra.plugin.ee.assets.Table",
    "display_name": null,
    "created": "2026-03-04 14:55:24.793142 +00:00",
    "updated": "2026-03-04 14:55:25.115897 +00:00",
    "deleted": false,
    "fulltext": null
  },
  {
    "key": "fde_memory.main.my_first_dbt_model",
    "value": {
      "id": "memory.main.my_first_dbt_model",
      "name": "my_first_dbt_model",
      "type": "io.kestra.core.models.assets.External",
      "schema": "main",
      "system": "duckdb",
      "created": "2026-03-04T14:55:24.793149Z",
      "deleted": false,
      "updated": "2026-03-04T14:55:25.143509Z",
      "database": "memory",
      "metadata": {
        "name": "my_first_dbt_model",
        "schema": "main",
        "system": "duckdb",
        "database": "memory"
      },
      "tenantId": "fde"
    },
    "tenant_id": "fde",
    "namespace": null,
    "id": "memory.main.my_first_dbt_model",
    "type": "io.kestra.core.models.assets.External",
    "display_name": null,
    "created": "2026-03-04 14:55:24.793149 +00:00",
    "updated": "2026-03-04 14:55:25.143509 +00:00",
    "deleted": false,
    "fulltext": null
  },
  {
    "key": "fde_memory.main.stg_orders",
    "value": {
      "id": "memory.main.stg_orders",
      "name": "stg_orders",
      "type": "io.kestra.core.models.assets.External",
      "schema": "main",
      "system": "duckdb",
      "created": "2026-03-04T14:55:24.793153Z",
      "deleted": false,
      "updated": "2026-03-04T14:55:25.169004Z",
      "database": "memory",
      "metadata": {
        "name": "stg_orders",
        "schema": "main",
        "system": "duckdb",
        "database": "memory"
      },
      "tenantId": "fde"
    },
    "tenant_id": "fde",
    "namespace": null,
    "id": "memory.main.stg_orders",
    "type": "io.kestra.core.models.assets.External",
    "display_name": null,
    "created": "2026-03-04 14:55:24.793153 +00:00",
    "updated": "2026-03-04 14:55:25.169004 +00:00",
    "deleted": false,
    "fulltext": null
  }
]

And for asset_lineage_events:

[
  {
    "key": "3u6NNEC1zDzar3Xh6NqFEO",
    "value": {
      "uid": "3u6NNEC1zDzar3Xh6NqFEO",
      "state": "SUCCESS",
      "flowId": "dbt_build_duckdb",
      "inputs": [
        {
          "id": "memory.main.stg_customers",
          "type": "io.kestra.plugin.ee.assets.Table",
          "tenantId": "fde"
        },
        {
          "id": "memory.main.stg_orders",
          "type": "io.kestra.plugin.ee.assets.Table",
          "tenantId": "fde"
        },
        {
          "id": "memory.main.my_first_dbt_model",
          "type": "io.kestra.plugin.ee.assets.Table",
          "tenantId": "fde"
        },
        {
          "id": "memory.main.stg_orders",
          "type": "io.kestra.plugin.ee.assets.Table",
          "tenantId": "fde"
        },
        {
          "id": "memory.main.stg_payments",
          "type": "io.kestra.plugin.ee.assets.Table",
          "tenantId": "fde"
        },
        {
          "id": "memory.main.fct_customer_summary",
          "type": "io.kestra.plugin.ee.assets.Table",
          "tenantId": "fde"
        },
        {
          "id": "memory.main.int_daily_revenue",
          "type": "io.kestra.plugin.ee.assets.Table",
          "tenantId": "fde"
        },
        {
          "id": "memory.main.int_customer_orders",
          "type": "io.kestra.plugin.ee.assets.Table",
          "tenantId": "fde"
        },
        {
          "id": "memory.main.int_order_payments",
          "type": "io.kestra.plugin.ee.assets.Table",
          "tenantId": "fde"
        },
        {
          "id": "memory.main.stg_orders",
          "type": "io.kestra.plugin.ee.assets.Table",
          "tenantId": "fde"
        },
        {
          "id": "memory.main.stg_payments",
          "type": "io.kestra.plugin.ee.assets.Table",
          "tenantId": "fde"
        }
      ],
      "taskId": "dbt_build",
      "created": "2026-03-04T14:55:24.866545Z",
      "endDate": "2026-03-04T14:55:24.841278Z",
      "outputs": [
        {
          "id": "memory.main.int_customer_orders",
          "type": "io.kestra.plugin.ee.assets.Table",
          "tenantId": "fde"
        },
        {
          "id": "memory.main.my_second_dbt_model",
          "type": "io.kestra.plugin.ee.assets.Table",
          "tenantId": "fde"
        },
        {
          "id": "memory.main.stg_customers",
          "type": "io.kestra.plugin.ee.assets.Table",
          "tenantId": "fde"
        },
        {
          "id": "memory.main.int_order_payments",
          "type": "io.kestra.plugin.ee.assets.Table",
          "tenantId": "fde"
        },
        {
          "id": "memory.main.fct_revenue_by_customer",
          "type": "io.kestra.plugin.ee.assets.Table",
          "tenantId": "fde"
        },
        {
          "id": "memory.main.stg_payments",
          "type": "io.kestra.plugin.ee.assets.Table",
          "tenantId": "fde"
        },
        {
          "id": "memory.main.fct_customer_summary",
          "type": "io.kestra.plugin.ee.assets.Table",
          "tenantId": "fde"
        },
        {
          "id": "memory.main.int_daily_revenue",
          "type": "io.kestra.plugin.ee.assets.Table",
          "tenantId": "fde"
        },
        {
          "id": "memory.main.my_first_dbt_model",
          "type": "io.kestra.plugin.ee.assets.Table",
          "tenantId": "fde"
        },
        {
          "id": "memory.main.stg_orders",
          "type": "io.kestra.plugin.ee.assets.Table",
          "tenantId": "fde"
        }
      ],
      "tenantId": "fde",
      "namespace": "acme.team",
      "startDate": "2026-03-04T14:54:59.164350Z",
      "taskRunId": "3eObZMTDCL4DeGNVe110hD",
      "executionId": "11DXfweOmd0PYlH0N9w4q0",
      "flowRevision": 1
    },
    "tenant_id": "fde",
    "namespace": "acme.team",
    "flow_id": "dbt_build_duckdb",
    "flow_revision": 1,
    "execution_id": "11DXfweOmd0PYlH0N9w4q0",
    "task_id": "dbt_build",
    "task_run_id": "3eObZMTDCL4DeGNVe110hD",
    "created": "2026-03-04 14:55:24.866545 +00:00"
  }
]

fdelbrayelle · 2026-03-05T10:19:52Z

🗒️ Following meeting with @Ben8t & @brian-mulier-p.

This PR can be merged because it relies on parent instead of depends of but next steps may be:

use dynamic tasks for plugin-dbt: emit task_id for every dbt model OR maybe thing about using something like dbt run --model int.*
think about adding a dedicated simple topology UI for now
then maybe include asset lineage in the topology UI

All of this should be discussed again in a discovery phase before the next cycle with @kestra-io/products.

fix(assets): use manifest parent_map for accurate lineage

344e464

Use the manifest-level parent_map (the canonical DAG) to resolve asset dependencies instead of node-level depends_on.nodes which can include extra edges not reflecting the real dbt DAG structure.

fdelbrayelle self-assigned this Mar 4, 2026

kestrabot Bot added this to Pull Requests Mar 4, 2026

github-project-automation Bot moved this to To review in Pull Requests Mar 4, 2026

fdelbrayelle requested a review from Malaydewangan09 March 4, 2026 14:19

feat(assets): ensure lineage from the example is correct

93ff8a4

fix(assets): properly set inputs & outputs

a5fa3a0

fdelbrayelle force-pushed the issues/239 branch from 7228b3c to a5fa3a0 Compare March 4, 2026 16:13

Malaydewangan09 approved these changes Mar 5, 2026

View reviewed changes

fdelbrayelle merged commit 8b6a420 into main Mar 5, 2026
6 checks passed

fdelbrayelle deleted the issues/239 branch March 5, 2026 10:28

github-project-automation Bot moved this from To review to Done in Pull Requests Mar 5, 2026

github-actions Bot mentioned this pull request Mar 9, 2026

fix: silently skip asset emission on OSS edition #247

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix(assets): use manifest parent_map for accurate lineage#240

fix(assets): use manifest parent_map for accurate lineage#240
fdelbrayelle merged 3 commits intomainfrom
issues/239

fdelbrayelle commented Mar 4, 2026

Uh oh!

github-actions Bot commented Mar 4, 2026 •

edited

Loading

Uh oh!

Ben8t commented Mar 4, 2026

Uh oh!

fdelbrayelle commented Mar 4, 2026

Uh oh!

fdelbrayelle commented Mar 5, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

fdelbrayelle commented Mar 4, 2026

Summary

Test plan

Uh oh!

github-actions Bot commented Mar 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

📦 Artifacts

🧪 Java Unit Tests

🔁 Unreleased Commits

Uh oh!

Ben8t commented Mar 4, 2026

Uh oh!

fdelbrayelle commented Mar 4, 2026

Uh oh!

fdelbrayelle commented Mar 5, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

github-actions Bot commented Mar 4, 2026 •

edited

Loading