patch/json-bigquery-compatibility #103

fivetran-joemarkiewicz · 2025-07-02T15:21:39Z

PR Overview

Package version introduced in this PR: v0.18.1

This PR addresses the following Issue/Feature(s): Shopify Issue #105

Summary of changes:

Introduces JSON BigQuery datatype support for the impacted models/columns.

Submission Checklist

Alignment meeting with the reviewer (if needed)
- Timeline and validation requirements discussed
Provide validation details:
- Validation Steps: Check for unintentional effects (e.g., add/run consistency & integrity tests)
  - Confirmed datatype JSON datatypes for BQ are converted back to string and there is no change for non JSON datatypes.
- Testing Instructions: Confirm the change addresses the issue(s)
  - BK tests updated to test JSON and non JSON versions.
  - Also confirmed downstream transform package works as expected per the passing BK in this branch that will not be merged.
- Focus Areas: Complex logic or queries that need extra attention
  - Confirm the datatype is converting properly, but also not converting when it doesn't need to.

Changelog

Draft changelog for PR
Final changelog for release review

fivetran-catfritz

Functionally everything is looking good. However before I approve, curious what you think about my suggestions.

models/stg_shopify__collection.sql

macros/json_to_string.sql

integration_tests/dbt_project.yml

fivetran-catfritz · 2025-07-03T20:59:50Z

.buildkite/scripts/run_models.sh

+if [ "$db" = "bigquery" ]; then
+dbt run --vars '{shopify_collection_identifier: shopify_collection_data_bq_json, shopify_order_identifier: shopify_order_bq_json_data, shopify_transaction_identifier: shopify_transaction_bq_json_data}' --target "$db" --full-refresh
+dbt test --target "$db"
+fi


We could remove this if we use the +alias in integration_tests/dbt_project.yml (I made a suggestion in that file). Could be easier for maintenance long term.

Suggested change

if [ "$db" = "bigquery" ]; then

dbt run --vars '{shopify_collection_identifier: shopify_collection_data_bq_json, shopify_order_identifier: shopify_order_bq_json_data, shopify_transaction_identifier: shopify_transaction_bq_json_data}' --target "$db" --full-refresh

dbt test --target "$db"

fi

See above comment for why I don't think we should take the alias approach in this case. The alias is great when we need to switch for a given destination. However, in this case we do want to test both files and the alias won't allow us to achieve that.

Ahh I see. Makes sense in that case!

Co-authored-by: fivetran-catfritz <[email protected]>

fivetran-catfritz

Thanks for the response and updates! All lgtm!

fivetran-catfritz · 2025-07-07T15:09:40Z

.buildkite/scripts/run_models.sh

+if [ "$db" = "bigquery" ]; then
+dbt run --vars '{shopify_collection_identifier: shopify_collection_data_bq_json, shopify_order_identifier: shopify_order_bq_json_data, shopify_transaction_identifier: shopify_transaction_bq_json_data}' --target "$db" --full-refresh
+dbt test --target "$db"
+fi


Ahh I see. Makes sense in that case!

.buildkite/scripts/run_models.sh

fivetran-avinash · 2025-07-07T19:47:57Z

CHANGELOG.md

+  - `stg_shopify__order`: `total_shipping_price_set` column
+  - `stg_shopify__transaction`: `receipt` column
+  - `stg_shopify__collection`: `rules` column
+- Introduced new the new `json_to_string()` macro to be used in the BigQuery JSON field support.


Suggested change

- Introduced new the new `json_to_string()` macro to be used in the BigQuery JSON field support.

- Introduced the new `json_to_string()` macro to be used in the BigQuery JSON field support.

Adjusted in latest commit

fivetran-avinash · 2025-07-07T19:51:31Z

CHANGELOG.md

+
+[PR #103](https://github.com/fivetran/dbt_shopify_source/pull/103) includes the following updates:
+
+## Under the Hood


since we're adding support for a new data type, isn't that technically a schema/breaking change?

This will not be a schema breaking change since this will not break/change the outcome of any existing workflows. This will only ensure support for the JSON datatype if it's used by the connection. This converts JSON to string, so the intention of this update is that no schema change will actually occur.

fivetran-avinash · 2025-07-07T19:52:52Z

CHANGELOG.md

+  - `stg_shopify__collection`: `rules` column
+- Introduced new the new `json_to_string()` macro to be used in the BigQuery JSON field support.
+- Included json versions to the integration tests to ensure json data type compatibility.
+


Looks like we also updated the pull request template, should we call that out.

Sure we can. I don't think it's all that necessary, but wouldn't hurt to include.

fivetran-avinash

@fivetran-joemarkiewicz Approved!

fivetran-jamie

Approved with a question in run_models.sh (I don't think action is needed there) and a suggestion in the changelog that shouldn't be blocking

fivetran-jamie · 2025-07-07T20:16:53Z

CHANGELOG.md

+  - `stg_shopify__order`: `total_shipping_price_set` column
+  - `stg_shopify__transaction`: `receipt` column
+  - `stg_shopify__collection`: `rules` column
+- Introduced the new `json_to_string()` macro to be used in the BigQuery JSON field support.


The name of the macro sorta already implies this, but I wonder if we should more explicitly call out that we are converting the JSON fields to strings, in case people are expecting to see JSONs

Agreed, updated in the latest commit

.buildkite/scripts/run_models.sh

patch/json-bigquery-compatibility

6c4e1c5

fivetran-joemarkiewicz self-assigned this Jul 2, 2025

fivetran-joemarkiewicz added 3 commits July 2, 2025 10:22

integration test and changelog updates

86510d3

target type integration test updates

6fe56df

integration updates

5f7f33a

fivetran-joemarkiewicz requested a review from fivetran-catfritz July 2, 2025 20:06

fivetran-joemarkiewicz marked this pull request as ready for review July 2, 2025 20:06

fivetran-catfritz reviewed Jul 3, 2025

View reviewed changes

fivetran-joemarkiewicz and others added 2 commits July 3, 2025 16:28

Apply suggestions from code review

c082c1b

Co-authored-by: fivetran-catfritz <[email protected]>

variable updates

d122e4f

fivetran-catfritz approved these changes Jul 7, 2025

View reviewed changes

fivetran-joemarkiewicz commented Jul 7, 2025

View reviewed changes

.buildkite/scripts/run_models.sh Outdated Show resolved Hide resolved

fivetran-joemarkiewicz added 2 commits July 7, 2025 12:20

Update .buildkite/scripts/run_models.sh

2183f56

Update json_to_string.sql

d9fc020

fivetran-avinash reviewed Jul 7, 2025

View reviewed changes

changelog review updates

714b80f

fivetran-avinash approved these changes Jul 7, 2025

View reviewed changes

fivetran-jamie approved these changes Jul 7, 2025

View reviewed changes

fivetran-joemarkiewicz added 4 commits July 7, 2025 15:56

changelog update

3caaa11

additional fields

4011b5b

shop field addition

53f2709

variable rename

ae09b34

fivetran-joemarkiewicz merged commit 9352c6c into main Jul 8, 2025
9 checks passed

fivetran-joemarkiewicz deleted the patch/json-bigquery-compatibility branch July 8, 2025 15:50

	- Introduced new the new `json_to_string()` macro to be used in the BigQuery JSON field support.
	- Introduced the new `json_to_string()` macro to be used in the BigQuery JSON field support.


		[PR #103](https://github.com/fivetran/dbt_shopify_source/pull/103) includes the following updates:

		## Under the Hood

patch/json-bigquery-compatibility #103

patch/json-bigquery-compatibility #103

Uh oh!

Conversation

fivetran-joemarkiewicz commented Jul 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR Overview

Submission Checklist

Changelog

Uh oh!

fivetran-catfritz left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fivetran-catfritz left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fivetran-joemarkiewicz Jul 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fivetran-avinash left a comment

Choose a reason for hiding this comment

Uh oh!

fivetran-jamie left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

fivetran-joemarkiewicz commented Jul 2, 2025 •

edited

Loading

fivetran-joemarkiewicz Jul 7, 2025 •

edited

Loading