Add load_id to arrow tables in extract step instead of normalize

[steinitzu]

Description

• Add _dlt_load_id to arrow tables in extract step before writing file to disk.
• Remove the load_id adding logic from normalize

Should be backwards compatible, aside from the edge case where you upgrade dlt in between extract and normalize

Will add/check tests

Related Issues

Step 1 of #1317

Additional Context

[netlify]

✅ Deploy Preview for dlt-hub-docs canceled.

Name	Link
🔨 Latest commit	0d4347a
🔍 Latest deploy log	https://app.netlify.com/sites/dlt-hub-docs/deploys/6671fc488b59f000080c58ca

[steinitzu]

This was my solution to support the old normalize config.
I think it makes sense to keep it and have it consistent with object normalizer config, rather than move to extract section.

[rudolfix]

hmmm idea is very good.. but maybe you could decorate a method in this class. not init? so you call it just to retrieve config,

[rudolfix]

LGTM!

I think we miss some tests. There's for sure test that checks if load_id is added. However now we need test two things:

1. adding load_id in extract + sensitivity to your config "hack" - just pipeline.extract() arrow table and see if extracted parquet has load_id
2. load some json data and request parquet in nromalize stage - this will create parquet file in normalizer and we test if load id is added there

[rudolfix]

@steinitzu thx for the tests!
still a few issues with normalizing column names. pls check

[steinitzu]

Updated with normalized identifiers :)

[rudolfix]

@steinitzu looks good! but new test is not passing. IMO some column mismatch?

[steinitzu]

@steinitzu looks good! but new test is not passing. IMO some column mismatch?

@rudolfix this is fixed by b1be9c9
The issue was that schema.update_table was adding the column at the front. I'm not sure if it's right to use that method here since it creates the table in schema and that should be left for after all columns are computed?
Added check for _dlt_load_id being last in common tests now too.

[rudolfix]

@steinitzu looks good! but new test is not passing. IMO some column mismatch?

@rudolfix this is fixed by b1be9c9 The issue was that schema.update_table was adding the column at the front. I'm not sure if it's right to use that method here since it creates the table in schema and that should be left for after all columns are computed? Added check for _dlt_load_id being last in common tests now too.

this is really weird. I added one more tests and the columns are added at the end. current version is btw. better, update_table is done at the end to modify actual schema