1.15.0

Breaking changes

This version will add .gz extensions to files that are compressed. That includes filesystem destinations, internal working directory and staging locations used to feed other destinations. A few practical hints:

• Existing filesystem destination will continue storing files without gz extension and they are not affected by the change (existing datasets will retain their behavior where this extension is not added for backwards compatibility)
• Compressed files uploaded to staging destinations will now have the .gz extension, also if dlt is configured to keep data in stage
• This does not apply to parquet files.
• More information can be found in the filesystem destination docs: https://dlthub.com/docs/dlt-ecosystem/destinations/filesystem#file-compression

Core Library

• [Databricks destination] Adding comment and tags for table and columns and applying primary and foreign key constraints in Unity Catalog by @bayees in #2674
• feat - add crlf support for csv exports by @7amza79 in #2783
• feat: add has_more boolean flag logic to RESTClient OffsetPaginator by @michaelconan in #2817
• rest_api: fix: make ProcessingSteps filter and map fields optional by @burnash in #2913
• Enable and test python 3.14 support by @sh-rp in #2789
• removes init files from dlt tables in filesystem by @rudolfix in #2868
• restclient: json param range paginator by @Giackgamba in #2917
• fix sync destination warning logging call by @sh-rp in #2927
• fix: missing __repr__ for @dlt.transformation by @zilto in #2940
• fix: restclient: handle null data in response by @burnash in #2936
• Fix: saving compressed load files with .gz extension by @anuunchin in #2835
• fix: prevent DuplicateSchema error when using public schema in Redshift by @franloza in #2953
• feat: Schema.to_dbml(), auto export schemas in dbml format by @zilto in #2929
• QoL: improve DataValidationError output: use identifying columns if present by @djudjuu in #2915
• callback collector by @djudjuu in #2922
• skips inferring incomplete column when already incomplete by @rudolfix in #2935
• 2946 sqlalchemy destination fixes (full support for mssq, partial for trino) by @rudolfix in #2951
• adds precision to _dlt_load_id and _dlt_id columns by @rudolfix in #2951
• adds json field support for mssql by @rudolfix in #2951
• fixes clickhouse temporary table engine not propagate to nodes (failed merges fix) by @rudolfix in #2951
• fixes BIGQUERY numeric creation (when scale was set to 0) by @rudolfix in #2951
• fix: replace arrow2 with arrow backend for connectorx, enables newest connectorx versions by @zilto in #2933
• AI Command: extended with IDEs (rules for all major IDEs are supported) by @anuunchin in #2937
• duckdb bumped to 1.3.2, iceberg scanners updated by @rudolfix in #2958
• Feat: Allow control over streamed_exec in delta merge upsert by @anuunchin in #2961
• fix failing top level module imports on projects in dirs that start with a dot by @sh-rp in #2963

Docs

• Fix docusaurus / netlify trailing slash issue by @sh-rp in #2878
• make docs snippets tests use local secrets by @sh-rp in #2903
• 2784: update dlt+ pojects.md sources example by @kaliole in #2861
• Docs: Some additions to the filesystem gdrive source by @anuunchin in #2912
• Release highlights 1.12.3-1.14.1 by @AstrakhantsevaAA in #2939
• Updating custom configurations with @configspec decorator by @dat-a-man in #2826
• docs: rest_api: add tip for escaping curly braces by @burnash in #2925
• link to add_map and add_yield_map usage example by @molkazhani2001 in #2916
• adjust cursor docs to new flow by @adrianbr in #2885
• Optimize add_limit docs title for the search by @VioletM in #2949
• Docs/dlt plus project docs rest api restructuring by @kaliole in #2911
• Updated merge loading docs on scd2 strategy handling nested structures by @dat-a-man in #2944

New Contributors

• @bayees made their first contribution in #2674
• @7amza79 made their first contribution in #2783
• @michaelconan made their first contribution in #2817
• @Giackgamba made their first contribution in #2917

Full Changelog: 1.14.1...1.15.0