-
Notifications
You must be signed in to change notification settings - Fork 418
Closed
Labels
spikeInvestigate and solve described problem. Propose implementationInvestigate and solve described problem. Propose implementation
Description
Master merge
We have a few bigger pull requests coming in that change internal object models and storage layouts. We want to release them as 0.4.0 version (which is a major release for us). Public APIs will not be impacted or changes will be backward compatible.
develwill be merged tomasterand 0.3.x released from there- docs will be updated from master branch
develwill collect the 0.4.0 features. docs will be released to devel site- we plan to merge everything to master end of November
Post merge branches and releases
After the 0.4.x merge we will use the following branches
develbranch will be the default branch to which majority of PRs will be acceptedmasterbranch will be used for hotfixes to code and documentation, also for publishing blogposts- on the release day, version of the library will be bumped on
develbranch and thendevelbranch will be merged tomaster - all library releases happen from
master. that includes regular releases and hotfix (post) releases - we may occasionally do pre-releases from the
develbranch
Note: we support two version of documentation:
- current
masterrelease on https://dlthub.com/docs/intro - incoming master / alpha release from
develon https://dlthub.com/devel/intro
0.4.1 release notes
- the destination adapters moved to separate namespace. Weaviate and Qdrant are affected:
from dlt.destinations.weaviate import weaviate_adapter->from dlt.destinations.adapters import weaviate_adapter - data and schema contracts (Handle bad records in "data contract" mode #135 and schema contract #594 ) - refactored extract and normalize processing, backward compatible API changes and no storage changes
- trace improvements - trace dumps with new layout and engine version
- extract → normalize data handover difference (normalize storage layout change and bump) (generate load_id in extract step #756) if you have extracted packages that are not yet normalized left from 0.3.x, the 0.4.x will not migrate the data but rather stop.
- Extract, Load and Normalize Info → way more data collected, interface incompatibilities, loadable dictionary representation will change so the schema if such data is loaded with dlt (refactor
NormalizeInfoandLoadInfo#757 Implement ExtractInfo #754) - source and schema names must be identical → warning / deprecation converted to error. (enforce schema and source name to be the same #758 )
- schema engine version bump (happened already many times - should have no impact on working dlt instances) (enforce schema and source name to be the same #758)
- object model for destinations and thus the way they are implemented changes to more look as sources (Parametrized destinations #746 ) to enable implementation of sinks (implement destination sinks #752 )
- telemetry changes:
destination_type: contains a full module of a destination (dlt.destinations.dummy)
destination_name: contains destination name OR fallback to the last part of destination type name (ie. dummy) - all context injected via Container is now thread-affine so pipelines may run in parallel. full backward compatibility is preserved allows to run parallel pipelines in separate threads #813
Removed deprecations
to_service_account_credentialsremoved inGcpCredentials
Metadata
Metadata
Assignees
Labels
spikeInvestigate and solve described problem. Propose implementationInvestigate and solve described problem. Propose implementation
Type
Projects
Status
Done