CHANGES.md

Changelog

Unreleased

• Kinesis: Added ctk kinesis CLI group with list-checkpoints and prune-checkpoints commands for checkpoint table maintenance
• Dependencies: Permitted installation of click 8.3

2026/03/16 v0.0.46

• I/O: API improvements: ctk {load,save} table became ctk {load,save} Also, the --cluster-url option becomes optional. That's a much more concise interface now.

2026/03/13 v0.0.45

• I/O: Refactored ctk.io subsystem
• I/O: Refurbished documentation of the I/O subsystem
• I/O: Started using ingestr for loading data from Amazon Kinesis
• I/O: Started propagating start_date parameter to ingestr's interval_start
• I/O: Started propagating batch_size parameter to ingestr's page_size
• Packaging: Renamed extras all → full, io-ingestr → io-ingest, and io-all → io-curated

2026/03/04 v0.0.44

• I/O: Added adapter for Apache Iceberg tables
• I/O: Added adapter for Delta Lake tables

2026/02/18 v0.0.43

• Retention: Validated integration with Azure blob storage using Azurite
• Fixed missing pandas package that prevented installation/execution without extras

2026/02/11 v0.0.42

• Settings: Stop flagging gateway.recover_after_time as a difference when both gateway.expected_nodes and gateway.expected_data_nodes are unset (-1).
• PyMongo adapter: Adapted to CrateDB 6.2+ which disallows writes to _id columns
• Dependencies: Permitted installation of pyarrow 23
• Dependencies: Permitted installation of duckdb 1.x
• Dependencies: Permitted installation of pandas 3.0
• CI: Validated against Python 3.14
• CI: Validated against InfluxDB 2.8
• CI: Validated against MongoDB 8

2025/08/19 v0.0.41

• I/O: Updated to influxio-0.6.0. Thanks, @ZillKhan.
  • The target table is now the measurement name when importing ILP files.
  • The InfluxDB source URL accepts a timeout query parameter (seconds) to configure the network timeout when talking to the InfluxDB API.
  • For ILP imports, the CrateDB URL no longer needs a table component; you can point it at the schema only (the measurement determines the table).

2025/08/19 v0.0.40

• I/O: Fixed MongoDB CDC invocation. Thanks, Mỹ Duyên.

2025/08/19 v0.0.39

• OCI: Started producing image ghcr.io/crate/cratedb-toolkit-ingest
• I/O: Added drivers for ODBC and Oracle to cratedb-toolkit-ingest
• I/O: Updated BSON library to support ARM64

2025/08/14 v0.0.38

• I/O: Updated to ingestr>=0.13.61
• CFR: Improved log output
• CFR: Fixed double quoting of table name. Thanks, @karynzv.
• CFR: When importing, started using replace strategy instead of append
• CFR: Improved importing data re. type mapping without NumPy
• CFR: Truncated target table before importing, using append strategy again, because replace doesn't do the right DDL.
• I/O: Tuned down ingestr, it masked native I/O adapters

2025/07/01 v0.0.37

• Settings: Fixed comparison of 0s vs 0ms. Thanks, @hlcianfagna.
• DMS: Provided a recipe file to relay primary key and column type map information
• DMS: Provided a recipe option to ignore processing DMS control DDL events
• DMS: Started using the "direct" column mapping by default, retaining the "universal" column mapping optionally.
• Dependencies: Updated to commons-codec>=0.0.23
• I/O: Adapter for PostgreSQL full-load using ingestr
• I/O: Added documentation about ingestr adapter

2025/06/23 v0.0.36

• Dependencies: Migrated from zyp to tikray. It's effectively the same, but provided using a dedicated package now
• Dependencies: Updated to croud-1.14
• Dependencies: Updated to async-kinesis-2.0.0. Thanks, @hampsterx.
• CDC: Added canonical SQL example for PostgreSQL from Ibis
• CDC: Enabled loading DMS events from Kinesis streams and stream-dump files
• CDC: Added subcommand ctk dms table-mappings

2025/05/13 v0.0.35

• Added lost pytest dependencies to cratedb-toolkit[testing]

2025/05/13 v0.0.34

• Downgraded to sqlalchemy-cratedb 0.41, version 0.42 is not GA yet

2025/05/12 v0.0.33

• CFR: Enhanced job statistics with optional reporting database support. Thanks, @WalBeh.
• Settings: Added settings comparison utility. Thanks, @WalBeh.
• Meta: Added parser for https://cratedb.com/releases.json file. Thanks, @WalBeh.
• CFR: Added the ability to anonymize queries recorded by collect
• Cloud API: SDK and CLI for CrateDB Cloud Cluster and Import APIs. Supports headless/unattended operations on CrateDB Cloud clusters, covering deploy/start/resume and data import procedures using fluent API and CLI.
• Cloud API: Added JWT authentication to client API and ctk shell.
• Cloud API: Added health and ping subcommands to ctk cluster
• CLI: Downgraded to Click 8.1, as the code is not compatible with 8.2 yet

Breaking changes

Naming things for CLI options and environment variables:

• Converged --cratedb-sqlalchemy-url vs. --cratedb-http-url options into single --cluster-url
• Converged CRATEDB_SQLALCHEMY_URL vs. CRATEDB_HTTP_URL env vars into single CRATEDB_CLUSTER_URL

2025/04/23 v0.0.32

• MCP: Add subsystem providing a few server and client utilities through the ctk query mcp {list,inquire,launch} subcommands.
• Docs API: Added extractors for CrateDB functions and settings
• Connect: Respect sslmode URI parameter when converting SQLAlchemy connection URLs to http(s)://

2025/01/31 v0.0.31

• Fixed connectivity for jobstats collect
• Refactored code and improved CLI interface of ctk info vs. ctk cfr
• Dependencies: Updated to crate-2.0.0, which uses orjson for JSON marshalling
• CFR: Job statistics and slow-query exploration per Marimo notebook. Thanks, @WalBeh.

2025/01/13 v0.0.30

• Dependencies: Minimize dependencies of core installation, defer polars to cratedb-toolkit[io].
• Fixed ctk cfr info record about too large values of ulimit_hard
• Improved ctk shell to also talk to CrateDB standalone databases
• Added basic utility command ctk tail, for tailing a database table, and optionally following the tail
• Table Loader: Added capability to load InfluxDB Line Protocol (ILP) files
• Query Collector: Now respects CRATEDB_CLUSTER_URL environment variable

2024/10/13 v0.0.29

• MongoDB: Added Zyp transformations to the CDC subsystem, making it more symmetric to the full-load procedure.
• Query Converter: Added very basic expression converter utility with CLI interface
• DynamoDB: Added query expression converter for relocating object references, to support query migrations after the breaking change with the SQL DDL schema, by v0.0.27.

2024/10/09 v0.0.28

• IO: Improved BulkProcessor when running per-record operations by also checking rowcount for handling INSERT OK, 0 rows responses
• MongoDB: Fixed BSON decoding of {"$date": 1180690093000} timestamps by updating to commons-codec 0.0.21.
• Testcontainers: Don't always pull the OCI image before starting. It is unfortunate in disconnected situations.

2024/10/01 v0.0.27

• MongoDB: Updated to pymongo 4.9
• DynamoDB: Change CrateDB data model to use (pk, data, aux) columns Attention: This is a breaking change.

2024/09/26 v0.0.26

• MongoDB: Configure MongoDBCrateDBConverter after updating to commons-codec 0.0.18
• DynamoDB CDC: Fix MODIFY operation to also propagate deleted attributes

2024/09/22 v0.0.25

• Table Loader: Improved conditional handling of "transformation" parameter
• Table Loader: Improved status reporting and error logging in BulkProcessor
• MongoDB: Improve error reporting
• MongoDB Full: Polars' read_ndjson doesn't load MongoDB JSON data well, use fsspec and orjson instead
• MongoDB Full: Improved initialization of transformation subsystem
• MongoDB Adapter: Improved performance of when computing collection cardinality by using collection.estimated_document_count()
• MongoDB Full: Optionally use limit parameter as number of total records
• MongoDB Adapter: Evaluate _id filter field by upcasting to bson.ObjectId, to convey a filter that makes ctk load table process a single document, identified by its OID
• MongoDB Dependencies: Update to commons-codec 0.0.17

2024/09/19 v0.0.24

• MongoDB Full: Refactor transformation subsystem to commons-codec
• MongoDB: Update to commons-codec v0.0.16

2024/09/16 v0.0.23

• MongoDB: Unlock processing multiple collections, either from server database, or from filesystem directory
• MongoDB: Unlock processing JSON files from HTTP resource, using https+bson://
• MongoDB: Optionally filter server collection using MongoDB query expression
• MongoDB: Improve error handling wrt. bulk operations vs. usability
• DynamoDB CDC: Add ctk load table interface for processing CDC events
• DynamoDB CDC: Accept a few more options for the Kinesis Stream: batch-size, create, create-shards, start, seqno, idle-sleep, buffer-time
• DynamoDB Full: Improve error handling wrt. bulk operations vs. usability

2024/09/10 v0.0.22

• MongoDB: Rename columns with leading underscores to use double leading underscores
• MongoDB: Add support for UUID types
• MongoDB: Improve reading timestamps in previous BSON formats
• MongoDB: Fix processing empty arrays/lists. By default, assume TEXT as inner type.
• MongoDB: For ctk load table, use "partial" scan for inferring the collection schema, based on the first 10,000 documents.
• MongoDB: Skip leaking UNKNOWN fields into SQL DDL. This means relevant column definitions will not be included into the SQL DDL.
• MongoDB: Make ctk load table use the data OBJECT(DYNAMIC) mapping strategy.
• MongoDB: Sanitize lists of varying objects
• MongoDB: Add treatment option for applying special treatments to certain items on real-world data
• MongoDB: Use pagination on source collection, for creating batches towards CrateDB
• MongoDB: Unlock importing MongoDB Extended JSON files using file+bson://...

2024/09/02 v0.0.21

• DynamoDB: Add special decoding for varied lists. Store them into a separate OBJECT(IGNORED) column in CrateDB.
• DynamoDB: Add pagination support for full-load table loader

2024/08/27 v0.0.20

• DMS/DynamoDB: Fix table name quoting within CDC processor handler

2024/08/26 v0.0.19

• MongoDB: Fix and verify Zyp transformations
• DMS/DynamoDB/MongoDB I/O: Use SQL with parameters instead of inlining values

2024/08/21 v0.0.18

• Dependencies: Unpin commons-codec, to always use the latest version
• Dependencies: Unpin lorrystream, to always use the latest version
• MongoDB: Improve type mapper by discriminating between INTEGER and BIGINT
• MongoDB: Improve type mapper by supporting BSON DatetimeMS, Decimal128, and Int64 types

2024/08/19 v0.0.17

• Processor: Updated Kinesis Lambda processor to understand AWS DMS
• MongoDB: Fix missing output on STDOUT for migr8 export
• MongoDB: Improve timestamp parsing by using python-dateutil
• MongoDB: Converge _id input field to id column instead of dropping it
• MongoDB: Make user interface use stderr, so stdout is for data only
• MongoDB: Make migr8 extract write to stdout by default
• MongoDB: Make migr8 translate read from stdin by default
• MongoDB: Improve user interface messages
• MongoDB: Strip single leading underscore character from all top-level fields
• MongoDB: Map OID types to CrateDB TEXT columns
• MongoDB: Make migr8 extract and migr8 export accept the --limit option
• MongoDB: Fix indentation in prettified SQL output of migr8 translate
• MongoDB: Add capability to give type hints and add transformations
• Dependencies: Adjust code for lorrystream version 0.0.3
• Dependencies: Update to lorrystream 0.0.4 and commons-codec 0.0.7
• DynamoDB: Add table loader for full-load operations

2024/07/25 v0.0.16

• ctk load table: Added support for MongoDB Change Streams
• Fix dependency with the kaggle package, downgrade to kaggle==1.6.14
• DynamoDB CDC: Add demo to support reading DynamoDB change data capture

2024/07/08 v0.0.15

• IO: Added the if-exists query parameter by updating to influxio 0.4.0.
• Rockset: Added CrateDB Rockset Adapter, a HTTP API emulation layer
• MongoDB: Added adapter amalgamating PyMongo to use CrateDB as backend
• SQLAlchemy: Clean up and refactor SQLAlchemy polyfills to cratedb_toolkit.util.sqlalchemy
• CFR: Build as a self-contained program using PyInstaller
• CFR: Publish self-contained application bundle to GitHub Workflow Artifacts

2024/06/18 v0.0.14

• Add ctk info and ctk cfr diagnostics programs
• Remove support for Python 3.7
• SQLAlchemy dialect: Use sqlalchemy-cratedb>=0.37.0 This includes the fix to the get_table_names() reflection method.

2024/06/11 v0.0.13

• Dependencies: Migrate from crate[sqlalchemy] to sqlalchemy-cratedb

2024/05/30 v0.0.12

• Fix InfluxDB Cloud <-> CrateDB Cloud connectivity by using ssl=true query argument also for influxdb2:// source URLs.

2024/05/30 v0.0.11

• Fix InfluxDB Cloud <-> CrateDB Cloud connectivity by propagating ssl=true query argument. Update dependencies to influxio>=0.2.1,<1.

2024/04/10 v0.0.10

• Dependencies: Unpin upper version bound of dask. Otherwise, compatibility issues can not be resolved quickly, like with Python 3.11.9. dask/dask#11038

2024/03/22 v0.0.9

• Dependencies: Use dask[dataframe]

2024/03/11 v0.0.8

• datasets: Fix compatibility with Python 3.7

2024/03/07 v0.0.7

• datasets: Fix dataset loader

2024/03/07 v0.0.6

• Added cratedb_toolkit.datasets subsystem, for acquiring datasets from cratedb-datasets and Kaggle.

2024/02/12 v0.0.5

• Do not always activate pytest11 entrypoint to pytest fixture cratedb_service, as it depends on the testcontainers package, which is not always installed.

2024/02/10 v0.0.4

• Packaging: Use cloud extra to install relevant packages
• Dependencies: Add testing extra, which installs testcontainers only
• Testing: Export cratedb_service fixture as pytest11 entrypoint
• Sandbox: Reduce number of extras by just using all

2024/01/18 v0.0.3

• Add SQL runner utility primitives to io.sql namespace
• Add import_csv_pandas and import_csv_dask utility primitives
• data: Add subsystem for "loading" data.
• Add SDK and CLI for CrateDB Cloud Data Import APIs ctk load table ...
• Add migr8 program from previous repository
• InfluxDB: Add adapter for influxio
• MongoDB: Add migr8 program from previous repository
• MongoDB: Improve UX by using ctk load table mongodb://...
• load table: Refactor to use more OO
• Add examples/cloud_import.py
• Adapt testcontainers to be agnostic of the testing framework. Thanks, @pilosus.

2023/11/06 v0.0.2

• CLI: Upgrade to click-aliases>=1.0.2, fixing erroring out when no group aliases are specified.

• Add support for Python 3.12

• SQLAlchemy: Improve UNIQUE constraints polyfill to accept multiple column names, for emulating unique composite keys.

2023/10/10 v0.0.1

• SQLAlchemy: Add a few patches and polyfills, which do not fit well into the vanilla Python driver / SQLAlchemy dialect.

• Retention: Refactor strategies delete, reallocate, and snapshot, to standalone variants.

• Retention: Bundle configuration and runtime settings into Settings entity, and use more OO instead of weak dictionaries: Add RetentionStrategy, TableAddress, and Settings entities, to improve information passing throughout the application and the SQL templates.

• Retention: Add --schema option, and CRATEDB_EXT_SCHEMA environment variable, to configure the database schema used to store the retention policy table. The default value is ext.

• Retention: Use full-qualified table names everywhere.

• Retention: Fix: Compensate for DROP REPOSITORY now returning RepositoryMissingException when the repository does not exist. With previous versions of CrateDB, it was RepositoryUnknownException.

2023/06/27 v0.0.0

• Import "data retention" implementation from https://github.com/crate/crate-airflow-tutorial. Thanks, @hammerhead.