Refurbish IO documentation (#698)

* IO: Update change log

* Project: Rename package extras `all -> full`, `io-ingestr -> io-ingest`

* IO documentation: Refurbish introduction and guidance

* Project: Rename package extras `io-all -> io-curated`

* IO documentation: New layout based on groups

Discriminate I/O adapters:

- Files
- Databases
- Streams
- Services
- Open Table Formats

* IO documentation: This and that

* IO documentation: Dissolve "ingestr" page

* Chore: Format `pyproject.toml`

* IO documentation: Add dedicated page about Kinesis

* IO: Use ingestr for loading data from Amazon Kinesis

* IO: Propagate `start_date` parameter to ingestr's `interval_start`

* IO documentation: Add dedicated page about Kafka

* IO: Start propagating `batch_size` parameter to ingestr's `page_size`

* IO documentation: Add subsection about data warehouses

* IO documentation: Copy editing

* IO documentation: Generalize CrateDB connection options

* IO documentation: Add dedicated page about SAP HANA

* IO documentation: Improve page about open table formats

* IO documentation: Add page about Elasticsearch

* IO documentation: Generalize CrateDB connection options

* IO misc: Implement suggestions by CodeRabbit

* IO documentation: Implement suggestions by CodeRabbit

* IO documentation: Add page about MotherDuck

* IO documentation: Copy editing

* IO documentation: Enumerate more data warehouses, databases and services

* IO documentation: Add logo icons across the board

* IO documentation: Fix and improve cross references

* IO documentation: Harmonize document titles

* IO documentation: Relocate page about InfluxDB

* IO documentation: Implement suggestions by CodeRabbit

* IO documentation: Implement suggestions by CodeRabbit

* CI: Fix integration tests with MongoDB for the CrateDB Cloud matrix slot

Author: amotl

.github/workflows/cratedb-cloud.yml

@@ -68,6 +68,13 @@ jobs:
         # Install package in editable mode.
         uv pip install --editable='.[full,test,develop]'
 
+        # MongoDB integration tests fail here. Why?
+        # pymongo.errors.ServerSelectionTimeoutError: localhost:32772:
+        # [Errno 111] Connection refused (configured timeouts: socketTimeoutMS: 5000.0ms,
+        #                                 connectTimeoutMS: 5000.0ms), Timeout: 5.0s
+        # https://github.com/crate/cratedb-toolkit/actions/runs/23055232713/job/66966725093#step:6:820
+        uv pip uninstall pymongo
+
     - name: Run linter and software tests
       env:
         KAGGLE_USERNAME: ${{ secrets.KAGGLE_USERNAME }}

.github/workflows/postgresql.yml

@@ -97,7 +97,7 @@ jobs:
       run: |
 
         # Install package in editable mode.
-        uv pip install --prerelease=allow --editable='.[io-ingestr,test]'
+        uv pip install --prerelease=allow --editable='.[io-ingest,test]'
 
     - name: Run software tests
       run: |

CHANGES.md

@@ -1,6 +1,14 @@
 # Changelog
 
 ## Unreleased
+- I/O: Refactored `ctk.io` subsystem
+- I/O: Refurbished documentation of the [I/O subsystem]
+- I/O: Started using ingestr for loading data from Amazon Kinesis
+- I/O: Started propagating `start_date` parameter to ingestr's `interval_start`
+- I/O: Started propagating `batch_size` parameter to ingestr's `page_size`
+- Packaging: Renamed extras `all` → `full`, `io-ingestr` → `io-ingest`, and `io-all` → `io-curated`
+
+[I/O subsystem]: https://cratedb-toolkit.readthedocs.io/io/
 
 ## 2026/03/04 v0.0.44
 - I/O: Added adapter for Apache Iceberg tables

cratedb_toolkit/io/exception.py

@@ -0,0 +1,2 @@
+class SkipAdapterException(Exception):
+    pass

cratedb_toolkit/io/ingestr/api.py

@@ -75,7 +75,19 @@ def ingestr_copy(source_url: str, target_address: DatabaseAddress, progress: boo
     source_url_obj = URL(source_url)
     source_table = source_url_obj.query.get("table")
     source_fragment = source_url_obj.fragment
-    source_url_obj = source_url_obj.without_query_params("table").with_fragment("")
+
+    batch_size_raw = source_url_obj.query.get("batch_size")
+    batch_size = None
+    if batch_size_raw is not None:
+        try:
+            batch_size = int(batch_size_raw)
+        except ValueError as ex:
+            raise ValueError("`batch_size` must be an integer") from ex
+        if batch_size <= 0:
+            raise ValueError("`batch_size` must be greater than 0")
+
+    start_date = source_url_obj.query.get("start_date")
+    source_url_obj = source_url_obj.without_query_params("table", "start_date", "batch_size").with_fragment("")
 
     target_uri, target_table_address = target_address.decode()
     target_table = target_table_address.fullname
@@ -94,15 +106,23 @@ def ingestr_copy(source_url: str, target_address: DatabaseAddress, progress: boo
     logger.info(f"Target URL: {target_url}")
     logger.info(f"Source Table: {source_table}")
     logger.info(f"Target Table: {target_table}")
+    logger.info(f"Start Date: {start_date}")
+    logger.info(f"Batch Size: {batch_size}")
+
+    ingest_kwargs = dict(  # noqa: C408
+        source_uri=str(source_url_obj),
+        dest_uri=str(target_url),
+        source_table=source_table,
+        dest_table=target_table,
+        yes=True,
+    )
+    if start_date is not None:
+        ingest_kwargs["interval_start"] = start_date
+    if batch_size is not None:
+        ingest_kwargs["page_size"] = batch_size
 
     try:
-        ingestr.main.ingest(
-            source_uri=str(source_url_obj),
-            dest_uri=str(target_url),
-            source_table=source_table,
-            dest_table=target_table,
-            yes=True,
-        )
+        ingestr.main.ingest(**ingest_kwargs)
         return True
     except ConfigFieldMissingException:
         logger.error(

cratedb_toolkit/io/kinesis/relay.py

@@ -11,6 +11,7 @@
 from tqdm.contrib.logging import logging_redirect_tqdm
 from yarl import URL
 
+from cratedb_toolkit.io.exception import SkipAdapterException
 from cratedb_toolkit.io.kinesis.adapter import KinesisAdapterBase
 from cratedb_toolkit.io.kinesis.model import RecipeDefinition
 from cratedb_toolkit.model import DatabaseAddress
@@ -52,7 +53,7 @@ def __init__(
                 primary_keys=pks, column_types=cms, mapping_strategy=mapping_strategy, ignore_ddl=ignore_ddl
             )
         else:
-            raise NotImplementedError(f"Data processing not implemented for {self.kinesis_url}")
+            raise SkipAdapterException(f"Not processing {self.kinesis_url} here")
 
         self.connection: sa.Connection
         self.progress_bar: tqdm

cratedb_toolkit/io/router.py

@@ -17,6 +17,7 @@
 from cratedb_toolkit.exception import (
     OperationFailed,
 )
+from cratedb_toolkit.io.exception import SkipAdapterException
 from cratedb_toolkit.model import DatabaseAddress, InputOutputResource
 from cratedb_toolkit.util.data import asbool
 
@@ -65,11 +66,14 @@ def load_table(
         elif source_url_obj.scheme.startswith("kinesis"):
             from cratedb_toolkit.io.kinesis.api import kinesis_relay
 
-            return kinesis_relay(
-                source_url=source_url_obj,
-                target_url=target_url,
-                recipe=transformation,
-            )
+            try:
+                return kinesis_relay(
+                    source_url=source_url_obj,
+                    target_url=target_url,
+                    recipe=transformation,
+                )
+            except SkipAdapterException:
+                pass
 
         elif source_url_obj.scheme in [
             "file+bson",
@@ -112,6 +116,8 @@ def load_table(
 
         from cratedb_toolkit.io.ingestr.api import ingestr_copy, ingestr_select
 
+        source_url = str(source_url_obj)
+
         if ingestr_select(source_url):
             return ingestr_copy(source_url, target, progress=True)
 

doc/_snippet/cloud-prerequisites.md

@@ -0,0 +1,46 @@
+## Prerequisites
+
+Before using CrateDB Cloud services, authenticate and select
+your target database cluster.
+
+### Authenticate
+
+When working with [CrateDB Cloud], you can select between two authentication variants.
+Either _interactively authorize_ your terminal session using `croud login`,
+```shell
+# Replace YOUR_IDP with one of: cognito, azuread, github, google.
+croud login --idp YOUR_IDP
+```
+or provide API access credentials per environment variables for _headless/unattended
+operations_ after creating them using the [CrateDB Cloud Console] or
+`croud api-keys create`.
+```shell
+# Provide CrateDB Cloud API authentication tokens.
+export CRATEDB_CLOUD_API_KEY='<YOUR_API_KEY>'
+export CRATEDB_CLOUD_API_SECRET='<YOUR_API_SECRET>'
+```
+
+### Select cluster
+
+Discover the list of available database clusters.
+```shell
+croud clusters list
+```
+
+Select the designated target database cluster using one of three variants,
+either by using CLI options or environment variables.
+- All address options are mutually exclusive.
+- CLI options take precedence over environment variables.
+- Environment variables can be stored into an `.env` file in your working directory.
+
+:CLI options:
+  `--cluster-id`, `--cluster-name`, `--cluster-url`
+:Environment variables:
+  `CRATEDB_CLUSTER_ID`, `CRATEDB_CLUSTER_NAME`, `CRATEDB_CLUSTER_URL`
+
+Before invoking any of the next steps, address the CrateDB Cloud Cluster
+you are aiming to connect to, for example by defining the cluster id
+using the `CRATEDB_CLUSTER_ID` environment variable.
+```shell
+export CRATEDB_CLUSTER_ID='<YOUR_CLUSTER_ID>'
+```

doc/_snippet/ingest-see-also.md

@@ -0,0 +1,10 @@
+CrateDB also provides native data import capabilities and support for different
+ETL applications and frameworks, see [load data into CrateDB].
+If you have additional requirements on this or other I/O adapters, for example
+to support advanced processing options or different data formats, or if you want
+us to provide a managed variant, please let us know through any of our [support
+channels], preferably on our [community forum].
+
+[load data into CrateDB]: inv:guide:std:label#ingest
+[community forum]: https://community.cratedb.com/t/loading-data-into-cratedb-weekly-edition/2052
+[support channels]: https://cratedb.com/support

doc/_snippet/install-ctk.md

@@ -0,0 +1,28 @@
+## Installation
+
+CrateDB Toolkit uses the Python programming language with native performance
+extensions, which is installed
+on most machines today. Otherwise, we recommend to [download and install
+Python][install-python] from the original source.
+For installing additional Python packages, we recommend to
+[install the uv package manager][install-uv].
+```shell
+uv tool install --upgrade 'cratedb-toolkit'
+```
+
+An alternative way to install Python packages is to use [pipx]
+or `pip install --user`.
+```shell
+pipx install 'cratedb-toolkit'
+```
+
+Another way to invoke CrateDB Toolkit without installing it is to use its
+container image with Docker, Podman, Kubernetes, and friends.
+```shell
+docker run --rm ghcr.io/crate/cratedb-toolkit ctk
+```
+
+
+[install-python]: https://www.python.org/downloads/
+[install-uv]: https://docs.astral.sh/uv/getting-started/installation/
+[pipx]: https://pipx.pypa.io/