|
| 1 | +--- |
| 2 | +position: 6 |
| 3 | +title: Migrating from Snowflake |
| 4 | +description: "How to map existing data and concepts from Snowflake to SurrealDB" |
| 5 | +--- |
| 6 | + |
| 7 | +# Migrating from Snowflake to SurrealDB |
| 8 | + |
| 9 | +This page shows how Snowflake data types map to their SurrealQL equivalents, then explains how to load Snowflake tables into SurrealDB with the [Surreal Sync](https://github.com/surrealdb/surreal-sync/) tool. |
| 10 | + |
| 11 | +## Data types |
| 12 | + |
| 13 | +The following chart shows Snowflake data types along with the equivalent or near-equivalent [SurrealQL data type](/docs/reference/query-language/datamodel) for each. Surreal Sync reads values through the Snowflake SQL REST API, which returns every cell as a string together with the column's logical type, and converts each one using that type. |
| 14 | + |
| 15 | +| Snowflake Data Type | SurrealDB Mapping | Notes | |
| 16 | +| ------------------- | ----------------- | ----- | |
| 17 | +| **BOOLEAN** | `bool` | | |
| 18 | +| **NUMBER / INT / INTEGER / BIGINT** | `int` | Whole numbers (scale 0) that fit a 64-bit integer | |
| 19 | +| **NUMBER(p,s) / DECIMAL / NUMERIC** | `number` | Values with a scale, or larger than a 64-bit integer, keep their precision and scale as a decimal | |
| 20 | +| **FLOAT / REAL / DOUBLE** | `float` (f64) | Double-precision floating point | |
| 21 | +| **VARCHAR / STRING / TEXT / CHAR** | `string` | | |
| 22 | +| **DATE** | `datetime` | Stored as days since the Unix epoch, converted to a datetime at midnight UTC | |
| 23 | +| **TIME** | `datetime` | Seconds since midnight | |
| 24 | +| **TIMESTAMP_NTZ / DATETIME** | `datetime` | No timezone; taken as the given wall-clock instant | |
| 25 | +| **TIMESTAMP_LTZ / TIMESTAMP_TZ / TIMESTAMP** | `datetime` | Timezone-aware; the absolute instant is preserved as UTC | |
| 26 | +| **VARIANT / OBJECT** | `object` | Parsed from the stored JSON document | |
| 27 | +| **ARRAY** | `array` | Parsed from the stored JSON array | |
| 28 | +| **BINARY / VARBINARY** | `bytes` | Hex-decoded to raw bytes | |
| 29 | + |
| 30 | +Any column whose logical type is not listed above is preserved as a `string` rather than failing the import. |
| 31 | + |
| 32 | +## Importing from Snowflake using Surreal Sync |
| 33 | + |
| 34 | +Surreal Sync can read tables from a Snowflake database and write them to SurrealDB. |
| 35 | + |
| 36 | +> [!NOTE] |
| 37 | +> The Snowflake source is ingestion-only. It performs a single full snapshot of the selected tables and does not support incremental synchronisation or change data capture. There is no durable cursor, so an interrupted run cannot be resumed part-way; re-run the command to start again. |
| 38 | +
|
| 39 | +Rows are read one Snowflake result partition at a time and written in batches, so a large table does not need to fit in memory. |
| 40 | + |
| 41 | +### Prerequisites |
| 42 | + |
| 43 | +Surreal Sync connects to Snowflake through the [SQL REST API](https://docs.snowflake.com/en/developer-guide/sql-api/index) using key-pair (JWT) authentication. Before running an import: |
| 44 | + |
| 45 | +1. Generate an unencrypted PKCS#8 RSA key pair. Encrypted private keys are not supported yet. |
| 46 | +2. Register the public key on the Snowflake user that Surreal Sync will authenticate as: |
| 47 | + |
| 48 | + ```sql |
| 49 | + ALTER USER my_user SET RSA_PUBLIC_KEY='MIIBIjANBgkq...'; |
| 50 | + ``` |
| 51 | + |
| 52 | +3. Make sure that user has a role able to read the target database and schema, and a virtual warehouse to run the queries. |
| 53 | + |
| 54 | +Keep the private key file accessible to Surreal Sync and pass its path with `--private-key-path`. |
| 55 | + |
| 56 | +### Usage |
| 57 | + |
| 58 | +```bash |
| 59 | +surreal-sync from snowflake \ |
| 60 | + --account myorg-myaccount \ |
| 61 | + --user my_user \ |
| 62 | + --private-key-path ./rsa_key.p8 \ |
| 63 | + --warehouse COMPUTE_WH \ |
| 64 | + --database MY_DB \ |
| 65 | + --schema PUBLIC \ |
| 66 | + --to-namespace my_namespace \ |
| 67 | + --to-database my_database \ |
| 68 | + --surreal-endpoint ws://localhost:8000 \ |
| 69 | + --surreal-username root \ |
| 70 | + --surreal-password root |
| 71 | +``` |
| 72 | + |
| 73 | +When `--tables` is omitted, every base table in the schema is imported (views and temporary tables are skipped). Each table becomes a table of the same name in SurrealDB, and each row becomes a record. Snowflake upper-cases unquoted identifiers, so table and column names arrive upper-cased. |
| 74 | + |
| 75 | +Most connection options also read from an environment variable, which is useful for keeping credentials out of shell history. |
| 76 | + |
| 77 | +### Options |
| 78 | + |
| 79 | +| Option | Environment variable | Required | Description | |
| 80 | +| ------ | -------------------- | -------- | ----------- | |
| 81 | +| `--account` | `SNOWFLAKE_ACCOUNT` | Yes | Account identifier as used in the host `<account>.snowflakecomputing.com`, for example `myorg-myaccount` or `xy12345.us-east-1`. | |
| 82 | +| `--user` | `SNOWFLAKE_USER` | Yes | User whose key pair is registered for JWT authentication. | |
| 83 | +| `--private-key-path` | `SNOWFLAKE_PRIVATE_KEY_PATH` | Yes | Path to the unencrypted PKCS#8 private key PEM file. | |
| 84 | +| `--private-key-passphrase` | `SNOWFLAKE_PRIVATE_KEY_PASSPHRASE` | No | Passphrase for an encrypted key. Not supported yet; setting it returns an error. | |
| 85 | +| `--warehouse` | `SNOWFLAKE_WAREHOUSE` | Yes | Virtual warehouse used to run the queries. | |
| 86 | +| `--database` | `SNOWFLAKE_DATABASE` | Yes | Database to read from. | |
| 87 | +| `--schema` | `SNOWFLAKE_SCHEMA` | No | Schema within the database. Defaults to `PUBLIC`. | |
| 88 | +| `--role` | `SNOWFLAKE_ROLE` | No | Role to assume for the session. | |
| 89 | +| `--tables` | — | No | Comma-separated list of tables to import. When omitted, all base tables in the schema are imported. | |
| 90 | +| `--id-columns` | — | No | Comma-separated columns forming the SurrealDB record ID. See [Record IDs](#record-ids). | |
| 91 | +| `--transforms-config` | — | No | Path to a TOML file describing a transform pipeline. Omit to import rows unchanged. | |
| 92 | +| `--to-namespace` | — | Yes | Target SurrealDB namespace. | |
| 93 | +| `--to-database` | — | Yes | Target SurrealDB database. | |
| 94 | + |
| 95 | +The `--surreal-endpoint`, `--surreal-username`, `--surreal-password`, `--batch-size`, and `--dry-run` options are shared with the other Surreal Sync sources. Use `--dry-run` to read and convert rows without writing to SurrealDB. |
| 96 | + |
| 97 | +### Record IDs |
| 98 | + |
| 99 | +Snowflake primary keys are optional and often absent, so the record ID for each row depends on `--id-columns`: |
| 100 | + |
| 101 | +- **Omitted:** a sequential per-table integer is generated. It is deterministic within a run but not stable across re-runs, so prefer explicit ID columns when you need stable IDs. |
| 102 | +- **A single column:** the value becomes the record ID, as an integer when it parses as one and otherwise as a string. |
| 103 | +- **Several columns:** the values are joined with `:` into a string ID. |
| 104 | + |
| 105 | +When ID columns are given, they are used only for the record ID and are not repeated as fields on the record. Column names are matched case-insensitively. |
0 commit comments