Skip to content

Commit 1e9eaba

Browse files
itsezcitsezc
authored
Add Snowflake docs (#138)
1 parent de0bd46 commit 1e9eaba

2 files changed

Lines changed: 174 additions & 0 deletions

File tree

‎README.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -48,6 +48,7 @@ See source-specific guides for more details:
4848
- **[Neo4j](docs/neo4j.md)**: Full and incremental sync using timestamp-based tracking
4949
- **[JSONL](docs/jsonl.md)**: Bulk import from JSON Lines files
5050
- **[Kafka](docs/kafka.md)**: Kafka consumer that subscribes to a topic, importing Kafka message payloads into SurrealDB with optional deduplication
51+
- **[Snowflake](docs/snowflake.md)**: Full one-shot snapshot ingestion via the SQL REST API v2 using key-pair (JWT) auth
5152
- **[How sync works](docs/sync-pipeline.md)**: End-to-end pipeline (source → apply/transforms → sink → watermark), including optional `--transforms-config`
5253

5354
## Development

‎docs/snowflake.md‎

Lines changed: 173 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,173 @@
1+
# Snowflake Source Usage Guide
2+
3+
The Snowflake source in surreal-sync imports tables from a Snowflake database into SurrealDB. This is a **full-snapshot source**: it runs a one-shot read of the selected tables via the Snowflake SQL REST API v2 and writes the rows to SurrealDB. There is no incremental/CDC mode for Snowflake.
4+
5+
Optional transforms: pass `--transforms-config` with a TOML file. Omit the flag to leave rows unchanged. Details: [How sync works](sync-pipeline.md).
6+
7+
## How It Works
8+
9+
The Snowflake source connects to your account over the SQL REST API v2, authenticating with key-pair (JWT) auth. It then reads the tables you name (or every table in the schema) and upserts each row into a SurrealDB table of the same name.
10+
11+
**Record IDs:** By default each table gets a sequential per-table index as its record ID. Pass `--id-columns` to build the SurrealDB record ID from one or more source columns instead. Two or more columns produce an Array record ID. See [How sync works — Record IDs](sync-pipeline.md#record-ids-and-composite-primary-keys).
12+
13+
## Prerequisites
14+
15+
Before using the Snowflake source, ensure you have:
16+
17+
1. **SurrealDB** running locally or accessible via network
18+
2. **surreal-sync** available in your PATH
19+
3. **A Snowflake account** reachable at `<account>.snowflakecomputing.com`
20+
4. **Key-pair auth configured** — an RSA key-pair registered for a Snowflake user, with the private key available as an unencrypted PKCS#8 PEM file
21+
22+
### Setting up key-pair authentication
23+
24+
Generate an unencrypted PKCS#8 private key and its public key:
25+
26+
```bash
27+
openssl genrsa 2048 | openssl pkcs8 -topk8 -inform PEM -out rsa_key.p8 -nocrypt
28+
openssl rsa -in rsa_key.p8 -pubout -out rsa_key.pub
29+
```
30+
31+
Register the public key on the Snowflake user (strip the PEM header/footer and newlines):
32+
33+
```sql
34+
ALTER USER sync_user SET RSA_PUBLIC_KEY='MIIBIjANBgkq...';
35+
```
36+
37+
> Encrypted private keys are not currently supported — `--private-key-passphrase` is accepted but the key must be unencrypted PKCS#8.
38+
39+
## Command Structure
40+
41+
```bash
42+
surreal-sync from snowflake \
43+
# Source (Snowflake) Settings
44+
--account <ACCOUNT> \
45+
--user <USER> \
46+
--private-key-path <PATH> \
47+
--warehouse <WAREHOUSE> \
48+
--database <DATABASE> \
49+
--schema <SCHEMA> \
50+
# Target (SurrealDB) Settings
51+
--to-namespace <TO_NAMESPACE> \
52+
--to-database <TO_DATABASE> \
53+
# Optional Behavior Settings
54+
[OPTIONS]
55+
```
56+
57+
## Required Flags
58+
59+
| Flag | Env var | Description |
60+
|------|---------|-------------|
61+
| `--account <ACCOUNT>` | `SNOWFLAKE_ACCOUNT` | Account identifier as used in `<account>.snowflakecomputing.com` (e.g. `myorg-myaccount`) |
62+
| `--user <USER>` | `SNOWFLAKE_USER` | Snowflake user whose key-pair is registered for JWT auth |
63+
| `--private-key-path <PATH>` | `SNOWFLAKE_PRIVATE_KEY_PATH` | Path to the unencrypted PKCS#8 private key PEM file |
64+
| `--warehouse <WAREHOUSE>` | `SNOWFLAKE_WAREHOUSE` | Virtual warehouse used to run the queries |
65+
| `--database <DATABASE>` | `SNOWFLAKE_DATABASE` | Database to read from |
66+
| `--to-namespace <NAMESPACE>` | — | Target SurrealDB namespace |
67+
| `--to-database <DATABASE>` | — | Target SurrealDB database |
68+
69+
## Optional Flags
70+
71+
### Snowflake Settings
72+
73+
| Flag | Env var | Default | Description |
74+
|------|---------|---------|-------------|
75+
| `--schema <SCHEMA>` | `SNOWFLAKE_SCHEMA` | `PUBLIC` | Schema within the database |
76+
| `--role <ROLE>` | `SNOWFLAKE_ROLE` | (session default) | Role to assume for the session |
77+
| `--private-key-passphrase <PASS>` | `SNOWFLAKE_PRIVATE_KEY_PASSPHRASE` | (none) | Passphrase for an encrypted private key (currently unsupported) |
78+
| `--tables <A,B,...>` | — | (all tables in the schema) | Comma-separated list of tables to ingest |
79+
| `--id-columns <A,B,...>` | — | (sequential per-table index) | Columns forming the SurrealDB record ID; two or more → Array ID |
80+
| `--transforms-config <PATH>` | — | (identity) | TOML file describing the transform pipeline (`[[transforms]]`) |
81+
82+
### SurrealDB Connection Settings
83+
84+
| Flag | Env var | Default | Description |
85+
|------|---------|---------|-------------|
86+
| `--surreal-endpoint <URL>` | `SURREAL_ENDPOINT` | `http://localhost:8000` | SurrealDB endpoint URL |
87+
| `--surreal-username <USER>` | `SURREAL_USERNAME` | `root` | SurrealDB username |
88+
| `--surreal-password <PASS>` | `SURREAL_PASSWORD` | `root` | SurrealDB password |
89+
| `--batch-size <COUNT>` | — | `1000` | Batch size for writing to SurrealDB |
90+
| `--dry-run` | — | `false` | Don't actually write data (testing mode) |
91+
92+
Run `surreal-sync from snowflake --help` for full flag details.
93+
94+
## Usage Examples
95+
96+
### Example 1: Basic snapshot of an entire schema
97+
98+
```bash
99+
surreal-sync from snowflake \
100+
--account myorg-myaccount \
101+
--user sync_user \
102+
--private-key-path ./rsa_key.p8 \
103+
--warehouse COMPUTE_WH \
104+
--database APP \
105+
--schema PUBLIC \
106+
--to-namespace production \
107+
--to-database app
108+
```
109+
110+
### Example 2: Selected tables with a composite record ID
111+
112+
```bash
113+
surreal-sync from snowflake \
114+
--account myorg-myaccount \
115+
--user sync_user \
116+
--private-key-path ./rsa_key.p8 \
117+
--warehouse COMPUTE_WH \
118+
--database APP \
119+
--schema SALES \
120+
--tables orders,order_items \
121+
--id-columns order_id,line_no \
122+
--to-namespace production \
123+
--to-database sales
124+
```
125+
126+
### Example 3: Using environment variables
127+
128+
```bash
129+
export SNOWFLAKE_ACCOUNT="myorg-myaccount"
130+
export SNOWFLAKE_USER="sync_user"
131+
export SNOWFLAKE_PRIVATE_KEY_PATH="./rsa_key.p8"
132+
export SNOWFLAKE_WAREHOUSE="COMPUTE_WH"
133+
export SNOWFLAKE_DATABASE="APP"
134+
export SURREAL_ENDPOINT="ws://localhost:8000"
135+
export SURREAL_USERNAME="admin"
136+
export SURREAL_PASSWORD="secure-password"
137+
138+
surreal-sync from snowflake \
139+
--to-namespace production \
140+
--to-database app
141+
```
142+
143+
## Embedding in Your Own Rust Binary
144+
145+
The Snowflake source is also available as a library entrypoint, so you can run the
146+
same import from your own binary and append in-process transforms written in Rust —
147+
for example to redact PII, rename fields, or promote foreign keys into SurrealDB
148+
record links. Your binary accepts the same flags as `surreal-sync from snowflake`
149+
(without the `from snowflake` prefix):
150+
151+
```rust
152+
use surreal_sync::snowflake;
153+
use surreal_sync::{FlattenId, InPlaceTransform};
154+
155+
#[tokio::main]
156+
async fn main() -> anyhow::Result<()> {
157+
snowflake::run([
158+
Box::new(FlattenId::default()) as Box<dyn InPlaceTransform>,
159+
// your own Box<dyn InPlaceTransform> stages …
160+
])
161+
.await
162+
}
163+
```
164+
165+
See [`examples/snowflake_custom_transform.rs`](../examples/snowflake_custom_transform.rs) for a complete, runnable example.
166+
167+
## Current Limitations
168+
169+
- **Full snapshot only.** There is no incremental or CDC mode for Snowflake; each run reads the selected tables in full.
170+
- **Key-pair (JWT) auth only.** Username/password and OAuth are not supported.
171+
- **Unencrypted private keys only.** `--private-key-passphrase` is accepted but encrypted keys are not currently supported.
172+
173+
If your use case requires additional capabilities, please file a feature request at: https://github.com/surrealdb/surreal-sync/issues

0 commit comments

Comments
 (0)