Skip to content

Commit 2a97b4f

Browse files
authored
Add catalog resolution (w/ OSS MySQL implementation) (#168)
* Add catalog to databases crd * Update views crd * Add MySQL docker deployment * Catalog implementation w/ MySQL * Add proper metadata handling of catalogs * Fix indexing issue for column types
1 parent b264429 commit 2a97b4f

File tree

81 files changed

+2649
-108
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

81 files changed

+2649
-108
lines changed

Makefile

Lines changed: 12 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -86,6 +86,15 @@ undeploy-kafka:
8686
kubectl delete -f ./deploy/samples/kafkadb.yaml || echo "skipping"
8787
kubectl delete namespace kafka || echo "skipping"
8888

89+
# Deploys MySQL cluster in docker with a few databases and tables.
90+
deploy-mysql: deploy deploy-flink
91+
docker compose -f ./deploy/docker/mysql/docker-compose.yaml up -d --wait
92+
kubectl apply -f ./deploy/samples/mysqldb.yaml
93+
94+
undeploy-mysql:
95+
kubectl delete -f ./deploy/samples/mysqldb.yaml || echo "skipping"
96+
docker compose -f ./deploy/docker/mysql/docker-compose.yaml down
97+
8998
# Deploys Venice cluster in docker and creates two stores in Venice. Stores are not managed via K8s for now.
9099
deploy-venice: deploy deploy-flink
91100
docker compose -f ./deploy/docker/venice/docker-compose-single-dc-setup.yaml up -d --wait
@@ -97,9 +106,9 @@ undeploy-venice:
97106
kubectl delete -f ./deploy/samples/venicedb.yaml || echo "skipping"
98107
docker compose -f ./deploy/docker/venice/docker-compose-single-dc-setup.yaml down
99108

100-
deploy-dev-environment: deploy deploy-demo deploy-flink deploy-kafka deploy-venice
109+
deploy-dev-environment: deploy deploy-demo deploy-flink deploy-kafka deploy-mysql deploy-venice
101110

102-
undeploy-dev-environment: undeploy-venice undeploy-kafka undeploy-flink undeploy-demo undeploy
111+
undeploy-dev-environment: undeploy-venice undeploy-mysql undeploy-kafka undeploy-flink undeploy-demo undeploy
103112
kubectl delete -f ./deploy/dev || echo "skipping"
104113

105114
# Integration test setup intended to be run locally
@@ -137,4 +146,4 @@ run-zeppelin: build-zeppelin
137146
--name hoptimator-zeppelin \
138147
hoptimator-zeppelin
139148

140-
.PHONY: install test build bounce clean quickstart deploy-config undeploy-config deploy undeploy deploy-demo undeploy-demo deploy-flink undeploy-flink deploy-kafka undeploy-kafka deploy-venice undeploy-venice build-zeppelin run-zeppelin integration-tests integration-tests-kind deploy-dev-environment undeploy-dev-environment generate-models release
149+
.PHONY: install test build bounce clean quickstart deploy-config undeploy-config deploy undeploy deploy-demo undeploy-demo deploy-flink undeploy-flink deploy-kafka undeploy-kafka deploy-mysql undeploy-mysql deploy-venice undeploy-venice build-zeppelin run-zeppelin integration-tests integration-tests-kind deploy-dev-environment undeploy-dev-environment generate-models release

deploy/docker/mysql/README.md

Lines changed: 146 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,146 @@
1+
# MySQL Docker Setup for Hoptimator
2+
3+
This directory contains the Docker Compose configuration and initialization scripts for running MySQL with Hoptimator.
4+
5+
## Overview
6+
7+
The setup creates a MySQL 8.0 instance with:
8+
- **Two databases**: `testdb` and `analytics`
9+
- **Pre-populated test data** for integration testing
10+
- **Catalog support** demonstrating Hoptimator's hierarchical data source capabilities
11+
12+
## Quick Start
13+
14+
```bash
15+
# Deploy MySQL with Hoptimator
16+
make deploy-mysql
17+
18+
# Undeploy MySQL
19+
make undeploy-mysql
20+
```
21+
22+
## Database Schema
23+
24+
### testdb Database
25+
26+
**users table:**
27+
- `user_id` (INT, PRIMARY KEY)
28+
- `username` (VARCHAR)
29+
- `email` (VARCHAR)
30+
- `created_at` (TIMESTAMP)
31+
- `is_active` (BOOLEAN)
32+
33+
**orders table:**
34+
- `order_id` (INT, PRIMARY KEY)
35+
- `user_id` (INT, FOREIGN KEY)
36+
- `product_name` (VARCHAR)
37+
- `quantity` (INT)
38+
- `price` (DECIMAL)
39+
- `order_date` (TIMESTAMP)
40+
- `status` (VARCHAR)
41+
42+
**products table:**
43+
- `product_id` (INT, PRIMARY KEY)
44+
- `product_name` (VARCHAR)
45+
- `category` (VARCHAR)
46+
- `price` (DECIMAL)
47+
- `stock_quantity` (INT)
48+
- `description` (TEXT)
49+
50+
### analytics Database
51+
52+
**daily_metrics table:**
53+
- `metric_id` (INT, PRIMARY KEY)
54+
- `metric_date` (DATE)
55+
- `metric_name` (VARCHAR)
56+
- `metric_value` (DECIMAL)
57+
- `created_at` (TIMESTAMP)
58+
59+
## Connection Details
60+
61+
- **Host**: `localhost` (or `host.docker.internal` from containers)
62+
- **Port**: `3306`
63+
- **Root Password**: `rootpassword`
64+
- **User**: `hoptimator`
65+
- **Password**: `hoptimator123`
66+
- **Databases**: `testdb`, `analytics`
67+
68+
## Sample Queries
69+
70+
Once deployed, you can query MySQL through Hoptimator:
71+
72+
```sql
73+
-- Query users from testdb
74+
SELECT * FROM "MYSQL"."testdb"."users";
75+
```
76+
77+
## Kubernetes Resources
78+
79+
The deployment creates:
80+
- **mysql-testdb** Database resource (catalog: testdb)
81+
- **mysql-analytics** Database resource (catalog: analytics)
82+
- **mysql-read-template** TableTemplate for reading
83+
- **mysql-write-template** TableTemplate for writing
84+
85+
## Testing Catalog Support
86+
87+
This setup demonstrates Hoptimator's catalog support:
88+
89+
1. **3-level naming**: `[catalog, schema, table]` internally
90+
2. **Multiple schemas**: Both `testdb` and `analytics` are accessible
91+
3. **JDBC metadata**: Schemas & tables are discovered automatically via MySQL JDBC metadata
92+
4. **Type mapping**: MySQL types are correctly mapped to Calcite types
93+
94+
## Direct MySQL Access
95+
96+
You can also connect directly to MySQL for debugging:
97+
98+
```bash
99+
# Connect to MySQL container
100+
docker exec -it hoptimator-mysql mysql -u hoptimator -phoptimatorpassword
101+
102+
mysql> show databases;
103+
+--------------------+
104+
| Database |
105+
+--------------------+
106+
| analytics |
107+
| information_schema |
108+
| performance_schema |
109+
| testdb |
110+
+--------------------+
111+
4 rows in set (0.01 sec)
112+
```
113+
114+
## Customization
115+
116+
To add more tables or data:
117+
1. Edit `init.sql` with your schema and data
118+
2. Run `make undeploy-mysql` to remove the old container
119+
3. Run `make deploy-mysql` to recreate with new schema
120+
121+
## Troubleshooting
122+
123+
**Container won't start:**
124+
```bash
125+
# Check logs
126+
docker logs hoptimator-mysql
127+
128+
# Ensure port 3306 is not in use
129+
lsof -i :3306
130+
```
131+
132+
**Connection refused:**
133+
```bash
134+
# Wait for MySQL to be ready (healthcheck)
135+
docker compose -f ./deploy/docker/mysql/docker-compose.yaml ps
136+
137+
# Test connection
138+
docker exec hoptimator-mysql mysqladmin ping -h localhost -u hoptimator -phoptimatorpassword
139+
```
140+
141+
**Tables not created:**
142+
```bash
143+
# Check if init.sql ran
144+
docker exec -it hoptimator-mysql mysql -u hoptimator -phoptimatorpassword -e "SHOW DATABASES;"
145+
docker exec -it hoptimator-mysql mysql -u hoptimator -phoptimatorpassword testdb -e "SHOW TABLES;"
146+
```
Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
services:
2+
mysql:
3+
image: mysql:8.0
4+
container_name: hoptimator-mysql
5+
hostname: mysql
6+
environment:
7+
MYSQL_ROOT_PASSWORD: rootpassword
8+
MYSQL_DATABASE: testdb
9+
MYSQL_USER: hoptimator
10+
MYSQL_PASSWORD: hoptimatorpassword
11+
ports:
12+
- "3306:3306"
13+
volumes:
14+
- ./init.sql:/docker-entrypoint-initdb.d/init.sql
15+
healthcheck:
16+
test: ["CMD", "mysqladmin", "ping", "-h", "localhost", "-u", "root", "-prootpassword"]
17+
start_period: 30s
18+
interval: 5s
19+
timeout: 5s
20+
retries: 10
21+
command: --default-authentication-plugin=mysql_native_password

deploy/docker/mysql/init.sql

Lines changed: 84 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,84 @@
1+
-- Create test database and tables for Hoptimator MySQL integration
2+
-- This script runs automatically when the MySQL container starts
3+
4+
USE testdb;
5+
6+
-- Create a users table
7+
CREATE TABLE IF NOT EXISTS users (
8+
user_id INT PRIMARY KEY AUTO_INCREMENT,
9+
username VARCHAR(50) NOT NULL UNIQUE,
10+
email VARCHAR(100) NOT NULL,
11+
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
12+
is_active BOOLEAN DEFAULT TRUE
13+
);
14+
15+
-- Create an orders table
16+
CREATE TABLE IF NOT EXISTS orders (
17+
order_id INT PRIMARY KEY AUTO_INCREMENT,
18+
user_id INT NOT NULL,
19+
product_name VARCHAR(100) NOT NULL,
20+
quantity INT NOT NULL,
21+
price DECIMAL(10, 2) NOT NULL,
22+
order_date TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
23+
status VARCHAR(20) DEFAULT 'pending',
24+
FOREIGN KEY (user_id) REFERENCES users(user_id)
25+
);
26+
27+
-- Create a products table
28+
CREATE TABLE IF NOT EXISTS products (
29+
product_id INT PRIMARY KEY AUTO_INCREMENT,
30+
product_name VARCHAR(100) NOT NULL,
31+
category VARCHAR(50),
32+
price DECIMAL(10, 2) NOT NULL,
33+
stock_quantity INT DEFAULT 0,
34+
description TEXT
35+
);
36+
37+
-- Insert sample data into users
38+
INSERT INTO users (username, email, is_active) VALUES
39+
('alice', '[email protected]', TRUE),
40+
('bob', '[email protected]', TRUE),
41+
('charlie', '[email protected]', FALSE),
42+
('diana', '[email protected]', TRUE);
43+
44+
-- Insert sample data into products
45+
INSERT INTO products (product_name, category, price, stock_quantity, description) VALUES
46+
('Laptop', 'Electronics', 999.99, 50, 'High-performance laptop'),
47+
('Mouse', 'Electronics', 29.99, 200, 'Wireless mouse'),
48+
('Desk Chair', 'Furniture', 199.99, 30, 'Ergonomic office chair'),
49+
('Coffee Mug', 'Kitchen', 12.99, 100, 'Ceramic coffee mug');
50+
51+
-- Insert sample data into orders
52+
INSERT INTO orders (user_id, product_name, quantity, price, status) VALUES
53+
(1, 'Laptop', 1, 999.99, 'completed'),
54+
(1, 'Mouse', 2, 29.99, 'completed'),
55+
(2, 'Desk Chair', 1, 199.99, 'pending'),
56+
(3, 'Coffee Mug', 3, 12.99, 'shipped'),
57+
(4, 'Laptop', 1, 999.99, 'pending');
58+
59+
-- Create a second schema for testing multi-schema support
60+
CREATE DATABASE IF NOT EXISTS analytics;
61+
62+
USE analytics;
63+
64+
-- Create a metrics table in the analytics schema
65+
CREATE TABLE IF NOT EXISTS daily_metrics (
66+
metric_id INT PRIMARY KEY AUTO_INCREMENT,
67+
metric_date DATE NOT NULL,
68+
metric_name VARCHAR(50) NOT NULL,
69+
metric_value DECIMAL(15, 2) NOT NULL,
70+
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
71+
);
72+
73+
-- Insert sample metrics data
74+
INSERT INTO daily_metrics (metric_date, metric_name, metric_value) VALUES
75+
(CURDATE(), 'revenue', 5000.00),
76+
(CURDATE(), 'orders', 150.00),
77+
(CURDATE(), 'active_users', 1200.00),
78+
(DATE_SUB(CURDATE(), INTERVAL 1 DAY), 'revenue', 4800.00),
79+
(DATE_SUB(CURDATE(), INTERVAL 1 DAY), 'orders', 145.00);
80+
81+
-- Grant permissions to hoptimator user on both databases
82+
GRANT ALL PRIVILEGES ON testdb.* TO 'hoptimator'@'%';
83+
GRANT ALL PRIVILEGES ON analytics.* TO 'hoptimator'@'%';
84+
FLUSH PRIVILEGES;

deploy/samples/mysqldb.yaml

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
apiVersion: hoptimator.linkedin.com/v1alpha1
2+
kind: Database
3+
metadata:
4+
name: mysql
5+
spec:
6+
catalog: MYSQL
7+
url: jdbc:mysql-hoptimator://url=jdbc:mysql://localhost:3306;user=hoptimator;password=hoptimatorpassword
8+
dialect: Calcite
9+
10+
---
11+
12+
apiVersion: hoptimator.linkedin.com/v1alpha1
13+
kind: TableTemplate
14+
metadata:
15+
name: mysql-template
16+
spec:
17+
databases:
18+
- mysql
19+
connector: |
20+
connector = mysql
21+
hostname = localhost
22+
port = 3306
23+
username = hoptimator
24+
password = hoptimatorpassword
25+
tables = {{table}}

hoptimator-api/src/main/java/com/linkedin/hoptimator/Source.java

Lines changed: 11 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -28,8 +28,18 @@ public String table() {
2828
return path.get(path.size() - 1);
2929
}
3030

31+
/**
32+
* Returns the schema name if present.
33+
*/
3134
public String schema() {
32-
return path.get(path.size() - 2);
35+
return path.size() >= 2 ? path.get(path.size() - 2) : null;
36+
}
37+
38+
/**
39+
* Returns the catalog name if present (3-level path), or null for 2-level paths.
40+
*/
41+
public String catalog() {
42+
return path.size() >= 3 ? path.get(path.size() - 3) : null;
3343
}
3444

3545
public List<String> path() {

hoptimator-api/src/main/java/com/linkedin/hoptimator/View.java

Lines changed: 11 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -22,8 +22,18 @@ public String table() {
2222
return path.get(path.size() - 1);
2323
}
2424

25+
/**
26+
* Returns the schema name if present.
27+
*/
2528
public String schema() {
26-
return path.get(path.size() - 2);
29+
return path.size() >= 2 ? path.get(path.size() - 2) : null;
30+
}
31+
32+
/**
33+
* Returns the catalog name if present (3-level path), or null for 2-level paths.
34+
*/
35+
public String catalog() {
36+
return path.size() >= 3 ? path.get(path.size() - 3) : null;
2737
}
2838

2939
public List<String> path() {

hoptimator-cli/build.gradle

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,7 @@ dependencies {
1111
implementation project(':hoptimator-demodb')
1212
implementation project(':hoptimator-jdbc')
1313
implementation project(':hoptimator-kafka')
14+
implementation project(':hoptimator-mysql')
1415
implementation project(':hoptimator-venice')
1516
implementation project(':hoptimator-k8s')
1617
implementation project(':hoptimator-util')

hoptimator-jdbc-driver-int/build.gradle

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,7 @@ dependencies {
1010
implementation project(':hoptimator-avro')
1111
implementation project(':hoptimator-demodb')
1212
implementation project(":hoptimator-kafka")
13+
implementation project(':hoptimator-mysql')
1314
implementation project(":hoptimator-venice")
1415
implementation libs.flink.jdbc
1516

hoptimator-jdbc/src/main/java/com/linkedin/hoptimator/jdbc/HoptimatorConnection.java

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,7 @@
66
import com.linkedin.hoptimator.avro.AvroConverter;
77
import com.linkedin.hoptimator.util.ConnectionService;
88
import com.linkedin.hoptimator.util.DelegatingConnection;
9+
import java.sql.DatabaseMetaData;
910
import java.sql.PreparedStatement;
1011
import java.sql.SQLException;
1112
import java.sql.Statement;
@@ -73,6 +74,12 @@ public PreparedStatement prepareStatement(String sql) throws SQLException {
7374
return connection.prepareStatement(sql);
7475
}
7576

77+
@Override
78+
public DatabaseMetaData getMetaData() throws SQLException {
79+
DatabaseMetaData metaData = connection.getMetaData();
80+
return new HoptimatorDatabaseMetaData(this, metaData);
81+
}
82+
7683
public Properties connectionProperties() {
7784
return connectionProperties;
7885
}

0 commit comments

Comments
 (0)