Commit 14fe460
authored
Introduce branching and support for spark.wap.branch (#406)
## Summary
Implement Iceberg table branching functionality in OpenHouse internal
catalog with comprehensive testing infrastructure.
Strategically, we trust that only a valid client performs the operation.
This is safe because in fact O.S. model has the client write
metadata.json and send the pointer to the catalog server. So,
GlueCatalog (e.g. gluetableoperations.java), DynamoDbCatalog,
jdbctableoperations.java do client-only metadata operations (no
server-side validation exists). The client behavior is also covered
extensively by Iceberg compatibility tests applied to openhouse.
When we trust the client, we simply serialize the client request and
later deserialize it. Since TableMetadata fields lack public access and
only some are exposed via the builder, we start from the base metadata,
take the serialized client request, and apply it to base by comparing
with request and: truncating snapshot IDs from base, adding new IDs from
the request, and moving the current pointer according to the request.
how to test
oh suite tests - this branch
```
cd /Users/chbush/code/li/project_openhouse/openhouse && \
export JAVA_HOME=$(/usr/libexec/java_home -v 17) && \
./gradlew clean \
:integrations:spark:spark-3.5:openhouse-spark-3.5-itest:catalogTest \
--tests "com.linkedin.openhouse.spark.catalogtest.BranchTestSpark3_5" \
:iceberg:openhouse:internalcatalog:test \
--tests "com.linkedin.openhouse.internal.catalog.SnapshotDiffApplierTest" \
--tests "com.linkedin.openhouse.internal.catalog.OpenHouseInternalTableOperationsTest" \
:integrations:spark:spark-3.1:openhouse-spark-itest:catalogTest \
--tests "com.linkedin.openhouse.spark.catalogtest.WapIdTest" \
:apps:openhouse-spark-apps-1.5_2.12:test \
--tests "com.linkedin.openhouse.catalog.e2e.WapIdJavaTest"
```
compatibility tests - includes this branch + new shim files, and also
requires this iceberg
https://github.com/cbb330/openhouse/tree/working-wap-compatibility
https://github.com/cbb330/iceberg/tree/test-harness
```
# 1. Clean up local maven repo to ensure fresh publish
rm -rf /Users/chbush/code/li/project_openhouse/openhouse/build && \
rm -rf /Users/chbush/.m2/repository/com/linkedin/openhouse/tables-test-fixtures* && \
# 2. Build and publish the fixtures jar (containing the new test list)
cd /Users/chbush/code/li/project_openhouse/openhouse && \
export JAVA_HOME=$(/usr/libexec/java_home -v 17) && \
./gradlew :tables-test-fixtures:tables-test-fixtures-iceberg-1.5_2.12:clean \
:tables-test-fixtures:tables-test-fixtures-iceberg-1.5_2.12:shadowJar \
:tables-test-fixtures:tables-test-fixtures-iceberg-1.5_2.12:publishToMavenLocal && \
# 3. Run the compatibility tests
cd /Users/chbush/code/li/project_openhouse/iceberg && \
export JAVA_HOME=$(/usr/libexec/java_home -v 17) && \
./gradlew :iceberg-core:cleanOpenhouseCompatibilityTest \
:iceberg-spark:iceberg-spark-3.5_2.12:cleanOpenhouseCompatibilityTest \
:iceberg-spark:iceberg-spark-extensions-3.5_2.12:cleanOpenhouseCompatibilityTest \
:iceberg-core:openhouseCompatibilityTest \
:iceberg-spark:iceberg-spark-3.5_2.12:openhouseCompatibilityTest \
:iceberg-spark:iceberg-spark-extensions-3.5_2.12:openhouseCompatibilityTest \
-PopenhouseCompatibilityCoordinate=com.linkedin.openhouse:tables-test-fixtures-iceberg-1.5_2.12:0.0.19-SNAPSHOT:uber
```
## Changes
- [ ] Client-facing API Changes
- [ ] Internal API Changes
- [ ] Bug Fixes
- [ ] New Features
- [ ] Performance Improvements
- [ ] Code Style
- [ ] Refactoring
- [ ] Documentation
- [ ] Tests
For all the boxes checked, please include additional details of the
changes made in this pull request.
## Testing Done
<!--- Check any relevant boxes with "x" -->
- [ ] Manually Tested on local docker setup. Please include commands
ran, and their output.
- [ ] Added new tests for the changes made.
- [ ] Updated existing tests to reflect the changes made.
- [ ] No tests added or updated. Please explain why. If unsure, please
feel free to ask for help.
- [ ] Some other form of testing like staging or soak time in
production. Please explain.
For all the boxes checked, include a detailed description of the testing
done for the changes made in this pull request.
# Additional Information
- [ ] Breaking Changes
- [ ] Deprecations
- [ ] Large PR broken into smaller PRs, and PR plan linked in the
description.
For all the boxes checked, include additional details of the changes
made in this pull request.1 parent ff9a40c commit 14fe460
File tree
14 files changed
+3281
-642
lines changed- iceberg/openhouse/internalcatalog/src
- main/java/com/linkedin/openhouse/internal/catalog
- test
- java/com/linkedin/openhouse/internal/catalog
- resources
- integrations/spark/spark-3.5/openhouse-spark-itest/src/test/java/com/linkedin/openhouse/spark/catalogtest
- e2e
- services/tables/src/test/java/com/linkedin/openhouse/tables/e2e/h2
14 files changed
+3281
-642
lines changedLines changed: 5 additions & 5 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
59 | 59 | | |
60 | 60 | | |
61 | 61 | | |
62 | | - | |
63 | | - | |
64 | 62 | | |
65 | 63 | | |
66 | 64 | | |
67 | 65 | | |
68 | 66 | | |
69 | 67 | | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
70 | 71 | | |
71 | 72 | | |
72 | | - | |
73 | | - | |
| 73 | + | |
74 | 74 | | |
75 | 75 | | |
76 | | - | |
| 76 | + | |
77 | 77 | | |
78 | 78 | | |
79 | 79 | | |
| |||
Lines changed: 40 additions & 141 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
23 | 23 | | |
24 | 24 | | |
25 | 25 | | |
26 | | - | |
27 | 26 | | |
28 | 27 | | |
29 | 28 | | |
| |||
34 | 33 | | |
35 | 34 | | |
36 | 35 | | |
37 | | - | |
38 | | - | |
39 | 36 | | |
40 | 37 | | |
41 | 38 | | |
| |||
44 | 41 | | |
45 | 42 | | |
46 | 43 | | |
47 | | - | |
48 | 44 | | |
49 | 45 | | |
50 | 46 | | |
| |||
56 | 52 | | |
57 | 53 | | |
58 | 54 | | |
| 55 | + | |
59 | 56 | | |
60 | 57 | | |
61 | 58 | | |
62 | 59 | | |
63 | | - | |
64 | 60 | | |
65 | 61 | | |
66 | 62 | | |
| |||
70 | 66 | | |
71 | 67 | | |
72 | 68 | | |
73 | | - | |
74 | | - | |
75 | 69 | | |
76 | 70 | | |
77 | 71 | | |
| |||
267 | 261 | | |
268 | 262 | | |
269 | 263 | | |
270 | | - | |
| 264 | + | |
271 | 265 | | |
272 | 266 | | |
273 | | - | |
274 | | - | |
| 267 | + | |
| 268 | + | |
275 | 269 | | |
276 | 270 | | |
277 | 271 | | |
278 | 272 | | |
279 | 273 | | |
280 | | - | |
281 | | - | |
282 | | - | |
283 | | - | |
284 | | - | |
285 | | - | |
286 | 274 | | |
287 | 275 | | |
288 | 276 | | |
289 | 277 | | |
290 | | - | |
291 | | - | |
292 | | - | |
| 278 | + | |
| 279 | + | |
| 280 | + | |
| 281 | + | |
| 282 | + | |
| 283 | + | |
| 284 | + | |
| 285 | + | |
| 286 | + | |
| 287 | + | |
| 288 | + | |
| 289 | + | |
| 290 | + | |
| 291 | + | |
| 292 | + | |
| 293 | + | |
| 294 | + | |
| 295 | + | |
| 296 | + | |
| 297 | + | |
| 298 | + | |
| 299 | + | |
| 300 | + | |
| 301 | + | |
| 302 | + | |
| 303 | + | |
| 304 | + | |
| 305 | + | |
| 306 | + | |
| 307 | + | |
| 308 | + | |
| 309 | + | |
| 310 | + | |
293 | 311 | | |
294 | 312 | | |
295 | | - | |
| 313 | + | |
296 | 314 | | |
297 | 315 | | |
298 | 316 | | |
| |||
314 | 332 | | |
315 | 333 | | |
316 | 334 | | |
317 | | - | |
| 335 | + | |
318 | 336 | | |
319 | 337 | | |
320 | 338 | | |
| |||
358 | 376 | | |
359 | 377 | | |
360 | 378 | | |
361 | | - | |
| 379 | + | |
362 | 380 | | |
363 | 381 | | |
364 | 382 | | |
| |||
531 | 549 | | |
532 | 550 | | |
533 | 551 | | |
534 | | - | |
535 | | - | |
536 | | - | |
537 | | - | |
538 | | - | |
539 | | - | |
540 | | - | |
541 | | - | |
542 | | - | |
543 | | - | |
544 | | - | |
545 | | - | |
546 | | - | |
547 | | - | |
548 | | - | |
549 | | - | |
550 | | - | |
551 | | - | |
552 | | - | |
553 | | - | |
554 | | - | |
555 | | - | |
556 | | - | |
557 | | - | |
558 | | - | |
559 | | - | |
560 | | - | |
561 | | - | |
562 | | - | |
563 | | - | |
564 | | - | |
565 | | - | |
566 | | - | |
567 | | - | |
568 | | - | |
569 | | - | |
570 | | - | |
571 | | - | |
572 | | - | |
573 | | - | |
574 | | - | |
575 | | - | |
576 | | - | |
577 | | - | |
578 | | - | |
579 | | - | |
580 | | - | |
581 | | - | |
582 | | - | |
583 | | - | |
584 | | - | |
585 | | - | |
586 | | - | |
587 | | - | |
588 | | - | |
589 | | - | |
590 | | - | |
591 | | - | |
592 | | - | |
593 | | - | |
594 | | - | |
595 | | - | |
596 | | - | |
597 | | - | |
598 | | - | |
599 | | - | |
600 | | - | |
601 | | - | |
602 | | - | |
603 | | - | |
604 | | - | |
605 | | - | |
606 | | - | |
607 | | - | |
608 | | - | |
609 | | - | |
610 | | - | |
611 | | - | |
612 | | - | |
613 | | - | |
614 | | - | |
615 | | - | |
616 | | - | |
617 | | - | |
618 | | - | |
619 | | - | |
620 | | - | |
621 | | - | |
622 | | - | |
623 | | - | |
624 | | - | |
625 | | - | |
626 | | - | |
627 | | - | |
628 | | - | |
629 | | - | |
630 | | - | |
631 | | - | |
632 | | - | |
633 | | - | |
634 | | - | |
635 | | - | |
636 | | - | |
637 | | - | |
638 | | - | |
639 | | - | |
640 | | - | |
641 | | - | |
642 | | - | |
643 | | - | |
644 | | - | |
645 | | - | |
646 | | - | |
647 | | - | |
648 | | - | |
649 | | - | |
650 | | - | |
651 | | - | |
652 | | - | |
653 | 552 | | |
654 | 553 | | |
655 | 554 | | |
| |||
Lines changed: 0 additions & 96 deletions
This file was deleted.
0 commit comments