Skip to content

Commit 2696cd1

Browse files
feat(s3tables-alpha): add support for partition spec, sort order, and table properties (#36811)
### Reason for this change This PR adds support for new S3 Tables Iceberg features to the L2 construct library, enabling users to configure partition specifications, sort orders, table properties, and schema field IDs when creating tables. These features are essential for optimizing query performance and data organization in Iceberg tables. ### Description of changes **Enhanced Table construct with new Iceberg metadata properties**: - Added `id` field to `SchemaFieldProperty` for schema field identification - Added `IcebergPartitionSpec` and `IcebergPartitionField` interfaces for partition configuration - Added `IcebergSortOrder` and `IcebergSortField` interfaces for sort order configuration - Added `tableProperties` support for custom Iceberg table properties - Updated `IcebergMetadataProperty` to include optional `icebergPartitionSpec`, `icebergSortOrder`, and `tableProperties` **Documentation**: - Added comprehensive examples for `IcebergPartitionField`, `IcebergPartitionSpec`, and `IcebergSortOrder` - Updated README with "Advanced Iceberg Table Configuration" section showing complete usage examples ### Description of how you validated changes - **Unit tests**: Added comprehensive test coverage for new features (192 tests passing) - **Integration test**: Created `integ.table-with-partition-sort.ts` to validate partition spec, sort order, and table properties - **Manual testing**: Successfully deployed and validated in Gamma environment where the CloudFormation resource type with new properties is available. Verified that partition spec, sort order, and table properties are correctly applied to the Iceberg table metadata. ### Checklist - [x] My code adheres to the [CONTRIBUTING GUIDE](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md) and [DESIGN GUIDELINES](https://github.com/aws/aws-cdk/blob/main/docs/DESIGN_GUIDELINES.md) ---- *By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
1 parent 7e68304 commit 2696cd1

26 files changed

Lines changed: 70990 additions & 2 deletions

packages/@aws-cdk/aws-s3tables-alpha/README.md

Lines changed: 76 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -94,6 +94,82 @@ const sampleTableWithSchema = new Table(scope, 'ExampleSchemaTable', {
9494

9595
Learn more about table buckets maintenance operations and default behavior from the [S3 Tables User Guide](https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-table-buckets-maintenance.html)
9696

97+
### Advanced Iceberg Table Configuration
98+
99+
You can configure partition specifications, sort orders, and table properties for optimized query performance.
100+
101+
The simplest way to add partitioning to your table:
102+
103+
```ts
104+
// Build a table with partition spec (minimal configuration)
105+
const partitionedTable = new Table(scope, 'PartitionedTable', {
106+
tableName: 'partitioned_table',
107+
namespace: namespace,
108+
openTableFormat: OpenTableFormat.ICEBERG,
109+
icebergMetadata: {
110+
icebergSchema: {
111+
schemaFieldList: [
112+
{ name: 'event_date', type: 'date', required: true },
113+
{ name: 'event_name', type: 'string' },
114+
],
115+
},
116+
icebergPartitionSpec: {
117+
fields: [
118+
{
119+
sourceId: 1,
120+
transform: IcebergTransform.IDENTITY,
121+
name: 'date_partition',
122+
},
123+
],
124+
},
125+
},
126+
});
127+
```
128+
129+
For full control, you can also configure sort orders and table properties:
130+
131+
```ts
132+
// Build a table with partition spec, sort order, and table properties
133+
const advancedTable = new Table(scope, 'AdvancedTable', {
134+
tableName: 'advanced_table',
135+
namespace: namespace,
136+
openTableFormat: OpenTableFormat.ICEBERG,
137+
icebergMetadata: {
138+
icebergSchema: {
139+
schemaFieldList: [
140+
{ id: 1, name: 'event_date', type: 'date', required: true },
141+
{ id: 2, name: 'user_id', type: 'string', required: true },
142+
],
143+
},
144+
icebergPartitionSpec: {
145+
specId: 0,
146+
fields: [
147+
{
148+
sourceId: 1,
149+
transform: IcebergTransform.IDENTITY,
150+
name: 'date_partition',
151+
fieldId: 1000,
152+
},
153+
],
154+
},
155+
icebergSortOrder: {
156+
orderId: 1,
157+
fields: [
158+
{
159+
sourceId: 1,
160+
transform: IcebergTransform.IDENTITY,
161+
direction: SortDirection.ASC,
162+
nullOrder: NullOrder.NULLS_LAST,
163+
},
164+
],
165+
},
166+
tableProperties: [
167+
{ key: 'write.format.default', value: 'parquet' },
168+
],
169+
},
170+
});
171+
```
172+
97173
### Controlling Table Bucket Permissions
98174

99175
```ts

0 commit comments

Comments
 (0)