S3 tree optimizations #2913
Open
S3 tree optimizations #2913
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Add AdaptiveEffort to dynamically adjust optimization parameters based on observed improvement rates (increase effort when making progress, decrease when plateauing) - Add block moves optimization that moves groups of dependent conditions together to escape local minima - Add cost-based tie-breaking using BddCostEstimator when multiple positions have the same node count
This commit adds a new function to the rules engine, ite, that performs
an if-then-else check on a boolean expression without branching. By not
needing to branch in the decision tree, we avoid SSA transforms on
divergent branches which would create syntactically different but
semantically identical expressions that the BDD cannot deduplicate.
This commit also adds an S3-specific decision tree transform that
canonicalizes S3Express rules for better BDD compilation:
1. AZ extraction: Rewrites position-dependent substring operations to
use a single split(Bucket, "--")[1] expression across all branches
2. URL canonicalization: Uses ITE to compute FIPS/DualStack URL segments,
collapsing 4 URL variants into a single template with {_s3e_fips} and
{_s3e_ds} placeholders
3. Auth scheme canonicalization: Uses ITE to select sigv4 vs
sigv4-s3express based on DisableS3ExpressSessionAuth
The transform makes the rules tree ~30% larger but enables dramatic BDD
compression by making URL templates identical across FIPS/DualStack/auth
variants. Endpoints that previously appeared distinct now collapse into
single BDD results, reducing nodes and results by ~43%.
Add nullable boolean transform. Improve variable and expression sharing. Add S3 BDD integration test. Add TreeMapper to simplify transforms.
Split the monolithic S3TreeRewriter into three independent transforms: - S3AzCanonicalizerTransform: substring → split for AZ extraction - S3RegionUnifierTransform: unify Region/bucketArn#region/us-east-1 - S3ExpressEndpointTransform: canonicalize FIPS/DualStack/auth Added region unification pass that rewrites all region references to _effective_std_region (for aws-global → us-east-1 mapping) or _effective_arn_region (for UseArnRegion logic). This enables additional BDD sharing across endpoints with different region sources. S3 BDD stats: 77 conditions, 97 results, 484 nodes after sifting.
mtdowling
force-pushed
the
s3-tree-optimizations
branch
from
543ac43 to
99d556a
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Background
Testing
Links
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.