-
Notifications
You must be signed in to change notification settings - Fork 122
Alias index transform #1049
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Alias index transform #1049
Conversation
Signed-off-by: n-dohrmann <[email protected]>
Signed-off-by: n-dohrmann <[email protected]>
Signed-off-by: n-dohrmann <[email protected]>
Signed-off-by: n-dohrmann <[email protected]>
Signed-off-by: n-dohrmann <[email protected]>
…anagement into alias_index_transform
Signed-off-by: n-dohrmann <[email protected]>
Codecov ReportAttention: Patch coverage is
❌ Your project check has failed because the head coverage (62.64%) is below the target coverage (70.00%). You can increase the head coverage or adjust the target coverage.
Additional details and impacted files@@ Coverage Diff @@
## main #1049 +/- ##
=============================================
- Coverage 74.81% 62.64% -12.18%
+ Complexity 2810 2339 -471
=============================================
Files 367 367
Lines 16518 16527 +9
Branches 2362 2366 +4
=============================================
- Hits 12358 10353 -2005
- Misses 2857 5020 +2163
+ Partials 1303 1154 -149 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
@@ -63,6 +65,20 @@ class TransformIndexer( | |||
throw TransformIndexException("Failed to create the target index") | |||
} | |||
} | |||
val writeIndexMetadata = clusterService.state().metadata.indicesLookup[targetIndex]!!.writeIndex |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This line should inside the below if block?
if (clusterService.state().metadata.hasAlias(targetIndex)) { | ||
// return error if no write index with the alias | ||
if (writeIndexMetadata == null) { | ||
throw TransformIndexException("Target index alias has no write index") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
target_index [$targetIndex] is an alias but doesn't have write index
val putMappingReq = PutMappingRequest(writeIndexMetadata?.index?.name).source(targetFieldMappings) | ||
val mapResp: AcknowledgedResponse = client.admin().indices().suspendUntil { | ||
putMapping(putMappingReq) | ||
} | ||
if (!mapResp.isAcknowledged) { | ||
logger.error("Target index mapping request failed") | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Also move this inside to above if block?
Signed-off-by: n-dohrmann <[email protected]>
Signed-off-by: n-dohrmann <[email protected]>
Signed-off-by: n-dohrmann <[email protected]>
Signed-off-by: n-dohrmann <[email protected]>
Signed-off-by: n-dohrmann <[email protected]>
Signed-off-by: n-dohrmann <[email protected]>
…anagement into alias_index_transform
Signed-off-by: n-dohrmann <[email protected]>
Signed-off-by: n-dohrmann <[email protected]>
Signed-off-by: n-dohrmann <[email protected]>
Signed-off-by: n-dohrmann <[email protected]>
Signed-off-by: n-dohrmann <[email protected]>
Signed-off-by: n-dohrmann <[email protected]>
Signed-off-by: n-dohrmann <[email protected]>
Hi, |
Any update on this? @bowenlan-amzn @n-dohrmann |
@HenryTheSir @MahendraAkkina I don't think @n-dohrmann still works on this. Would you want to create another PR from this? |
@bowenlan-amzn Thanks for checking on this. I am not familiar with the dev process. Is there a doc talking about it?. I see that you were involved heavily in reviewing it. So wanted to check if its something you can help with creating a new PR and help with getting it merged? |
@MahendraAkkina this doc should help https://github.com/opensearch-project/index-management/blob/main/DEVELOPER_GUIDE.md |
@bowenlan-amzn @vaibhoag I am looking at the fix and am little confused. I see that createTargetIndex is failing as it is not able to create an index which is actually an alias. The code fix has additional code after the first if. I think all this code needs to go under first if check. @bowenlan-amzn Is that what you were suggesting? |
Something like this, that is I moved the additional code into the first if
|
@MahendraAkkina commented on Apr 16, 2025, 6:00 PM PDT:
Yeah, we should move the alias check to the top, first see if provided targetIndex is an alias, then see if we want to create that index. |
Instead of updating the mappings every time a transform runs, especially in continuous mode, I think choosing not to update them for aliases. My reasoning is that updating mappings on each run feels inconsistent with how index mappings are handled, where they are typically defined once at index creation. In the specific scenario with rollover aliases, we should rely on index templates to pre-define all the necessary fields when new indices are created, thus eliminating the need for continuous mapping updates. @vaibhoag, thoughts? Can you or someone else at amazon :-) help with contributing to the this project?
|
Hey @MahendraAkkina I think instead of updating mappings, we should first validate the Actually this depends on how we are creating targetAlias, are we expecting users to create it? @bowenlan-amzn |
@vaibhoag That should work. I would assume it would check this every time the transform operation runs (Example: In the case when continuous is set to true and is scheduled to run every hour, this check would happen once an hour). We can imagine the writeIndex to be the same for a long time (as it typically depends on the rollover policy). Might not be a big performance hit, but just wanted to point out. |
@vaibhoag Will you be able to contribute by creating a new PR and help with getting it merged? Would be very helpful. |
Issue #, if available:
#656
Description of changes:
Adding to
createTargetIndex
method in theTransformIndexer
module - want to support using aliases for variable target indices on transform jobs.CheckList:
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.