-
Notifications
You must be signed in to change notification settings - Fork 9.1k
HDFS-17711. Change fsimage loading progress percentage discontinuous to continuous #7299
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
💔 -1 overall
This message was automatically generated. |
Closed and reopened PR due to not linked jira issue properly. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
+1
Pre-submit gave it a -1 for no new tests, but I don't think we can easily write a meaningful new test for this.
I committed this to trunk and branch-3.5. @Last-remote11 , thank you for the contribution! |
Thank you for review! |
Description of PR
When the name node is restarted with dfs.image.parallel.load enabled, the completion percentage(at namenode web) is reflacted after each thread finished after fsimage loading.
For example, if dfs.image.parallel.target.sections=12(default), the completion percentage increases by 0% -> 8.3% (1/12) -> 16.6% (2/12) ->.... -> 100%.
This is because each thread uses a separate counter.
According docs of
org.apache.hadoop.hdfs.server.namenode.startupprogress.getCounter
, the counter guarantees thread-safe, so it seems that each fimage loading thread can use the same counter.How was this patch tested?
run hadoop unit test and tested my test hadoop cluster (version 3.4.0)
For code changes:
LICENSE
,LICENSE-binary
,NOTICE-binary
files?