refactor: improve the image building performance#2175
Merged
Conversation
…get to reduce unnecessary 'docker cp'" Signed-off-by: zyy17 <zyylsxm@gmail.com>
Signed-off-by: zyy17 <zyylsxm@gmail.com>
Signed-off-by: zyy17 <zyylsxm@gmail.com>
waynexia
approved these changes
Aug 15, 2023
Member
waynexia
left a comment
There was a problem hiding this comment.
looks fascinating, just believe
Codecov Report
@@ Coverage Diff @@
## develop #2175 +/- ##
===========================================
- Coverage 85.01% 84.49% -0.53%
===========================================
Files 696 698 +2
Lines 111743 111745 +2
===========================================
- Hits 95000 94419 -581
- Misses 16743 17326 +583 |
MichaelScofield
pushed a commit
to MichaelScofield/greptimedb
that referenced
this pull request
Aug 16, 2023
* feat: handle multiple grpc deletes (GreptimeTeam#2150) * feat: handle multiple grpc deletes * fix: make DistDeleter::grpc_delete return usize * fix: remove backtrace from MissingTimeIndexColumn * fix: avoid using unwrap in PartitionRuleManager::split_delete_request * fix: simplify MissingTimeIndexColumn * refactor: improve the image building performance (GreptimeTeam#2175) * refactor: use '--output type=local' in 'build-greptime-by-buildx' target to reduce unnecessary 'docker cp'" Signed-off-by: zyy17 <zyylsxm@gmail.com> * refactor: improve the image building performance * ci: release centos dev builder * ci: use 'make build-by-dev-builder' to improve docker build performance * refactor: add 'which' command in centos * fix: add 'OUTPUT_DIR' to fix 'make docker-image-buildx' error * fix: fix incorrect dockerfile path Signed-off-by: zyy17 <zyylsxm@gmail.com> * refactor: remove configure-aws-credentials action and use env variables Signed-off-by: zyy17 <zyylsxm@gmail.com> * ci: update slack notification prompt * refactor: clean up the target directory before building artifacts of centos7 --------- Signed-off-by: zyy17 <zyylsxm@gmail.com> * docs: rfc of update metadata in single txn (GreptimeTeam#2165) * docs: rfc of update metadata in single txn * chore: apply suggestion from CR * feat: grpc handler result (GreptimeTeam#2107) * feat: grpc handler inner result * feat: ext header, x-greptime-err-code, x-greptime-err-msg * fix: sqlness case * chore: by comment * fix: convert status to Error * chore: minor `auth` crate change (GreptimeTeam#2176) * chore: pub auth_mysql * chore: pub all error * chore: remove back to error * chore: wrap failed permission check result to err * chore: minor change * feat: don't enable telemetry for debug building (GreptimeTeam#2177) --------- Signed-off-by: zyy17 <zyylsxm@gmail.com> Co-authored-by: Niwaka <61189782+NiwakaDev@users.noreply.github.com> Co-authored-by: zyy17 <zyylsxm@gmail.com> Co-authored-by: Weny Xu <wenymedia@gmail.com> Co-authored-by: JeremyHi <jiachun_feng@proton.me> Co-authored-by: shuiyisong <113876041+shuiyisong@users.noreply.github.com> Co-authored-by: dennis zhuang <killme2008@gmail.com>
paomian
pushed a commit
to paomian/greptimedb
that referenced
this pull request
Oct 19, 2023
* refactor: use '--output type=local' in 'build-greptime-by-buildx' target to reduce unnecessary 'docker cp'" Signed-off-by: zyy17 <zyylsxm@gmail.com> * refactor: improve the image building performance * ci: release centos dev builder * ci: use 'make build-by-dev-builder' to improve docker build performance * refactor: add 'which' command in centos * fix: add 'OUTPUT_DIR' to fix 'make docker-image-buildx' error * fix: fix incorrect dockerfile path Signed-off-by: zyy17 <zyylsxm@gmail.com> * refactor: remove configure-aws-credentials action and use env variables Signed-off-by: zyy17 <zyylsxm@gmail.com> * ci: update slack notification prompt * refactor: clean up the target directory before building artifacts of centos7 --------- Signed-off-by: zyy17 <zyylsxm@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
I hereby agree to the terms of the GreptimeDB CLA
What's changed and what's your intention?
Major changes
According to #2113, the current image building is too slow because of the poor performance of
make build-greptime-by-buildx.Why
make build-greptime-by-buildxis so slow? There are the following reasons:greptimebinary building is completed, we have to create a temporary image to copy the binary in which the image is too big because of too many rust object files. It costs lots of time in disk operations;To avoid that, using a dev-builder container to build the binary will be a better way. The
make build-by-dev-builderwill mount the source code into the container and compile it. After the building is completed, the binary is also on the host, and no need to copy it from the image.After testing, the nightly building time from 1h 32m 44s to 31m 26s. Maybe it's not fair because we don't run the integration tests for the second buildings, which will increase by about 15min.
Other changes
configure-aws-credentials;docker/;dev-builder-ubuntuanddev-builder-centos);make docker-imagecommand by usingmake build-by-dev-builder;Checklist
Refer to a related PR or issue link (optional)