Skip to content

refactor: improve the image building performance#2175

Merged
zyy17 merged 10 commits intodevelopfrom
refactor/refine-makefile
Aug 15, 2023
Merged

refactor: improve the image building performance#2175
zyy17 merged 10 commits intodevelopfrom
refactor/refine-makefile

Conversation

@zyy17
Copy link
Copy Markdown
Collaborator

@zyy17 zyy17 commented Aug 15, 2023

I hereby agree to the terms of the GreptimeDB CLA

What's changed and what's your intention?

Major changes

According to #2113, the current image building is too slow because of the poor performance of make build-greptime-by-buildx.

Why make build-greptime-by-buildx is so slow? There are the following reasons:

  1. Cannot re-use compiled objects when using Docker BuildKit(maybe can use cache?);
  2. When the greptime binary building is completed, we have to create a temporary image to copy the binary in which the image is too big because of too many rust object files. It costs lots of time in disk operations;

To avoid that, using a dev-builder container to build the binary will be a better way. The make build-by-dev-builder will mount the source code into the container and compile it. After the building is completed, the binary is also on the host, and no need to copy it from the image.

After testing, the nightly building time from 1h 32m 44s to 31m 26s. Maybe it's not fair because we don't run the integration tests for the second buildings, which will increase by about 15min.

Other changes

  1. Refine the upload-artifacts action by removing configure-aws-credentials;
  2. Refine the directory layout of docker/;
  3. Refine the dev-builder process(there is two different dev-builder, dev-builder-ubuntu and dev-builder-centos);
  4. Refine the make docker-image command by using make build-by-dev-builder;
  5. Refine the slack notification message;

Checklist

  • I have written the necessary rustdoc comments.
  • I have added the necessary unit tests and integration tests.

Refer to a related PR or issue link (optional)

Copy link
Copy Markdown
Member

@waynexia waynexia left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks fascinating, just believe

@codecov
Copy link
Copy Markdown

codecov Bot commented Aug 15, 2023

Codecov Report

Merging #2175 (aeb1302) into develop (e924b44) will decrease coverage by 0.53%.
Report is 3 commits behind head on develop.
The diff coverage is n/a.

@@             Coverage Diff             @@
##           develop    #2175      +/-   ##
===========================================
- Coverage    85.01%   84.49%   -0.53%     
===========================================
  Files          696      698       +2     
  Lines       111743   111745       +2     
===========================================
- Hits         95000    94419     -581     
- Misses       16743    17326     +583     

Copy link
Copy Markdown
Member

@daviderli614 daviderli614 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ditto

@zyy17 zyy17 added this pull request to the merge queue Aug 15, 2023
Merged via the queue into develop with commit 32c3ac4 Aug 15, 2023
@zyy17 zyy17 deleted the refactor/refine-makefile branch August 15, 2023 09:38
MichaelScofield pushed a commit to MichaelScofield/greptimedb that referenced this pull request Aug 16, 2023
* feat: handle multiple grpc deletes (GreptimeTeam#2150)

* feat: handle multiple grpc deletes

* fix: make DistDeleter::grpc_delete return usize

* fix: remove backtrace from MissingTimeIndexColumn

* fix: avoid using unwrap in PartitionRuleManager::split_delete_request

* fix: simplify MissingTimeIndexColumn

* refactor: improve the image building performance (GreptimeTeam#2175)

* refactor: use '--output type=local' in 'build-greptime-by-buildx' target to reduce unnecessary 'docker cp'"

Signed-off-by: zyy17 <zyylsxm@gmail.com>

* refactor: improve the image building performance

* ci: release centos dev builder

* ci: use 'make build-by-dev-builder' to improve docker build performance

* refactor: add 'which' command in centos

* fix: add 'OUTPUT_DIR' to fix 'make docker-image-buildx' error

* fix: fix incorrect dockerfile path

Signed-off-by: zyy17 <zyylsxm@gmail.com>

* refactor: remove configure-aws-credentials action and use env variables

Signed-off-by: zyy17 <zyylsxm@gmail.com>

* ci: update slack notification prompt

* refactor: clean up the target directory before building artifacts of centos7

---------

Signed-off-by: zyy17 <zyylsxm@gmail.com>

* docs: rfc of update metadata in single txn (GreptimeTeam#2165)

* docs: rfc of update metadata in single txn

* chore: apply suggestion from CR

* feat: grpc handler result (GreptimeTeam#2107)

* feat: grpc handler inner result

* feat: ext header, x-greptime-err-code, x-greptime-err-msg

* fix: sqlness case

* chore: by comment

* fix: convert status to Error

* chore: minor `auth` crate change (GreptimeTeam#2176)

* chore: pub auth_mysql

* chore: pub all error

* chore: remove back to error

* chore: wrap failed permission check result to err

* chore: minor change

* feat: don't enable telemetry for debug building (GreptimeTeam#2177)

---------

Signed-off-by: zyy17 <zyylsxm@gmail.com>
Co-authored-by: Niwaka <61189782+NiwakaDev@users.noreply.github.com>
Co-authored-by: zyy17 <zyylsxm@gmail.com>
Co-authored-by: Weny Xu <wenymedia@gmail.com>
Co-authored-by: JeremyHi <jiachun_feng@proton.me>
Co-authored-by: shuiyisong <113876041+shuiyisong@users.noreply.github.com>
Co-authored-by: dennis zhuang <killme2008@gmail.com>
paomian pushed a commit to paomian/greptimedb that referenced this pull request Oct 19, 2023
* refactor: use '--output type=local' in 'build-greptime-by-buildx' target to reduce unnecessary 'docker cp'"

Signed-off-by: zyy17 <zyylsxm@gmail.com>

* refactor: improve the image building performance

* ci: release centos dev builder

* ci: use 'make build-by-dev-builder' to improve docker build performance

* refactor: add 'which' command in centos

* fix: add 'OUTPUT_DIR' to fix 'make docker-image-buildx' error

* fix: fix incorrect dockerfile path

Signed-off-by: zyy17 <zyylsxm@gmail.com>

* refactor: remove configure-aws-credentials action and use env variables

Signed-off-by: zyy17 <zyylsxm@gmail.com>

* ci: update slack notification prompt

* refactor: clean up the target directory before building artifacts of centos7

---------

Signed-off-by: zyy17 <zyylsxm@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants