Proposal: remove big old blobs from the git history

[tbruyelle]

The CLI repo currently has a huge amount of big blobs, in particular because it contains many binary files. The largest one, nodetime, represents itself 5,5 Gb of data !

This has bad consequences, first, clone or any operations based on history are very slow (clone takes more than 4min), but more importantly, this forces us to disable the sumdb when we request hash (see golang/go#56174).

There's no million way to fix this, on my side, I know only once, it's called BFG (for those who has the reference : ) https://rtyley.github.io/bfg-repo-cleaner/

When used with the --strip-blobs-bigger-than flag, it removes all big old blobs larger than the value of the flag. Existing big blogs stay unmodified. For the demo, I ran it with --strip-blobs-bigger-than 10M, and pushed the result to https://github.com/tbruyelle/cli-diet. This new repo takes less ~30s to be cloned.

⚠️ Now the downsides ⚠️

• history is rewritten, so during the process, everyone should stop interacting with the repo, until the new cleaned repo is pushed with --force. Then everyone should clone it again.
• since old binaries won't exist anymore, it's no longer possible to build a CLI from an old tag (or more precisely, the binary won't contain the binaries, which could lead to failures with some commands)
• Since the commits are rewritten, there's new hashes, I need to check what's happening with existing github releases, since they are based on hash.

[fadeev]

Does this mean the curl | bash method of installing older binaries will not work?

[tbruyelle]

Does this mean the curl | bash method of installing older binaries will not work?

No, this will continue to work, even with old releases, because get.ignite.com uses the github releases and the binaries are stored in their assets.

I just need to ensure that github releases are preserved when we rewrite completely the history of the repo 😅 (ref last downside)

[aljo242]

I think this is worth exploring further for sure

[tbruyelle]

I investigated a little bit more :

• BFG considers 3 kinds of commits : clean, dirty and modified.

  • a clean commit is unchanged, it keeps its original hash
  • a dirty commit is a commit that doesn't fulfill the requirements (for example with the flag --strip-blobs-bigger-than 10M, the requirement is the commit size shouldn't be larger than 10M)
  • a modified commit is a commit that fulfill the requirement but doesn't keep its original hash

• BFG starts from the first commit and as soon as it detects a dirty commit, rewrites that dirty commit and all the following ones until the latest commit. All commits before the first dirty are clean (unchanged), while all the commits after are either dirty or modified, depending on whether they fulfill the requirement or not.

• github releases are preserved, because tags that are based on a dirty/modified commit are preserved.

• dirty/modified commits receives a message suffix Former-commit-id: sha-of-the-original-commit

• HEAD is considered protected, so even if it's dirty, it's not cleaned by BFG (so for instance files larger than 10M present in HEAD are preserved with --strip-blobs-bigger-than 10M).

[ilgooz]

My 2c, this could be great but if it requires rewriting the history (hash), I would say we shouldn't because we shouldn't change the truth and comparability between releases and past commits.

[tbruyelle]

My 2c, this could be great but if it requires rewriting the history (hash), I would say we shouldn't because we shouldn't change the truth and comparability between releases and past commits.

That's also a downside yes, typically, if history is rewritten, and a user downloads a binary from an old release, ignite version will output a hash that no longer exists in the repo :/

I analyzed a little further the ouput of git rev-list --objects --all | git cat-file --batch-check='%(objecttype) %(objectname) %(objectsize) %(rest)' | sed -n 's/^blob //p' | sort --numeric-sort --key=2 | cut -c 1-12,41- | $(command -v gnumfmt || echo numfmt) --field=2 --to=iec-i --suffix=B --padding=7 --round=nearest which lists all the blobs per size, and wrote a program that sums the size per file.

The following list gives the size per file, where the size is the sum of all the blobs where the file is modified (I filtered out the size that are smaller than 1Mb) :

1.027MB   starport/services/chain/chain.go
1.047MB   docs/blogcopy/docs/static/openapi.yml
1.053MB   starport/templates/typed/map/stargate.go
1.076MB   docs/03
1.084MB   docs/cli/index.md
1.137MB   ignite/services/network/publish_test.go
1.233MB   ignite/cli/ghrs-data/stargazers.csv
1.363MB   starport/ui/public/images/brian-workshop.png
1.4MB     ignite/pkg/cosmosclient/cosmosclient.go
1.433MB   starport/services/networkbuilder/networkbuilder.go
1.634MB   starport/interface/cli/starport/packrd/packed-packr.go
1.705MB   tendermint/starport/ghrs-data/views_clones_aggregate.csv
1.748MB   starport/templates/app/stargate/app/app.go.plush
1.807MB   tendermint/starport/ghrs-data/stargazers.csv
1.983MB   starport/services/chain/serve.go
2.307MB   assets/ignitecli.png
2.307MB   ignite/services/chain/testdata/version/mars.v0.2.tar.gz
2.307MB   ignite/services/chain/testdata/version/mars.v0.2-3-gaae48b7.tar.gz
2.517MB   assets/ignite-cli.png
2.732MB   starport/templates/app/launchpad/vue/package-lock.json
2.828MB   go.mod
2.831MB   starport/ui/app/dist/js/chunk-vendors.664483e9.js
3.168MB   starport/templates/app/stargate/vue/package-lock.json
4.823MB   starport/services/plugin/fixtures/fixtures.so
4.86MB    starport/templates/app/stargate/go.sum
5.579MB   changelog.md
5.977MB   starport/pkg/protoc/data/protoc-linux-arm64
6.537MB   ignite/templates/app/stargate/go.sum
7.723MB   scripts/data/gen-nodetime/package-lock.json
8.913MB   starport/ui/app/dist/js/chunk-vendors.664483e9.js.map
9.212MB   starport/ui/package-lock.json
9.647MB   ignite/pkg/protoc/data/protoc-linux-amd64
9.961MB   ignite/pkg/protoc/data/protoc-linux-arm64
10.02MB   ignite-hq/cli/latest-report/report.md
10.2MB    ignite-hq/cli/latest-report/report.html
10.2MB    ignite-hq/cli/latest-report/report_for_pdf.html
10.59MB   starport/pkg/protoc/data/protoc-linux-amd64
12.58MB   ignite/pkg/protoc/data/protoc-darwin-arm64
12.84MB   ignite-hq/cli/latest-report/report.pdf
13.78MB   docs/package-lock.json
15.83MB   ignite/pkg/protoc-gen-dart/data/protoc-gen-dart_darwin_arm64
19.52MB   ignite/cli/latest-report/report.md
19.81MB   ignite/cli/latest-report/report.html
19.81MB   ignite/cli/latest-report/report_for_pdf.html
21.17MB   ignite/cli/latest-report/report.pdf
24.43MB   ignite/pkg/protoc-gen-dart/data/protoc-gen-dart_darwin_amd64
25.17MB   ignite/pkg/protoc/data/protoc-darwin-amd64
25.17MB   starport/pkg/protoc/data/protoc-darwin-amd64
34.71MB   ignite/pkg/protoc-gen-dart/data/protoc-gen-dart_linux_arm64
36.07MB   ignite/pkg/protoc-gen-dart/data/protoc-gen-dart_linux_amd64
43.51MB   tendermint/starport/latest-report/report.pdf
77.72MB   go.sum
79.69MB   starport/pkg/protobufjs/js_darwin.go
81.68MB   starport/pkg/protoc-gen-dart/data/protoc-gen-dart_darwin_amd64
87.03MB   starport/pkg/protobufjs/js_linux.go
95.42MB   starport/pkg/nodetime/nodetime-linux-arm64.tar.gz
106.8MB   starport/pkg/protoc-gen-dart/data/protoc-gen-dart_linux_amd64
114.1MB   tendermint/starport/latest-report/report.md
114.8MB   tendermint/starport/latest-report/report.html
114.8MB   tendermint/starport/latest-report/report_for_pdf.html
157.9MB   starport/ui/dist-go/statik/statik.go
160.4MB   starport/pkg/nodetime/nodetime-darwin-arm64.tar.gz
242.2MB   starport/pkg/nodetime/nodetime-darwin.tar.gz
255.9MB   starport/pkg/nodetime/nodetime-linux.tar.gz
354.4MB   starport/pkg/nodetime/data/nodetime-darwin-amd64.tar.gz
367MB     starport/pkg/nodetime/data/nodetime-linux-amd64.tar.gz
556.8MB   ignite/pkg/nodetime/data/nodetime-linux-arm64.tar.gz
558.9MB   ignite/pkg/nodetime/data/nodetime-linux-amd64.tar.gz
602.9MB   ignite/pkg/nodetime/data/nodetime-darwin-arm64.tar.gz
626MB     ignite/pkg/nodetime/data/nodetime-darwin-amd64.tar.gz
1.021GB   starport/pkg/nodetime/nodetime-darwin-amd64.tar.gz
1.076GB   starport/pkg/nodetime/nodetime-linux-amd64.tar.gz
7.452GB   TOTAL

Problem, the biggest ones date from starport, which means we cannot mitigate the rewrite to only recent commits, it wouldn't be efficient.

Surprisingly, go.sum changes take 70Mb of data in the repo :)

[tbruyelle]

I found an interesting alternative, that also has the benefit of solving the issue for the future. Indeed, even if we fix our history with BFG, in the future the binary files will be changed again and again, which will grow the git history again an again.

The alternative is git lfs. This is a simple git extension that uses the github Large File Storage API. With lfs, large files are no longer stored in the repo, only a pointer to that file is stored.

Now the cool part is git lfs migrate which rewrites the git history to ensure every file we selected is moved to a git lfs pointer. That means unlike BFG which removes the larges files, which makes the old versions unusable, git lfs migrate keeps the old version usable, the large files have just moved to pointers.

I need to make further tests, but this looks like a promising alternative.

[gjermundgaraba]

Since the binaries are built during CI, could we not release the binaries as an actual release and just download them in the make file?
There could also be a separate repo that just deals with this to not mess up history.

Keep them out of the repo entierly. They are not source code and it should be possible to keep them somewhere else. If you need the version used to be in source control, then the version alone could be in source control, and the bump triggered by releases.

[tbruyelle]

As mentioned by @clockworkgr, lfs was disabled because of some issues, so finally it's probably not a solution https://github.com/ignite/cli/pull/755/files. Apparently this was due to bandwidth limitation, do you have more details on that @ilgooz ?

[tbruyelle]

Since the binaries are built during CI, could we not release the binaries as an actual release and just download them in the make file? There could also be a separate repo that just deals with this to not mess up history.

Keep them out of the repo entierly. They are not source code and it should be possible to keep them somewhere else. If you need the version used to be in source control, then the version alone could be in source control, and the bump triggered by releases.

This would work ofc, but if we could avoid having external storage to manage, that would be simpler. That's why we explore other solutions first.

[aljo242]

@albttx adding you to discussiion

[albttx]

I'm new to this, but here is my thought,

I didn't suffer (yet) from the size of the git repository, and i don't like the idea of re-writing the history.

I believe ~4m is ok for a programmer to clone the repository, and we can still use --depth=1 in scripts and reduce the clone time to ~10s

$ time git clone --depth=1 https://github.com/ignite/cli
[...]
real	0m10.989s
user	0m4.723s
sys	0m2.105s

[tbruyelle]

Hey @albttx 👋

Indeed most of us don't suffer from the current size of the repo, but sum.golang.org does !
See the related golang issue I created [0], we actually can't run go get on a pseudo-version of the CLI, because sum.golang.org timeouts, because of the size of the repo.

[0] golang/go#56174