Repo Size

[moonmeister]

Summary

Cloning the Gatsby repository is becoming a little absurd due to its size. The git clone transfers 550 MB compressed and uncompressed on disk the repo is 854 MB ( this takes several minutes to clone even on a decent connection). There has been one attempt to fix this in the past(#6486) though I'm not sure if they rewrote git history to purge the large files.

Getting this reduced would help everyone but would also encourage contributions from countries and areas where slower connections are the norm. At this point the repository size is not helping.

Relevant information

There are 92 files in the repo over 1mb.
Compressed (by git) size: 550 MB
Size on Disk: 850+ MB

124K    ./examples/no-plugins
172K    ./examples/using-asciidoc
244K    ./examples/hn
316K    ./examples/simple-auth
136K    ./examples/no-trailing-slashes
164K    ./examples/using-contentful
100K    ./examples/using-glamor
1.7M    ./examples/image-processing
100K    ./examples/feed
84K     ./examples/using-styled-jsx
164K    ./examples/using-typescript
156K    ./examples/sitemap
144K    ./examples/using-faker
72K     ./examples/using-styletron
112K    ./examples/using-stylus
5.9M    ./examples/using-remark-copy-linked-files
176K    ./examples/using-emotion-prismjs
168K    ./examples/using-redirects
188K    ./examples/using-page-transitions
88K     ./examples/using-styled-components
120K    ./examples/using-path-prefix
780K    ./examples/using-gatsby-image
2.5M    ./examples/using-sqip
516K    ./examples/using-shopify
148K    ./examples/using-css-modules
176K    ./examples/using-mobx
104K    ./examples/using-sass
108K    ./examples/using-excel
272K    ./examples/styleguide
468K    ./examples/using-type-definitions
180K    ./examples/using-drupal
172K    ./examples/using-wordpress
1004K   ./examples/using-javascript-transforms
100K    ./examples/using-csv
128K    ./examples/client-only-paths
84K     ./examples/using-emotion
132K    ./examples/using-hjson
544K    ./examples/using-jest
164K    ./examples/using-redux
368K    ./examples/using-i18n
144K    ./examples/using-mongodb
88K     ./examples/using-cxs
88K     ./examples/using-medium
332K    ./examples/graphql-reference
156K    ./examples/using-multiple-providers
96K     ./examples/using-page-loading-indicator
14M     ./examples/gatsbygram
72K     ./examples/using-jss
300K    ./examples/using-gatsby-source-graphql
152K    ./examples/using-local-plugins
108K    ./examples/using-gatsby-without-graphql
140K    ./examples/using-prefetching-preloading-modules
128K    ./examples/using-js-search
13M     ./examples/using-remark
46M     ./examples
60K     ./infrastructure/functions
128K    ./infrastructure
4.0K    ./.git/branches
44K     ./.git/refs
529M    ./.git/objects
16K     ./.git/info
56K     ./.git/logs
140K    ./.git/hooks
531M    ./.git
44K     ./www/plugins
28K     ./www/__mocks__
1.7M    ./www/static
4.1M    ./www/src
6.0M    ./www
104K    ./integration-tests/long-term-caching
412K    ./integration-tests/gatsby-pipeline
544K    ./integration-tests
48K     ./scripts/site-up-checker
28K     ./scripts/add-npm-owner
28K     ./scripts/check-publish-access
28K     ./scripts/get-unowned-packages
60K     ./scripts/gatsby-plugin-checker
364K    ./scripts
112K    ./packages/gatsby-plugin-subfont
104K    ./packages/gatsby-transformer-pdf
192K    ./packages/gatsby-plugin-page-creator
112K    ./packages/gatsby-remark-custom-blocks
128K    ./packages/gatsby-plugin-styletron
148K    ./packages/gatsby-transformer-xml
976K    ./packages/gatsby-source-wordpress
124K    ./packages/gatsby-remark-copy-linked-files
204K    ./packages/gatsby-plugin-sitemap
264K    ./packages/gatsby-cli
244K    ./packages/gatsby-source-drupal
108K    ./packages/gatsby-source-medium
212K    ./packages/gatsby-plugin-postcss
192K    ./packages/gatsby-remark-images
132K    ./packages/gatsby-remark-responsive-iframe
180K    ./packages/gatsby-plugin-catch-links
152K    ./packages/gatsby-source-graphql
128K    ./packages/gatsby-plugin-flow
128K    ./packages/gatsby-plugin-typescript
108K    ./packages/gatsby-source-hacker-news
268K    ./packages/gatsby-plugin-offline
204K    ./packages/gatsby-plugin-feed
144K    ./packages/gatsby-plugin-fullstory
168K    ./packages/gatsby-plugin-netlify
120K    ./packages/babel-preset-gatsby-package
120K    ./packages/gatsby-remark-katex
196K    ./packages/gatsby-transformer-documentationjs
116K    ./packages/gatsby-cypress
136K    ./packages/gatsby-plugin-nprogress
216K    ./packages/gatsby-remark-code-repls
548K    ./packages/gatsby-remark-prismjs
132K    ./packages/gatsby-plugin-preact
164K    ./packages/gatsby-remark-autolink-headers
696K    ./packages/gatsby-plugin-sharp
260K    ./packages/gatsby-transformer-react-docgen
196K    ./packages/gatsby-source-mongodb
296K    ./packages/gatsby-transformer-remark
172K    ./packages/gatsby-plugin-netlify-cms
4.7M    ./packages/gatsby
160K    ./packages/gatsby-plugin-facebook-analytics
108K    ./packages/gatsby-transformer-javascript-frontmatter
280K    ./packages/gatsby-source-filesystem
148K    ./packages/gatsby-transformer-excel
144K    ./packages/graphql-skip-limit
180K    ./packages/gatsby-remark-graphviz
112K    ./packages/gatsby-source-npm-package-search
160K    ./packages/gatsby-plugin-layout
64K     ./packages/gatsby-plugin-no-sourcemaps
172K    ./packages/gatsby-plugin-sass
104K    ./packages/gatsby-transformer-asciidoc
132K    ./packages/gatsby-plugin-remove-trailing-slashes
1.3M    ./packages/gatsby-source-lever
120K    ./packages/gatsby-plugin-jss
120K    ./packages/gatsby-plugin-cxs
1.4M    ./packages/gatsby-transformer-sqip
240K    ./packages/gatsby-telemetry
140K    ./packages/gatsby-transformer-toml
124K    ./packages/gatsby-source-wikipedia
124K    ./packages/gatsby-remark-smartypants
148K    ./packages/gatsby-transformer-yaml
124K    ./packages/gatsby-plugin-emotion
152K    ./packages/gatsby-transformer-json
136K    ./packages/gatsby-plugin-guess-js
208K    ./packages/gatsby-image
136K    ./packages/gatsby-transformer-hjson
108K    ./packages/babel-preset-gatsby
144K    ./packages/gatsby-react-router-scroll
152K    ./packages/gatsby-plugin-canonical-urls
144K    ./packages/gatsby-transformer-csv
116K    ./packages/gatsby-plugin-styled-jsx
120K    ./packages/gatsby-plugin-styled-components
208K    ./packages/gatsby-transformer-sharp
708K    ./packages/gatsby-codemods
140K    ./packages/gatsby-plugin-react-helmet
160K    ./packages/gatsby-plugin-coffeescript
104K    ./packages/gatsby-plugin-lodash
188K    ./packages/gatsby-link
260K    ./packages/gatsby-dev-cli
45M     ./packages/gatsby-transformer-screenshot
104K    ./packages/gatsby-source-faker
108K    ./packages/gatsby-transformer-javascript-static-exports
140K    ./packages/gatsby-plugin-google-analytics
128K    ./packages/gatsby-plugin-glamor
140K    ./packages/gatsby-plugin-twitter
108K    ./packages/gatsby-plugin-google-tagmanager
148K    ./packages/gatsby-remark-images-contentful
604K    ./packages/gatsby-source-contentful
152K    ./packages/gatsby-plugin-google-gtag
108K    ./packages/gatsby-plugin-react-css-modules
188K    ./packages/gatsby-plugin-less
108K    ./packages/gatsby-plugin-create-client-paths
148K    ./packages/babel-plugin-remove-graphql-queries
176K    ./packages/gatsby-plugin-stylus
160K    ./packages/gatsby-plugin-typography
156K    ./packages/gatsby-source-shopify
144K    ./packages/gatsby-remark-embed-snippet
272K    ./packages/gatsby-plugin-manifest
69M     ./packages
52K     ./benchmarks/plugin-manifest
124K    ./benchmarks/markdown
100K    ./benchmarks/create-pages
88K     ./benchmarks/query
368K    ./benchmarks
1.7M    ./starters/blog
576K    ./starters/hello-world
1008K   ./starters/default
3.2M    ./starters
134M    ./docs/blog
12M     ./docs/tutorial
912K    ./docs/contributing
39M     ./docs/docs
28K     ./docs/features
4.8M    ./docs/creators
190M    ./docs
76K     ./plop-templates/package
64K     ./plop-templates/example
144K    ./plop-templates
344K    ./themes/gatsby-theme-blog-mdx
112K    ./themes/gatsby-theme-blog-core
1.1M    ./themes/gatsby-theme-blog
64K     ./themes/gatsby-starter-theme-blog-mdx
1.6M    ./themes
40K     ./.github/ISSUE_TEMPLATE
68K     ./.github
316K    ./e2e-tests/path-prefix
1.1M    ./e2e-tests/development-runtime
1.8M    ./e2e-tests/gatsby-image
720K    ./e2e-tests/production-runtime
3.8M    ./e2e-tests
1.7M    ./flow-typed/npm
1.7M    ./flow-typed
16K     ./.forestry/snippets
32K     ./.forestry/front_matter
64K     ./.forestry
20K     ./.circleci
854M    .

Optional Solutions

• Clean up stale branches that are not needed
  My thought was starting by deleting branches that have merge or closed PRs. Anything that hasn't had a PR + is older than a certain time maybe we delete (or maybe give the author notice it will be deleted). Anything with an open pr can be left.

• Cleanup detatched commits and other git things that don't matter -
  I've run across this but I don't entirely understand what it is doing and if there are other things that could be done

• Compressing images and purge from history - http://blog.jessitron.com/2013/08/finding-and-removing-large-files-in-git.html
  Troubles here is people might just keep adding large files...Not sure if it's possible to write a script that gets triggered by git hooks to compress any images being added to the repo.

• fix gatsby-plugin-screenshots to not need to bundle chrome - @Ankcorn
  Chrome bundle is 44MB.

• Git-LFS - this has been brought up before and we'd need to look into the affect on ease of contributions.

• Move images out of the repo -
  If LFS isn't an option maybe moving to CMS like contentful that could handle assets would be a better alternative. If that's not an option the website/images/blog cloud be move to its own
  repository.

Prompts

What methods are we okay to move ahead with?

If a method has been given the go ahead and you want to tackle a method let us know and submit a PR...

What other options are there for reducing repo size that we can consider?

[anantoghosh]

I would like to mention bfg https://rtyley.github.io/bfg-repo-cleaner/ which has worked very well for me.

[DSchau]

Agreed on a lot of this, and I think this is certainly worth considering, but your initial numbers are a little off.

We recommend using a shallow clone (e.g. --depth 1), which in doing so gets the repo size down to 246.98 Mb (on my machine?)

I think you raise some work that's worth doing, specifically investigation into whether we can avoid bundling Chrome.

As far as the others:

Cleaning up branches

(not as concerning, and in cloning you only get the master branch, right?)

Compressing images and purge from history

I get a little squeamish about re-writing history, but if this can be done cleanly (e.g. on a test repo/fork of Gatsby perhaps?) I'd be open to it.

Git-LFS

This is for truly huge files, right? I'm not sure we have many that would be worthy of this. Also - I want to keep ease of contribution in the forefront of our minds.

In general, cloning is a one-time, sunk cost, so I'd urge us to not make changes that improve the initial set-up cost, but degrade or complicate the experience down the line.

Move images out of the repo

We feel pretty strongly that whenever possible we want to keep content in the monorepo, so same idea here!

[moonmeister]

@DSchau Yes, using --depth 1 is helpful but how is the developer experience? The means you're only downloading the latest copy...I don't think you can even checkout other branches, and if you're trying to look at commit history you might really me lost. this might be useful for some but not sure it solves this problem in its entirety.

Cleaning up branches will help a full clone. Git makes copies of all changed files...any branches with commits will add to the size of the repository as a whole.

I agree that we can't trade long-term usability for upfront costs. I won't worry about LFS, or additional repo solutions.

One thing your repo size from using depth tells me is that there is a lot in history that probably needs cleaning up. Second, it also tells me that those ~90 files that are over 1MB represent a significant more portion of the repo size then I initially thought. Chrome is a 5th of the repo it seems.

I think right now the things we should focus on are:

• Getting chrome removed
• compressing images and videos as best as possible (and make this run on git hooks)
• removing old versions of these from history (@anantoghosh Yes I've seen bfg but never used it, do you have experience here, are you able to assist on this?)
• cleaning up unused branches (I'll start by deleting closed/merged branches if that's okay?)

[thecodingaviator]

Netlify has recently released a new feature for serving large images, I'm not sure if it is for large files too. I see we're hosted on Netlify so we can make use of that feature.

[Daniel15]

If you really do want large files in a Git repo (instead of using something like Git LFS), another option is to store the large assets in a separate repo and pull it in as a submodule. Then at least people that don't want/need the static files can just avoid updating the submodules when pulling.

That Netlify feature looks pretty useful though!

[gatsbot]

Hiya!

This issue has gone quiet. Spooky quiet. 👻

We get a lot of issues, so we currently close issues after 30 days of inactivity. It’s been at least 20 days since the last update here.

If we missed this issue or if you want to keep it open, please reply here. You can also add the label "not stale" to keep this issue open!

As a friendly reminder: the best way to see this issue, or any other, fixed is to open a Pull Request. Check out gatsby.dev/contributefor more information about opening PRs, triaging issues, and contributing!

Thanks for being a part of the Gatsby community! 💪💜

[gatsbot]

Hey again!

It’s been 30 days since anything happened on this issue, so our friendly neighborhood robot (that’s me!) is going to close it.

Please keep in mind that I’m only a robot, so if I’ve closed this issue in error, I’m HUMAN_EMOTION_SORRY. Please feel free to reopen this issue or create a new one if you need anything else.

As a friendly reminder: the best way to see this issue, or any other, fixed is to open a Pull Request. Check out gatsby.dev/contribute for more information about opening PRs, triaging issues, and contributing!

Thanks again for being part of the Gatsby community!