Always use user-level pub cache #121802

sigurdm · 2023-03-02T15:41:53Z

Current flutter has a .pub_cache inside the flutter_root and merges that with ~/.pub_cache if that exists,

With this PR we will always use pubs own resolution for finding the pub cache.
To add packages to the flutter.zip download they are packaged as tar.gz and added to the pub-cache on first run by using the new pub cache preload functionality.

Fixes #37481

Pre-launch Checklist

I read the Contributor Guide and followed the process outlined there for submitting PRs.
I read the Tree Hygiene wiki page, which explains my responsibilities.
I read and followed the Flutter Style Guide, including Features we expect every widget to implement.
I signed the CLA.
I listed at least one issue that this PR fixes in the description above.
I updated/added relevant documentation (doc comments with ///).
I added new tests to check the change I am making, or this PR is test-exempt.
All existing and new tests are passing.

If you need help, consider asking for advice on the #hackers-new channel on Discord.

sigurdm · 2023-03-06T11:37:27Z

@godofredoc you might want to look at the changes to the release packaging script dev/bots/prepare_package.dart.

jonasfj

LGTM

dev/bots/prepare_package.dart

christopherfujino · 2023-03-06T20:18:31Z

dev/bots/prepare_package.dart

+  /// These archives will be installed in the user-level cache on first
+  /// following flutter command that accesses the cache.
+  ///
+  /// This assumes that all packages currently in the cache are installed from


Is this a safe assumption? What about users overriding PUB_HOSTED_URL? https://docs.flutter.dev/community/china#configuring-flutter-to-use-a-mirror-site

Yes, this assumption is only as a precondition to the function, inside the prepare_package script.

Edited the comment to clarify.

oh, I missed what library we were in :)

christopherfujino · 2023-03-07T17:55:58Z

dev/bots/prepare_package.dart

+  Future<void> _downloadPubPackageArchives() async {
+    final Pool pool = Pool(10); // Number of simultaneous downloads.
+    final http.Client client = http.Client();
+    final Directory preloadCache = Directory(path.join(flutterRoot.path, '.pub-preload-cache'));


Do we need to add this to the top-level repo .gitignore?

Yes! good point.
Done

christopherfujino · 2023-03-09T17:56:32Z

merged in upstream to fix linux build tests

christopherfujino

@Jasguerrero can you give final approval?

Jasguerrero

LGTM

christopherfujino

LGTM

zanderso

DBC

Did we manually inspect the results of prepare_package.dart to ensure that it has the same size and rough contents?

There are also many style issues in this PR. In particular, the style guide asks us to avoid breaking a line on an assignment operator. See: https://github.com/flutter/flutter/wiki/Style-guide-for-Flutter-repo#prefer-a-maximum-line-length-of-80-characters

zanderso · 2023-03-13T15:21:34Z

dev/bots/prepare_package.dart

+      final String name = package.key;
+      final Map<String, dynamic> versions = package.value as Map<String, dynamic>;
+      for (final String version in versions.keys) {
+        downloads.add(fetchPackageArchive(name, version));


If the pub cache on CI has multiple versions of the same package, will they all be included in the flutter.zip?

They should, as we iterate all versions for each package, and we store them as $name-$version.tar.gz.

Is that the same as the old behavior, or will the release .zip now contain many versions of the same package that aren't needed. Remember that the flutter/flutter pubspec.yaml files pin all packages to a single version.

We should have exactly the same set of package-versions as before.

The difference is that they are distributed as tar.gz files instead of inside a prepared cache.

We do the same steps to "warm up" a cache (by running pub get for some set of packages) and then, instead of folding that cache inside the zip, we iterate it and download the archive for each package.

DBC

Did we manually inspect the results of prepare_package.dart to ensure that it has the same size and rough contents?

I'm working on this currently, will update.

I ran prepare_package.dart before and after this change on the same beta commit:

dart prepare_package.dart --branch=beta --no-publish --revision=1a0a03a41d6613c10bab23508dee456f184b181b --temp_dir=$HOME/git/tmp/prepare_package

And verified that the set of packages are the same, as well as the versions. The only difference is that after this change the individual packages are stored as tarballs.

sigurdm · 2023-03-13T16:06:00Z

There are also many style issues in this PR. In particular, the style guide asks us to avoid breaking a line on an assignment operator. See: https://github.com/flutter/flutter/wiki/Style-guide-for-Flutter-repo#prefer-a-maximum-line-length-of-80-characters

If this sticks I can do a follow-up to clean up.

Follow-up to #121802 resolving some style issues.

Follow-up to flutter#121802 resolving some style issues.

Always use local cache

b084df8

flutter-dashboard bot added c: contributor-productivity Team-specific productivity, code health, technical debt. tool Affects the "flutter" command-line tool. See also t: labels. labels Mar 2, 2023

sigurdm requested a review from jonasfj March 2, 2023 15:43

sigurdm added 3 commits March 3, 2023 12:07

Fix test

52a4135

Fix test

feee392

Merge

467e5cc

sigurdm requested a review from christopherfujino March 6, 2023 11:36

sigurdm requested a review from godofredoc March 6, 2023 11:37

sigurdm changed the title ~~Always use local cache~~ Always use user-level pub cache Mar 6, 2023

sigurdm marked this pull request as ready for review March 6, 2023 11:54

jonasfj approved these changes Mar 6, 2023

View reviewed changes

dev/bots/prepare_package.dart Outdated Show resolved Hide resolved

dev/bots/prepare_package.dart Outdated Show resolved Hide resolved

Better retry, hash checks, use correct api, fix bugs

23510fa

christopherfujino reviewed Mar 6, 2023

View reviewed changes

Improve dartdoc

d342670

christopherfujino reviewed Mar 7, 2023

View reviewed changes

sigurdm and others added 2 commits March 9, 2023 08:39

Update .gitignore to ignore the preload cache

c67c69b

Merge branch 'master' into merged_cache

d580bdf

christopherfujino reviewed Mar 9, 2023

View reviewed changes

christopherfujino requested a review from Jasguerrero March 9, 2023 21:42

Jasguerrero approved these changes Mar 10, 2023

View reviewed changes

christopherfujino approved these changes Mar 10, 2023

View reviewed changes

sigurdm merged commit 1fd84f8 into flutter:master Mar 13, 2023

engine-flutter-autoroll mentioned this pull request Mar 13, 2023

Roll Flutter from 9ba0d08ebc07 to 739dc8b5a57c (2 revisions) flutter/packages#3449

Closed

engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request Mar 13, 2023

1fd84f8 Always use user-level pub cache (flutter/flutter#121802)

7172b83

zanderso reviewed Mar 13, 2023

View reviewed changes

engine-flutter-autoroll mentioned this pull request Mar 13, 2023

Roll Flutter from 9ba0d08ebc07 to 2d1a5cdc9506 (3 revisions) flutter/packages#3451

Closed

engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request Mar 13, 2023

1fd84f8 Always use user-level pub cache (flutter/flutter#121802)

a61e44c

engine-flutter-autoroll mentioned this pull request Mar 13, 2023

Roll Flutter from 9ba0d08ebc07 to 12ef7535a801 (8 revisions) flutter/packages#3453

Closed

engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request Mar 13, 2023

1fd84f8 Always use user-level pub cache (flutter/flutter#121802)

3ae10b6

sigurdm mentioned this pull request Mar 14, 2023

Fix style issues #122586

Merged

engine-flutter-autoroll mentioned this pull request Mar 14, 2023

Roll Flutter from 9ba0d08ebc07 to ac37dd261110 (23 revisions) flutter/packages#3456

Closed

engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request Mar 14, 2023

1fd84f8 Always use user-level pub cache (flutter/flutter#121802)

cfd3a13

engine-flutter-autoroll mentioned this pull request Mar 14, 2023

Roll Flutter from 9ba0d08ebc07 to d8a721106108 (24 revisions) flutter/packages#3457

Merged

engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request Mar 14, 2023

1fd84f8 Always use user-level pub cache (flutter/flutter#121802)

2e3aeff

gilice mentioned this pull request Apr 2, 2023

flutter: 3.3.8 -> 3.7.9 NixOS/nixpkgs#224090

Closed

12 tasks

engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request May 10, 2023

1fd84f8 Always use user-level pub cache (flutter/flutter#121802)

1ebc94c

engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request May 10, 2023

1fd84f8 Always use user-level pub cache (flutter/flutter#121802)

889215d

auto-submit bot pushed a commit that referenced this pull request May 16, 2023

Fix style issues (#122586)

788ceb3

Follow-up to #121802 resolving some style issues.

CaseyHillers pushed a commit to CaseyHillers/flutter that referenced this pull request May 24, 2023

Fix style issues (flutter#122586)

a0f72f5

Follow-up to flutter#121802 resolving some style issues.

sigurdm mentioned this pull request Jan 4, 2024

flutter create emits confusing language about caches unprompted #140628

Closed

Always use user-level pub cache #121802

Always use user-level pub cache #121802

Uh oh!

Conversation

sigurdm commented Mar 2, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pre-launch Checklist

Uh oh!

sigurdm commented Mar 6, 2023

Uh oh!

jonasfj left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

christopherfujino commented Mar 9, 2023

Uh oh!

christopherfujino left a comment

Choose a reason for hiding this comment

Uh oh!

Jasguerrero left a comment

Choose a reason for hiding this comment

Uh oh!

christopherfujino left a comment

Choose a reason for hiding this comment

Uh oh!

zanderso left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sigurdm commented Mar 13, 2023

Uh oh!

Uh oh!

sigurdm commented Mar 2, 2023 •

edited

Loading