Migrated packages can't be successfully added to deps, even if in the allow list

[jakemac53]

See my pending cl here to pull in the migrated fixnum, for nnbd strong mode benchmarks https://dart-review.googlesource.com/c/sdk/+/150170.

There are some build actions that appear to not be picking up the correct .dart_tool/package_config.json file, or something along those lines.

[jakemac53]

cc @franklinyow we can't publish any of these packages until we have an sdk where they are in the allow list, so this is a blocker for flutter depending on any NNBD migrated packages (and thus a blocker for the flutter tech preview)

[franklinyow]

@leafpetersen

[jakemac53]

Note that this is also blocking all packages developed in the sdk from migrating to nnbd (meta, js, are important ones)

[leafpetersen]

@nshahan does dartdevc correctly handle package_config.json yet?

[leafpetersen]

@johnniwinther is there any work required from the back ends to make the allow list work?

[nshahan]

@nshahan does dartdevc correctly handle package_config.json yet?

We didn't implement any handling of the package_config.json file in DDC. If we need to pass some information to the CFE we can do that but I'll wait to hear what Johnni says.

[leafpetersen]

@nshahan suggested the possibility that the error is from the pre-built sdk that is checked in rather than from dartdevc. @jakemac53 with your CL , if you try to run dartdevc.dart with the version of dart from the pre-built sdk, do you see this error? If so, this is the problem.

[leafpetersen]

Ok, that's the issue. If I patch in the CL from @jakemac53, gclient sync, build and then try running with the local build and the pre-built I get the following:

leafp-macbookpro:sdk leafp$ ./xcodebuild/ReleaseX64/dart-sdk/bin/dart pkg/dev_compiler/bin/dartdevc.dart 
Please specify the output file location. For example:
    -o PATH/TO/OUTPUT_FILE.js
leafp-macbookpro:sdk leafp$ ./tools/sdks/dart-sdk/bin/dart pkg/dev_compiler/bin/dartdevc.dart 
third_party/pkg/fixnum/lib/src/int64.dart:446:10: Error: This requires the 'non-nullable' language feature to be enabled.
Try updating your pubspec.yaml to set the minimum SDK constraint to 2.9 or higher, and running 'pub get'.
    Int64? o;
         ^

@athomas I think we just need a new prebuilt SDK checked in.

[jakemac53]

Sounds good, are we going to have to do this for every package we want to migrate in? Should we try to do them all in one shot then?

[sigmundch]

This should only affect packages that are indirectly used by dartdevc. My guess is that most of these are indirect dependencies from using bazel_worker.

If the whitelist already includes all those dependencies already, I expect this would be a one-time thing. A prebuild SDK that has the full whitelist will be good to go, even if you roll the other packages later.

An alternative solution here is to reduce the ddc dependencies. In this case, we can provide a separate dartdevc entrypoint that we use while building the SDK. Since we only have the dependency on bazel_worker to make the integration with webdev smoother, it won't really a necessary dependency of the compiler for the local build. If we add a second entrypoint that doesn't have this dependency and use it in the build steps, we'd also elide this issue. For instance, something among these lines: https://dart-review.googlesource.com/c/sdk/+/150860

[athomas]

I have https://dart-review.googlesource.com/c/sdk/+/150931 in the CQ to update the checked-in SDKs.

[jakemac53]

If the whitelist already includes all those dependencies already, I expect this would be a one-time thing. A prebuild SDK that has the full whitelist will be good to go, even if you roll the other packages later.

The allow list does not contain anything today. It sounds like this will happen for ~all updates to that list, so we should probably update it all in one go to avoid this recurring issue.

[jakemac53]

An alternative solution here is to reduce the ddc dependencies. In this case, we can provide a separate dartdevc entrypoint that we use while building the SDK.

That could help, but I assume it still has some other deps like source_maps etc?

[sigmundch]

There are some, but not as many. Here is the list from under dev_compiler/lib:
args build_integration cli_util dev_compiler _fe_analyzer_shared front_end kernel meta path source_maps source_span

Dart2js emits sourcemaps straight as a string and elides the last two dependencies. Probably not worth looking into that for dartdevc at this time.

[leafpetersen]

Per discussion offline, I think the plan for this is that we preemptively add all of the packages of interest to the allow list, roll that new pre-built SDK in, and then we believe that we should at that point be able to roll in ported versions of the packages without a new pre-built SDK being required.

[jakemac53]

Ok, who is going to own adding all the packages to the list?

[leafpetersen]

@jakemac53 If we get you the list, would you mind wrangling the CL since you've already gone through it once? @kevmoo and/or @srawlins could probably put together the package list, maybe cross check with @jonahwilliams for the flutter side? It's not the end of the world if don't get it 100% on the first go, just need enough so that we don't have to go through this every day.

[jakemac53]

@leafpetersen Sure, I can build the list with @kevmoo ?

Is what we care about the list of packages required to run flutter and dart tests with strong null safety?

[leafpetersen]

Is what we care about the list of packages required to run flutter and dart tests in with strong null safety?

That's a reasonable start. Probably everything that we pull in to the sdk in DEPS is worth adding as well (so long as they are packages that we control).

[jakemac53]

https://dart-review.googlesource.com/c/sdk/+/151400 - I did not add the packages in DEPS for now. That expands the list quite a lot and I don't think we really want to do that (it increases the risk for every additional dep we add there)

[jakemac53]

@sortie have you been able to look at the benchmark failure for fixnum? Is this also using the pre-built sdk or something along those lines or is there additional work that is needed to unblock landing these DEPS changes?

[jakemac53]

It looks like I can't even add these packages to the allow list successfully, there is some sort of hack in for package:js today so that its language version is left empty (opting it in). This does not match the actual SDK constraint it contains.

Additionally after fixing this the analyzer job still complains - it appears to think that it is in fact opted in regardless of the .dart_tool/package_config.json contents. cc @scheglov any ideas?

And apparently we also have jobs that are compiling it with strong mode, even though it has not been migrated. cc @nshahan can we remove this?

[scheglov]

As it is implemented in analyzer, allowed_experiments.json takes precedence over .dart_tool/package_config.json. Given that we use this file for things that you are doing now, I can change it to do whatever you want. I'm working on a related issue now, so I could change behavior here as well.

[jakemac53]

All that allowed_experiments.json is supposed to do is allow those packages to use nnbd, if they are opted in. It is not intended to opt them in.

[scheglov]

I will fix the analyzer implementation.
Sorry about this.

[scheglov]

https://dart-review.googlesource.com/c/sdk/+/151423 that should fix it landed.

[nshahan]

I filed an issue for the missing compile time error in package:js #42362. I think fixing that will actually break the SDK build so we can't just yet.

@jakemac53 and I discussed offline and our plan is to migrate package:js along with the change for package:fixnum and rely on temporarily using the current SDK instead of the prebuilt to get those two packages on the allow list and opted in.

[athomas]

@jakemac53 The benchmark problem is caused by golem not sending the package-config.json to the benchmark runners and the tryjob correctly detects that it would break golem. @sortie will have to fix that so that your CL can land.

[jakemac53]

Ok, in the meantime I have landed changes to the allow list - so at this point we are blocked only on pulling in nnbd migrated DEPS.

I do think we should do that, to help make sure we don't break any packages that are being pinned for the tech preview (or are at least aware they will break so we can forward-fix them, although that won't be an option for flutter due to pinning).

[sortie]

I have deployed the change adding .dart_tool/package_config.json to our benchmark builds.

[kevmoo]

@jakemac53 – we're still blocked here by flutter-specific tests, right?

[franklinyow]

Currently blocking by b/159723440 SDK build rules do not support NNBD deps

[franklinyow]

b/159723440 resolved
Block by b/159903870 Flutter tests fail on NNBD packages

[jakemac53]

OK - so I have tried running with the flutter engine patch workflow (see https://dart-review.googlesource.com/c/sdk/+/152860). This has gotten me past some of the failures but I still see some others.

For the flutter-hhh bots I still see this failure specifically https://logs.chromium.org/logs/dart/buildbucket/cr-buildbucket.appspot.com/8876134992926522320/+/steps/build_host_debug/0/stdout, which I think still means something is picking up the old package:collection, but is opting it in?

[jakemac53]

For some more information here, this looks to be failing while trying to compile this analyzer script third_party/dart/pkg/analyzer/tool/summary/build_sdk_summaries.dart, and it looks pretty clearly like it is using an outdated version of package:collection still during this step.

@a-siva any ideas?

[jakemac53]

With flutter/engine#19469, package:collection and package:typed_data have successfully landed all the way through to flutter.

This process does currently require breaking the HHH bots, so we will want to land prs in batches together to avoid to much breakage.

However I think this is sufficient to close this issue.