D8 dex merger fails when a synthetic class is placed on a different shard than its container class

[mauriciogg]

Description of the bug:

a desugared lambda is packed in a different shard than the one
from its original class D8 will fail with the following message:

Attempt at compiling intermediate artifact without its context
e.g.

I'm encountering an issue when compiling my app with the following classes:
org/chromium/base/MemoryPressureListener ``org/chromium/base/MemoryPressureListener$$ExternalSyntheticLambda0

bazel fails with the following message

Error in bazel-out/k8-fastbuild/bin/apps/hermosa/hermosa-services/realtime/mirroring-sample-app/dexsplits/apk/9.shard.zip:org/chromium/base/MemoryPressureListener$$ExternalSyntheticLambda0.class.dex:
Attempt at compiling intermediate artifact without its context
Merge failed: Compilation failed to complete, origin: bazel-out/k8-fastbuild/bin/apps/hermosa/hermosa-services/realtime/mirroring-sample-app/dexsplits/apk/9.shard.zip:org/chromium/base/MemoryPressureListener$$ExternalSyntheticLambda0.class.dex
Exception in thread "main" com.android.tools.r8.CompilationFailedException: Merge failed: Compilation failed to complete, origin: bazel-out/k8-fastbuild/bin/apps/hermosa/hermosa-services/realtime/mirroring-sample-app/dexsplits/apk/9.shard.zip:org/chromium/base/MemoryPressureListener$$ExternalSyntheticLambda0.class.dex
at com.google.devtools.build.android.r8.DexFileMerger.main(DexFileMerger.java:389)
Target //apps/hermosa/hermosa-services/realtime/mirroring-sample-app:apk failed to build

Here the container class is in shard #8 and the lambda class is in shard #9

Running with d8 dexmerger and native multidex.

What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

Its hard to provide a small reproducible example since it needs to be an app large enough to exceed the size of a dex file and the sharding has to result in a lambda class ending up in a different shard than the container class.

Which operating system are you running Bazel on?

Linux/Mac

What is the output of bazel info release?

Internal build merged with master at commit b6803a2

If bazel info release returns development version or (@non-git), tell us how you built Bazel.

No response

What's the output of git remote get-url origin; git rev-parse master; git rev-parse HEAD ?

No response

Have you found anything relevant by searching the web?

No response

Any other information, logs, or outputs that you want to share?

No response

[nkoroste]

@ahumesky

[Bencodes]

We are also seeing this in our repo using the latest rolling release (6.0.0-pre.20220922.1)

[lukaciko]

@ahumesky can this issue be added to Bazel 6.1. release blockers? As you noted #16369 does not completely solve it. We are hesitant to upgrade to 6.0 even with that fix since we might run into the same issue with another of synthetic classes at any point.

[ahumesky]

Yes I think we can get the fix in for 6.1.

But a question for those of you running into this issue -- we're not seeing this internally, so I'm trying to figure out what differences might be causing this with bazel. Is anyone using the min_sdk_version attribute of android_binary (related: bazelbuild/rules_android#319), or perhaps modified the hard coded dexing API level?

[pswaminathan]

We are not setting either of those. We are setting min API level in the manifest using the uses-sdk tag. For an application that exhibits it, it is happening consistently. However, triggering it doesn't seem to be related to changing build rules. i.e. We have experienced with an application in which the change that triggered this behavior is purely a code change. It's a little hard for me to create a repro, given that it depends on internal code.

Also—I don't see this added to 6.1 release blockers. Is that still a possibility?

[ahumesky]

Thanks -- turns out the reason we weren't seeing this internally isn't related to API level exactly, but internally we have other code paths that avoid this problem. We're removing these codepath soon so that the code matches internally and externally, so we'll want the fix for this as well.

It's been pretty tricky to get the fix for this working in all cases, and has required reworking the PR for this a fair bit (#16369 (comment)). I'm rerunning all our tests again to see if anything still fails with the updates to the PR, so I'm a little hesitant now to block 6.1.0 on this if there are more cases that need to be addressed. If we can't get the fix in for 6.1.0 we cherrypick this into 6.2.0.

[ahumesky]

After working through #16369 and talking with the D8 team, they propose (copied from #16369):

1. Update CompatDexBuilder to use the D8 DexFilePerClassFile consumer instead of DexIndexed (this should be a no op for the build as the current build there is already doing one dex per class file, but it uses another D8 API such that it can get the synthetic info).

2. Update the D8 DexFilePerClassFile accept function to also receive the "synthetic context" information if the compiled class is a compiler synthesized class.

3. Update CompatDexBuilder to write the synthetic context information to some build meta info for later access by the DexFileSplitter. Something like this is already done for the desugar dependencies which are written to a proto in META-INF/desugar_deps.

4. Update DexFileSplitter to use the meta-inf of synthetic classes to group with the right contexts.

The D8 team would handle 1 and 2, and the bazel team and/or community could do 3 and 4. This avoids coupling to D8 implementation details and having to parse dex files and hopefully reduces the code complexity a bit.