Binding Kotlin itself crashes `generator --only-xml-adjuster` #466

mattleibow · 2019-08-13T19:18:15Z

I am trying to bind Kotlin so that it can be consumed by Kotlin-based libraries, and the generator is not happy.

This is the library kotlin-stdlib-1.3.41.jar

I just create a new binding project and build. Because at this point I don't want any actual types, I added <remove-node path="/api/package" /> to the transforms.

The generator crashes with this error:
https://gist.github.com/mattleibow/92dff72582ec972b214283d8fe739a9c

The api.xml.class-parse file is this:
https://gist.github.com/mattleibow/34af078b7aca818dd0c5381e075896d1

I can reproduce the crash at this point in the build:

mono /Library/Frameworks/Xamarin.Android.framework/Versions/10.0.0.4/lib/xamarin.android/xbuild/Xamarin/Android/generator.exe \
    api.xml.class-parse \
    --only-xml-adjuster \
    --xml-adjuster-output=api.xml \
    --csdir=src \
    --ref=/Library/Frameworks/Mono.framework/External/xbuild-frameworks/MonoAndroid/v1.0/mscorlib.dll \
    --ref=/Library/Frameworks/Mono.framework/External/xbuild-frameworks/MonoAndroid/v1.0/Java.Interop.dll \
    --ref=/Library/Frameworks/Mono.framework/External/xbuild-frameworks/MonoAndroid/v9.0/Mono.Android.dll \
    --ref=/Library/Frameworks/Mono.framework/External/xbuild-frameworks/MonoAndroid/v1.0/mscorlib.dll \
    --ref=/Library/Frameworks/Mono.framework/External/xbuild-frameworks/MonoAndroid/v1.0/System.Core.dll \
    --ref=/Library/Frameworks/Mono.framework/External/xbuild-frameworks/MonoAndroid/v1.0/System.dll \
    --ref=/Library/Frameworks/Mono.framework/External/xbuild-frameworks/MonoAndroid/v1.0/System.Net.Http.dll \
    --ref=/Library/Frameworks/Mono.framework/External/xbuild-frameworks/MonoAndroid/v1.0/Facades/System.Runtime.dll \
    --ref=/Library/Frameworks/Mono.framework/External/xbuild-frameworks/MonoAndroid/v1.0/System.Xml.dll \
    --api-level=28 \
    --global \
    --public

The text was updated successfully, but these errors were encountered:

mattleibow · 2019-08-13T19:34:52Z

More info can be found at: #464

Context: dotnet#466 While investigating issue dotnet#466, I ran across an "oddity" within the type `kotlin/collections/AbstractCollection\$toString\$1.class`: <api api-source="class-parse"> <package name="kotlin.collections" jni-name="kotlin/collections"> <class name="AbstractCollection.toString.1" jni-signature="Lkotlin/collections/AbstractCollection$toString$1;"> <implements name="kotlin.jvm.functions.Function1" name-generic-aware="kotlin.jvm.functions.Function1<E, java.lang.CharSequence>" jni-type="Lkotlin/jvm/functions/Function1<TE;Ljava/lang/CharSequence;>;" /> <method name="invoke" jni-signature="(Ljava/lang/Object;)Ljava/lang/CharSequence;"> <parameter name="it" type="E" jni-type="TE;" /> </method> </class> </package> </api> (Abbreviated output). What's *bizarre* is that the `invoke()` method references the type `E`, but there is no declaration of what `E` *is*. *Normally*, there'd be a `<typeParameters/>` element, which is provided by the `Signature` annotation. Note that the above does *not* contain `<typeParameters/>`! On the vague thought that maybe `RuntimeVisibleAnnotations` were being used to store generic type information, add support to `class-parse` to extract `RuntimeVisibleAnnotations` and `RuntimeInvisibleAnnotations` annotation blobs.

jonpryor · 2019-08-13T21:14:22Z

I don't know what's going on. Time for some musings to collect later...

I thought that RuntimeVisibleAnnotations might be involved, so I added support for parsing RuntimeVisibleAnnotations and RuntimeInvisibleAnnotations blobs to class-parse: #467.

This might be overthinking it, though the commit message in #467 points out an important thought:

How/Why does kotlin/collections/AbstractCollection$toString$1''s invoke() method mention the type E, when class-parse-generated XML output doesn't mention E within a <typeParameters/> block?

<typeParameters/> in turn comes from the Signature annotation, which I should have looked at earlier. For comparison purposes, consider android.app.FragmentHostCallback. If we run class-parse --dump android/app/FragmentHostCallback.class, we get:

Attributes Count: 5
        0: Signature(<E:Ljava/lang/Object;>Landroid/app/FragmentContainer;)

The Signature annotation blob has a grammar for ClassSignature, which has FormalTypeParameters at the start of the string, between <...> brackets. This is what we see with android.app.FragmentHostCallback: a FormalTypeParameters specification of <E:Ljava/lang/Object;>, introducing the type parameter E which is constrained to java.lang.Object, followed by the signature of the super (base) class, android.app.FragmentContainer.

Now, what does kotlin/collections/AbstractCollection\$toString\$1.class have for it's Signature blob?

Attributes Count: 5
        0: InnerClasses(Count=1, InnerClass(InnerClass='kotlin/collections/AbstractCollection$toString$1', OuterClass='', InnerName='', InnerClassAccessFlags=Static, Final))
        1: Unknown[EnclosingMethod](4)
        2: Signature(Lkotlin/jvm/internal/Lambda;Lkotlin/jvm/functions/Function1<TE;Ljava/lang/CharSequence;>;)
        3: Unknown[SourceFile](2)
        4: RuntimeVisibleAnnotations(Annotation('Lkotlin/Metadata;', {mv: Array(Integer(1), Integer(1), Integer(15)), bv: Array(Integer(1), Integer(0), Integer(3)), k: Integer(3), d1: Array(Utf8("...")), d2: Array(Utf8("<anonymous>"), Utf8(""), Utf8("E"), Utf8("it"), Utf8("invoke"), Utf8("(Ljava/lang/Object;)Ljava/lang/CharSequence;"))}))

The Signature blob does not specify any type parameter information!

This leaves the question unanswered: where is E coming from? (I do see E in the RuntimeVisibleAnnotations for d2, but I don't want to begin to figure out what d1 and d2 are supposed to mean...)

Could the EnclosingMethod annotation be relevant?

Context: dotnet#466 While looking at `class-parse --dump` output for various Kotlin-compiled `.class` files, providing support for the `EnclosingMethod` and `SourceFile` annotation blobs look to be useful. Add support for parsing the `EnclosingMethod` and `SourceFile` annotations.

jonpryor · 2019-08-14T01:34:04Z

Thus, it appears to be due to attempted processing of generic types and their constraints.

It would be useful to know which exact type & method causes things to break. @jpobst mentioned a signature of ([Ljava/lang/Object;Lkotlin/jvm/functions/Function2;)Ljava/lang/Object; on kotlin/collections/ArraysKt___ArraysKt.class, but those methods look reasonable:

      <method
        abstract="false"
        deprecated="not deprecated"
        final="true"
        name="reduce"
        native="false"
        return="S"
        jni-return="TS;"
        static="true"
        synchronized="false"
        visibility="public"
        bridge="false"
        synthetic="false"
        jni-signature="([Ljava/lang/Object;Lkotlin/jvm/functions/Function2;)Ljava/lang/Object;">
        <typeParameters>
          <typeParameter
            name="S"
            jni-classBound="Ljava/lang/Object;"
            classBound="java.lang.Object"
            interfaceBounds=""
            jni-interfaceBounds="" />
          <typeParameter
            name="T"
            jni-classBound=""
            classBound=""
            interfaceBounds="S"
            jni-interfaceBounds="TS;" />
        </typeParameters>
        <parameter
          name="$this$reduce"
          type="T[]"
          jni-type="[TT;" />
        <parameter
          name="operation"
          type="kotlin.jvm.functions.Function2&lt;? super S, ? super T, ? extends S&gt;"
          jni-type="Lkotlin/jvm/functions/Function2&lt;-TS;-TT;+TS;&gt;;" />
      </method>
      <method
        abstract="false"
        deprecated="not deprecated"
        final="true"
        name="reduceRight"
        native="false"
        return="S"
        jni-return="TS;"
        static="true"
        synchronized="false"
        visibility="public"
        bridge="false"
        synthetic="false"
        jni-signature="([Ljava/lang/Object;Lkotlin/jvm/functions/Function2;)Ljava/lang/Object;">
        <typeParameters>
          <typeParameter
            name="S"
            jni-classBound="Ljava/lang/Object;"
            classBound="java.lang.Object"
            interfaceBounds=""
            jni-interfaceBounds="" />
          <typeParameter
            name="T"
            jni-classBound=""
            classBound=""
            interfaceBounds="S"
            jni-interfaceBounds="TS;" />
        </typeParameters>
        <parameter
          name="$this$reduceRight"
          type="T[]"
          jni-type="[TT;" />
        <parameter
          name="operation"
          type="kotlin.jvm.functions.Function2&lt;? super T, ? super S, ? extends S&gt;"
          jni-type="Lkotlin/jvm/functions/Function2&lt;-TT;-TS;+TS;&gt;;" />
      </method>

...although...perhaps it's having T have an interface bound on S which is causing the problem?

        <typeParameters>
          <typeParameter
            name="S"
            jni-classBound="Ljava/lang/Object;"
            classBound="java.lang.Object"
            interfaceBounds=""
            jni-interfaceBounds="" />
          <typeParameter
            name="T"
            jni-classBound=""
            classBound=""
            interfaceBounds="S"
            jni-interfaceBounds="TS;" />
        </typeParameters>

Which is, I believe, equivalent to:

<S, T implements S>
public static S reduceRight(T[] $this$reduceRight, Function2<? super T, ? super S, ? extends S> operation);

Context: dotnet#466 While looking at `class-parse --dump` output for various Kotlin-compiled `.class` files, providing support for the `EnclosingMethod` and `SourceFile` annotation blobs looks to be useful. Add support for parsing the `EnclosingMethod` and `SourceFile` annotations. Update the generated XML so that the `SourceFile` annotation is avaialble from the `//class/@source-file-name` or `//interface/@source-file-name` attributes. Update the generated XML so that the `EnclosingMethod` annotation is available from these new attributes on `//class` or `//interface`: * `enclosing-method-jni-type`: The JNI signature of the type declaring the enclosing method. * `enclosing-method-name`: The name of the enclosing method. * `enclosing-method-signature`: The JNI signature of the enclosing method. For example, if `TestType.action()` had an anonymous inner class: class TestType { public void action (Object value) { Runnable r = new Runnable () { public void run() { System.out.println ("foo"); } }; } } This could result in the creation of a new `TestType$1` class for the anonymous inner class, with the resulting `enclosing-*` attributes: enclosing-method-jni-type="Lcom/xamarin/JavaType;" enclosing-method-name="action" enclosing-method-signature="(Ljava/lang/Object;)V"

Context: #466 While looking at `class-parse --dump` output for various Kotlin-compiled `.class` files, providing support for the `EnclosingMethod` and `SourceFile` annotation blobs looks to be useful. Add support for parsing the `EnclosingMethod` and `SourceFile` annotations. Update the generated XML so that the `SourceFile` annotation is avaialble from the `//class/@source-file-name` or `//interface/@source-file-name` attributes. Update the generated XML so that the `EnclosingMethod` annotation is available from these new attributes on `//class` or `//interface`: * `enclosing-method-jni-type`: The JNI signature of the type declaring the enclosing method. * `enclosing-method-name`: The name of the enclosing method. * `enclosing-method-signature`: The JNI signature of the enclosing method. For example, if `TestType.action()` had an anonymous inner class: class TestType { public void action (Object value) { Runnable r = new Runnable () { public void run() { System.out.println ("foo"); } }; } } This could result in the creation of a new `TestType$1` class for the anonymous inner class, with the resulting `enclosing-*` attributes: enclosing-method-jni-type="Lcom/xamarin/JavaType;" enclosing-method-name="action" enclosing-method-signature="(Ljava/lang/Object;)V"

jonpryor · 2019-08-19T21:00:39Z

Revisiting #466 (comment)

How/Why does kotlin/collections/AbstractCollection$toString$1''s invoke() method mention the type E, when class-parse-generated XML output doesn't mention E within a <typeParameters/> block?

With PR #469 we now parse the EnclosingMethod attribute and export it from class-parse-generated XML. From it we see:

$ mono class-parse.exe kotlin/collections/AbstractCollection\$toString\$1.class
...
    <class
      abstract="false"
      deprecated="not deprecated"
      enclosing-method-jni-type="Lkotlin/collections/AbstractCollection;"
      enclosing-method-name="toString"
      enclosing-method-signature="()Ljava/lang/String;"
      jni-extends="Lkotlin/jvm/internal/Lambda;"
      extends="kotlin.jvm.internal.Lambda"
      extends-generic-aware="kotlin.jvm.internal.Lambda"
      final="true"
      name="AbstractCollection.toString.1"
      jni-signature="Lkotlin/collections/AbstractCollection$toString$1;"
      source-file-name="AbstractCollection.kt"
      static="true"
      visibility="">
...

The enclosing type is Lkotlin/collections/AbstractCollection;.

If we turn our attention to AbstractCollection, we see:

$ mono class-parse.exe kotlin/collections/AbstractCollection.class
...
    <class
      abstract="true"
      deprecated="not deprecated"
      jni-extends="Ljava/lang/Object;"
      extends="java.lang.Object"
      extends-generic-aware="java.lang.Object"
      final="false"
      name="AbstractCollection"
      jni-signature="Lkotlin/collections/AbstractCollection;"
      source-file-name="AbstractCollection.kt"
      static="false"
      visibility="public">
      <typeParameters>
        <typeParameter
          name="E"
          jni-classBound="Ljava/lang/Object;"
          classBound="java.lang.Object"
          interfaceBounds=""
          jni-interfaceBounds="" />

i.e. AbstractCollection does contain a declaration for the type parameter E, which is used/referenced by AbstractCollection$toString$1.

Thus, we can infer a possible fix: if a type has an enclosing-method-jni-type attribute, "lookup" the specified type (and optional method) to find any type parameters specified in the enclosing scope, and if necessary copy those <typeParameters/> into the resulting XML.

Alternatively, instead of having class-parse do that "copy <typeParameters/> to the referencing scope" logic, have generator perform that copy.

My concern over doing the copy at all is it alters the semantics of class-parse lib.jar vs. class-parse type.class: the former can perform "enclosing type" lookups; the latter can't. It may thus be preferable to have generator do the lookups, as it should have all necessary context.

…495) Context: #466 Context: #467 When attempting to bind [`kotlin-stdlib-1.3.41.jar`][0], `generator` crashes with the following exception: System.NullReferenceException: Object reference not set to an instance of an object at Xamarin.Android.Tools.ApiXmlAdjuster.JavaApiXmlGeneratorExtensions.Save (Xamarin.Android.Tools.ApiXmlAdjuster.JavaTypeParameters typeParameters, System.Xml.XmlWriter writer, System.String indent) at Xamarin.Android.Tools.ApiXmlAdjuster.JavaApiXmlGeneratorExtensions.SaveCommon (Xamarin.Android.Tools.ApiXmlAdjuster.JavaMember m, System.Xml.XmlWriter writer, System.String elementName, System.String abs, System.String native, System.String ret, System.String sync, System.String transient, System.String type, System.String typeGeneric, System.String value, System.String volat, Xamarin.Android.Tools.ApiXmlAdjuster.JavaTypeParameters typeParameters, System.Collections.Generic.IEnumerable`1[T] parameters, System.Collections.Generic.IEnumerable`1[T] exceptions, System.Nullable`1[T] extBridge, System.String jniReturn, System.Nullable`1[T] extSynthetic) at Xamarin.An/droid.Tools.ApiXmlAdjuster.JavaApiXmlGeneratorExtensions.Save (Xamarin.Android.Tools.ApiXmlAdjuster.JavaMethod method, System.Xml.XmlWriter writer) at Xamarin.Android.Tools.ApiXmlAdjuster.JavaApiXmlGeneratorExtensions.SaveTypeCommon (Xamarin.Android.Tools.ApiXmlAdjuster.JavaType cls, System.Xml.XmlWriter writer, System.String elementName, System.String abs, System.String ext, System.String extgen, System.String jniExt) at Xamarin.Android.Tools.ApiXmlAdjuster.JavaApiXmlGeneratorExtensions.Save (Xamarin.Android.Tools.ApiXmlAdjuster.JavaClass cls, System.Xml.XmlWriter writer) at Xamarin.Android.Tools.ApiXmlAdjuster.JavaApiXmlGeneratorExtensions.Save (Xamarin.Android.Tools.ApiXmlAdjuster.JavaApi api, System.Xml.XmlWriter writer) at Xamarin.Android.Tools.ApiXmlAdjuster.JavaApiXmlGeneratorExtensions.Save (Xamarin.Android.Tools.ApiXmlAdjuster.JavaApi api, System.String xmlfile) at Xamarin.Android.Tools.ApiXmlAdjuster.Adjuster.Process (System.String inputXmlFile, MonoDroid.Generation.CodeGenerationOptions opt, MonoDroid.Generation.GenBase[] gens, System.String outputXmlFile, System.Int32 reportVerbosity) at Xamarin.Android.Binder.CodeGenerator.Run (Xamarin.Android.Binder.CodeGeneratorOptions options, Java.Interop.Tools.Cecil.DirectoryAssemblyResolver resolver) at Xamarin.Android.Binder.CodeGenerator.Run (Xamarin.Android.Binder.CodeGeneratorOptions options) at Xamarin.Android.Binder.CodeGenerator.Main (System.String[] args) We are still investigating *why* this happens and how it should be appropriately fixed. In the meantime, to "unblock" developers attempting to bind binding Kotlin libraries in Visual Studio 16.4 Preview 2, simply prevent the `NullReferenceException` from being generated generated. This at least lets people get past the crash so they can apply fixups in `Metadata.xml` to continue. (This `NullReferenceException` happens before the `metadata` stage; there's nothing a user can currently do.) I will be making our Kotlin bindings more "correct" in future sprints. [0]: https://search.maven.org/remotecontent?filepath=org/jetbrains/kotlin/kotlin-stdlib/1.3.41/kotlin-stdlib-1.3.41.jar

Changes: dotnet/java-interop@cd91e48...2716edf Context: dotnet/java-interop#466 Updates `generator` to avoid a `NullReferenceException` commonly encountered when binding `kotlin-stdlib-1.3.41.jar`.

Changes: dotnet/java-interop@b98f423...35b08ba Context: dotnet/java-interop#466 Updates `generator` to avoid a `NullReferenceException` commonly encountered when binding `kotlin-stdlib-1.3.41.jar`.

Context: #466 While investigating issue #466, I ran across an "oddity" within the type `kotlin/collections/AbstractCollection\$toString\$1.class`: <api api-source="class-parse"> <package name="kotlin.collections" jni-name="kotlin/collections"> <class name="AbstractCollection.toString.1" jni-signature="Lkotlin/collections/AbstractCollection$toString$1;"> <implements name="kotlin.jvm.functions.Function1" name-generic-aware="kotlin.jvm.functions.Function1<E, java.lang.CharSequence>" jni-type="Lkotlin/jvm/functions/Function1<TE;Ljava/lang/CharSequence;>;" /> <method name="invoke" jni-signature="(Ljava/lang/Object;)Ljava/lang/CharSequence;"> <parameter name="it" type="E" jni-type="TE;" /> </method> </class> </package> </api> (Abbreviated output). What's *bizarre* is that the `invoke()` method references the type `E`, but there is no declaration of what `E` *is*. *Normally*, there'd be a `<typeParameters/>` element, which is provided by the `Signature` annotation. Note that the above does *not* contain `<typeParameters/>`! On the vague thought that maybe `RuntimeVisibleAnnotations` were being used to store generic type information, add support to `class-parse` to extract `RuntimeVisibleAnnotations` and `RuntimeInvisibleAnnotations` annotation blobs.

#502) Context: #466 Context: #467 Context: #499 Add support for parsing the [`RuntimeVisibleAnnotations`][0] and [`RuntimeInvisibleAnnotations`][1] attributes in Java bytecode. These attributes store Java programming language [annotations][2] placed on language constructs such as types, fields, and methods, and are analogous with C# custom attributes. Kotlin stores various bits of information in these attributes, and it will be necessary to parse these attributes to read that information; see also PR #499. [0]: https://docs.oracle.com/javase/specs/jvms/se7/html/jvms-4.html#jvms-4.7.16 [1]: https://docs.oracle.com/javase/specs/jvms/se7/html/jvms-4.html#jvms-4.7.17 [2]: https://docs.oracle.com/javase/specs/jls/se7/html/jls-9.html#jls-9.7

jpobst · 2019-10-31T15:49:37Z

Fixed in #495.

brendanzagaeski · 2019-12-04T02:17:40Z

Release status update

A new Release version of Xamarin.Android has now been published on Windows that includes the fix for this item. The fix is not yet published in a Release version on macOS. I will update this item again when a Release version is available on macOS that includes the fix.

Fix included in Xamarin.Android 10.1.0.30.

Fix included on Windows in Visual Studio 2019 version 16.4. To get the new version that includes the fix, check for the latest updates or install the latest version from https://visualstudio.microsoft.com/downloads/.

(Fix also included on macOS in Visual Studio 2019 for Mac version 8.4 Preview 2.1 and higher. To try the Preview version that includes the fix, check for the latest updates on the Preview updater channel.)

brendanzagaeski · 2020-01-08T21:51:02Z

Release status update

A new Release version has now been published on macOS that includes the fix for this item.

Fix included in Xamarin.Android 10.1.1.0.

Fix included on macOS in Visual Studio 2019 for Mac version 8.4. To get the new version that includes the fix, check for the latest updates on the Stable updater channel.

(Fix also included on Windows in Visual Studio 2019 version 16.4 and higher. To get the new version that includes the fix, check for the latest updates or install the latest version from https://visualstudio.microsoft.com/downloads/.)

jonpryor mentioned this issue Aug 13, 2019

[Xamarin.Android.Tools.Bytecode] Support RuntimeVisibleAnnotations #467

Closed

jonpryor mentioned this issue Aug 14, 2019

[Xamarin.Android.Tools.Bytecode] Add EnclosingMethod, SourceFile #469

Merged

This was referenced Aug 14, 2019

Add tooling to help with binding Kotlin libraries xamarin/XamarinComponents#642

Merged

Update Kotlin to v1.2.71 xamarin/XamarinComponents#643

Merged

jonpryor changed the title ~~Binding Kotlin itself crashes the class-parse generator~~ Binding Kotlin itself crashes generator --only-xml-adjuster Aug 19, 2019

jpobst mentioned this issue Sep 30, 2019

[ApiXmlAdjuster] Guard against NRE when writing generic constraints. #495

Merged

jonpryor mentioned this issue Oct 1, 2019

Bump to xamarin/Java.Interop/master@2716edfd dotnet/android#3709

Merged

mattleibow mentioned this issue Oct 3, 2019

Renaming methods in an abstract class does not rename all the components #496

Closed

jpobst mentioned this issue Oct 7, 2019

Generator doesn't generate method consistently #501

Closed

jpobst mentioned this issue Oct 8, 2019

[Xamarin.Android.Tools.Bytecode] Support Runtime[In]VisibleAnnotations #502

Merged

jpobst added this to the d16-4 milestone Oct 31, 2019

jpobst closed this as completed Oct 31, 2019

brendanzagaeski modified the milestones: d16-4, Published: 10.1 (d16-4) Dec 4, 2019

github-actions bot locked and limited conversation to collaborators Apr 13, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Binding Kotlin itself crashes `generator --only-xml-adjuster` #466

Binding Kotlin itself crashes `generator --only-xml-adjuster` #466

mattleibow commented Aug 13, 2019

mattleibow commented Aug 13, 2019

jonpryor commented Aug 13, 2019

jonpryor commented Aug 14, 2019

jonpryor commented Aug 19, 2019

jpobst commented Oct 31, 2019

brendanzagaeski commented Dec 4, 2019

brendanzagaeski commented Jan 8, 2020

Binding Kotlin itself crashes generator --only-xml-adjuster #466

Binding Kotlin itself crashes generator --only-xml-adjuster #466

Comments

mattleibow commented Aug 13, 2019

mattleibow commented Aug 13, 2019

jonpryor commented Aug 13, 2019

jonpryor commented Aug 14, 2019

jonpryor commented Aug 19, 2019

jpobst commented Oct 31, 2019

brendanzagaeski commented Dec 4, 2019

brendanzagaeski commented Jan 8, 2020

Binding Kotlin itself crashes `generator --only-xml-adjuster` #466

Binding Kotlin itself crashes `generator --only-xml-adjuster` #466