Skip to content

[XA.Tools.Bytecode] Add Kotlin parsing and honor Kotlin class visibility #499

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Oct 15, 2019

Conversation

jpobst
Copy link
Contributor

@jpobst jpobst commented Oct 4, 2019

Kotlin Metadata

The bulk of this PR is adding plumbing support for the metadata that Kotlin adds to compiled classes.

This metadata is added to a RuntimeVisibleAnnotation on the class with a key of Lkotlin/Metadata. This Annotation can contain the following key/values:

  • k - The kind of metadata annotation (required)
  • mv - The version of metadata (required)
  • bv - The version of the byte code (required)
  • d1 - The encoded metadata (optional)
  • d2 - Additional strings (optional)

As of this PR, we only support Kind == Class.

The Class metadata is encoded as an array of strings. First we have to decode the d1 data into bytes using a variety of potential encodings. Once we have bytes, they are formatted as:

  • First byte - Length of the string table
    • optional additional bytes to encode length of string table
  • Protobuf encoded string data table is next length bytes
  • Remaining bytes are Protobuf encoded data

The string data table provides an indexed lookup table used by the Protobuf metadata to retrieve strings from one of:

  • Directly encoded in the string table
  • The string array contained in the d2 field
  • A predefined hardcoded string

The Protobuf encoded data is relatively straightforward. All fields that would be strings are instead indexes into the string data table. It provides metadata about the class itself and every field/method/property/parameter/etc for the class.

Metadata Consumption

With this PR we are starting small to consume this metadata:

  • Respect the Kotlin class visibility, so we mark a class as private if it is Kotlin-Private or Kotlin-Internal.
  • Kotlin has a lot of interesting non-Class constructs it adds .class files for. Since our code doesn't understand them yet they do not generate usable bindings. Until we have completed our support for these constructs this PR marks them as private so we do not attempt to bind them. If a user really wants to bind these constructs they can mark them as public in metadata when adding the other metadata needed to make them bindable.

Protobuf Generated File

https://github.com/xamarin/java.interop/blob/b6e01bc2bea72f5524f89e0f1f6fede6c13183b2/src/Xamarin.Android.Tools.Bytecode/Kotlin/KotlinProtobufDefinition.cs is generated using this tool from Protobuf.Net:

https://protogen.marcgravell.com/

Paste in this file:
https://github.com/JetBrains/kotlin/blob/master/core/metadata.jvm/src/jvm_metadata.proto

The tool does not support #import so you'll need to replace those lines with the contents of the files:

Generate with the default options to create the .cs file.

@jpobst jpobst force-pushed the kotlin-metadata branch 2 times, most recently from 213ad21 to 7ca54f5 Compare October 4, 2019 19:27
@jpobst jpobst marked this pull request as ready for review October 7, 2019 15:28
@jonpryor
Copy link
Member

jonpryor commented Oct 7, 2019

See also: 39a3b87

This PR will also need to update src/Xamarin.Android.Tools.ApiXmlAdjuster/JavaApiXmlLoaderExtensions.cs so that Xamarin.Android.Tools.ApiXmlAdjuster doesn't throw an exception when reading an XML file containing the new annotation XML elements.

...once the new XML elements are emitted...

jonpryor pushed a commit that referenced this pull request Oct 9, 2019
#502)

Context: #466
Context: #467
Context: #499

Add support for parsing the [`RuntimeVisibleAnnotations`][0] and
[`RuntimeInvisibleAnnotations`][1] attributes in Java bytecode.
These attributes store Java programming language [annotations][2]
placed on language constructs such as types, fields, and methods, and
are analogous with C# custom attributes.

Kotlin stores various bits of information in these attributes, and it
will be necessary to parse these attributes to read that information;
see also PR #499.

[0]: https://docs.oracle.com/javase/specs/jvms/se7/html/jvms-4.html#jvms-4.7.16
[1]: https://docs.oracle.com/javase/specs/jvms/se7/html/jvms-4.html#jvms-4.7.17
[2]: https://docs.oracle.com/javase/specs/jls/se7/html/jls-9.html#jls-9.7

namespace Xamarin.Android.Tools.Bytecode.Kotlin
{
public static class KotlinFixups
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this need to be public?

foreach (var c in classes) {
// See if this is a Kotlin class
var attr = c.Attributes.OfType<RuntimeVisibleAnnotationsAttribute> ().FirstOrDefault ();
var kotlin = attr?.Annotations.FirstOrDefault (a => a.Type == "Lkotlin/Metadata;");
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should use .SingleOrDefault().

@jonpryor
Copy link
Member

Summary: [Xamarin.Android.Tools.Bytecode] Honor Kotlin class visibility (#499)

Body:

The [Kotlin language][0] stores Kotlin-specific metadata within a
[`RuntimeVisibleAnnotation` bytecode attribute blob][1] with a type
name value of `Lkotlin/Metadata;`.  The annotation can contain the
following keys:

  * `k`: The kind of metadata annotation (required)
  * `mv`: The version of metadata (required)
  * `bv`: The version of the byte code (required)
  * `d1`: The encoded metadata (optional)
  * `d2`: Additional strings (optional)

Add support for a kind value of `Class`.

The `Class` metadata is encoded as an array of strings.

First we have to decode the `d1` data into bytes using a variety of
potential encodings.  Once we have bytes, they are formatted as:

  * First byte - Length of the string table
    * optional additional bytes to encode length of string table
  * Protobuf encoded string data table is next `length` bytes
  * Remaining bytes are Protobuf encoded data

The string data table provides an indexed lookup table used by the
Protobuf metadata to retrieve strings from one of:

  * Directly encoded in the string table
  * The string array contained in the `d2` field
  * A predefined hardcoded string

The Protobuf encoded data is relatively straightforward.  All fields
that would be strings are instead indexes into the string data table.
It provides metadata about the class itself and every
field/method/property/parameter/etc. for the class.

Once the Protobuf data is parsed, we can use this information to
influence the normal XML declarations that `class-parse` produces:

  * Respect the Kotlin class visibility: if the Kotlin metadata
    specifies visibility as `Kotlin-Private` or `Kotlin-Internal`,
    set e.g. `//class/@visibility` to `private`, which prevents the
    type from being emitted into the XML description.

  * Filter out non-bindable constructs: Kotlin has a lot of
    interesting non-Class constructs it uses `.class` files for.
    Since our code doesn't understand them yet they do not generate
    usable bindings for these constructs.  Until we have completed
    our support for these constructs we mark them as `private` so we
    do not attempt to bind them.

~~ Protobuf Generation ~~

`src/Xamarin.Android.Tools.Bytecode/Kotlin/KotlinProtobufDefinition.cs`
is generated by using [`protogen`][2], by providing Kotlin's
[protobuf schema][3] as input.

Note that `protogen` doesn't support `import`, so the input text
provided to `protogen` will need to replace:

	import "core/metadata/src/ext_options.proto";

with the contents of [`core/metadata/src/ext_options.proto`][4], and
replace

	import "core/metadata/src/metadata.proto";

with the contents of [`core/metadata/src/metadata.proto`][5].

The default `protogen` options were used to generate
`KotlinProtobufDefinition.cs`.

[0]: https://kotlinlang.org
[1]: https://docs.oracle.com/javase/specs/jvms/se7/html/jvms-4.html#jvms-4.7.16
[2]: https://protogen.marcgravell.com
[3]: https://github.com/JetBrains/kotlin/blob/master/core/metadata.jvm/src/jvm_metadata.proto
[4]: https://github.com/JetBrains/kotlin/blob/master/core/metadata/src/ext_options.proto
[5]: https://github.com/JetBrains/kotlin/blob/master/core/metadata/src/metadata.proto

@jonpryor jonpryor merged commit 73096d9 into master Oct 15, 2019
@jonpryor jonpryor deleted the kotlin-metadata branch October 15, 2019 19:55
@github-actions github-actions bot locked and limited conversation to collaborators Apr 13, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants