From abc5d766b963c102b5b72ed4eb03cca0f1270f48 Mon Sep 17 00:00:00 2001 From: Peter Hofer Date: Mon, 21 Aug 2023 15:31:08 +0200 Subject: [PATCH 1/5] Changelog entry for GR-29688 method handle changes. --- substratevm/CHANGELOG.md | 1 + 1 file changed, 1 insertion(+) diff --git a/substratevm/CHANGELOG.md b/substratevm/CHANGELOG.md index 850f4dc9fe8c..24ff112dcb19 100644 --- a/substratevm/CHANGELOG.md +++ b/substratevm/CHANGELOG.md @@ -20,6 +20,7 @@ This changelog summarizes major changes to GraalVM Native Image. * (GR-43920) Add support for executing native image bundles as jar files with extra options `--with-native-image-agent` and `--container`. * (GR-43920) Add `,container[=]`, `,dockerfile=` and `,dry-run` options to `--bundle-create`and `--bundle-apply`. * (GR-46420) Switch to directly using cgroup support from the JDK. +* (GR-29688) More sophisticated intrinsification and inlining of method handle usages, both explicit and implicit (lambdas, string concatenation and record classes), and various fixes for non-intrinsified usages. ## GraalVM for JDK 17 and GraalVM for JDK 20 (Internal Version 23.0.0) * (GR-40187) Report invalid use of SVM specific classes on image class- or module-path as error. As a temporary workaround, `-H:+AllowDeprecatedBuilderClassesOnImageClasspath` allows turning the error into a warning. From 7b778791bb708ac59f0f11036a4e072141f1b219 Mon Sep 17 00:00:00 2001 From: Peter Hofer Date: Mon, 21 Aug 2023 15:10:15 +0200 Subject: [PATCH 2/5] Elaborate on foreign function downcalls change. --- substratevm/CHANGELOG.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/substratevm/CHANGELOG.md b/substratevm/CHANGELOG.md index 24ff112dcb19..b5e0b19aa474 100644 --- a/substratevm/CHANGELOG.md +++ b/substratevm/CHANGELOG.md @@ -13,7 +13,7 @@ This changelog summarizes major changes to GraalVM Native Image. * (GR-38994) Together with Red Hat, we added support for `-XX:+HeapDumpOnOutOfMemoryError`. * (GR-47365) Throw `MissingReflectionRegistrationError` when attempting to create a proxy class without having it registered at build-time, instead of a `VMError`. * (GR-46064) Add option `-H:±IndirectBranchTargetMarker` to mark indirect branch targets on AMD64 with an endbranch instruction. This is a prerequisite for future Intel CET support. -* (GR-46740) Add support for foreign downcalls (part of "Project Panama") on the AMD64 platform. +* (GR-46740) Preview of [Foreign Function & Memory API downcalls](https://github.com/oracle/graal/blob/master/docs/reference-manual/native-image/ForeignInterface.md) (part of "Project Panama", [JEP 442](https://openjdk.org/jeps/442)) on AMD64. Must be enabled with `--enable-preview`. * (GR-27034) Add `-H:ImageBuildID` option to generate Image Build ID, which is a 128-bit UUID string generated randomly, once per bundle or digest of input args when bundles are not used. * (GR-47647) Add `-H:±UnlockExperimentalVMOptions` for unlocking access to experimental options similar to HotSpot's `-XX:UnlockExperimentalVMOptions`. Explicit unlocking will be required in a future release, which can be tested with the env setting `NATIVE_IMAGE_EXPERIMENTAL_OPTIONS_ARE_FATAL=true`. For more details, see [issue #7105](https://github.com/oracle/graal/issues/7105). * (GR-47647) Add `--color[=WHEN]` option to color the output WHEN ('always', 'never', or 'auto'). This API option supersedes the experimental option `-H:+BuildOutputColorful`. From 0e610ff1ed15bb2ec75fe13277ee2c749d87335a Mon Sep 17 00:00:00 2001 From: Peter Hofer Date: Mon, 21 Aug 2023 18:08:10 +0200 Subject: [PATCH 3/5] Revise Foreign Function & Memory API documentation. --- .../native-image/ForeignInterface.md | 37 +++++++++++-------- 1 file changed, 21 insertions(+), 16 deletions(-) diff --git a/docs/reference-manual/native-image/ForeignInterface.md b/docs/reference-manual/native-image/ForeignInterface.md index af3c3c448072..68cebe8a2e9d 100644 --- a/docs/reference-manual/native-image/ForeignInterface.md +++ b/docs/reference-manual/native-image/ForeignInterface.md @@ -5,31 +5,34 @@ link_title: Foreign Interface permalink: /reference-manual/native-image/dynamic-features/foreign-interface/ --- -# Foreign Interface in Native Image +# Foreign Function & Memory API in Native Image -The Foreign Interface is a native API that enables Java code to interact with native code and vice versa. -It is currently a preview API of the Java platform and must be enabled with `--enable-preview`. -This page gives an overview of its support in Native Image. +The Foreign Function & Memory (FFM) API is a native interface that enables Java code to interact with native code and vice versa. +As of [JEP 442](https://openjdk.org/jeps/442){:target="_blank"}, it is a preview API of the Java platform and must be enabled with `--enable-preview`. +Modules that are permitted to perform "restricted" native operations (including creating handles for calls to or from native code) must be specified using `--enable-native-access=`. +This page gives an overview of support for the FFM API in Native Image. ## Foreign memory -Shared arenas are not supported. +Foreign memory functionality is generally supported. Shared arenas are currently not supported. ## Foreign functions -The Foreign Functions Interface (FFI) allows Java code to call native functions, and conversely allows native code to invoke Java method handles. -These two kind of calls are referred to as "downcalls" and "upcalls" respectively and are collectively referred to as "foreign calls". +The FFM API enables Java code to call _down_ to native functions, and conversely allows native code to call _up_ to invoke Java code via method handles. +These two kinds of calls are referred to as "downcalls" and "upcalls" respectively and are collectively referred to as "foreign calls". -This feature is currently only supported on the AMD64 platform. +Currently, only downcalls are supported, and only on the AMD64 architecture. ### Looking up native functions -FFI provides the `SymbolLookup` interface which allows to search native libraries for functions by name. -`loaderLookup` is currently the only supported `SymbolLookup`. +The FFM API provides the `SymbolLookup` interface to find functions in native libraries by name. +`SymbolLookup.loaderLookup()` is currently the only supported kind of `SymbolLookup`. ### Registering foreign calls -In order to perform a call to native, some glue code is required and thus must be generated at build time. -Therefore, a list of the types of downcall which will be performed must be provided to the `native-image` builder. +In order to perform calls to native code at runtime, supporting code must be generated at image build time. +Therefore, the `native-image` tool must be provided with descriptors that characterize functions to which downcalls may be performed at runtime. -This list can be specified using a custom `Feature`. For example: +These descriptors can be registered using a custom `Feature`, for example: ```java +import static java.lang.foreign.ValueLayout.*; + class ForeignRegistrationFeature implements Feature { public void duringSetup(DuringSetupAccess access) { RuntimeForeignAccess.registerForDowncall(FunctionDescriptor.ofVoid()); @@ -40,7 +43,9 @@ class ForeignRegistrationFeature implements Feature { } } ``` -To activate the custom feature `--features=` needs to be passed to native-image. -[Native Image Build Configuration](BuildConfiguration.md#embed-a-configuration-file) explains how this can be automated with a `native-image.properties` file in `META-INF/native-image`. +To activate the custom feature, `--features=com.example.ForeignRegistrationFeature` (the fully-qualified name of the feature class) needs to be passed to `native-image`. +It is recommended to do so [with a _native-image.properties_ file](BuildConfiguration.md#embed-a-configuration-file). + +### Upcalls -Upcalls are currently not supported. \ No newline at end of file +Upcalls are not yet supported. From 610bffebc148b74cfa5a1b3ee58ce781f5b6bcbe Mon Sep 17 00:00:00 2001 From: Peter Hofer Date: Tue, 29 Aug 2023 18:45:29 +0200 Subject: [PATCH 4/5] If old method handle intrinsics are requested, disable method handle invoker renaming. --- .../hosted/methodhandles/MethodHandleFeature.java | 13 ++++++++++--- 1 file changed, 10 insertions(+), 3 deletions(-) diff --git a/substratevm/src/com.oracle.svm.hosted/src/com/oracle/svm/hosted/methodhandles/MethodHandleFeature.java b/substratevm/src/com.oracle.svm.hosted/src/com/oracle/svm/hosted/methodhandles/MethodHandleFeature.java index 97fecdb804f8..36b70f1220f0 100644 --- a/substratevm/src/com.oracle.svm.hosted/src/com/oracle/svm/hosted/methodhandles/MethodHandleFeature.java +++ b/substratevm/src/com.oracle.svm.hosted/src/com/oracle/svm/hosted/methodhandles/MethodHandleFeature.java @@ -42,6 +42,7 @@ import com.oracle.graal.pointsto.meta.AnalysisMetaAccess; import com.oracle.graal.pointsto.meta.AnalysisType; +import com.oracle.svm.core.SubstrateOptions; import com.oracle.svm.core.feature.AutomaticallyRegisteredFeature; import com.oracle.svm.core.feature.InternalFeature; import com.oracle.svm.core.fieldvaluetransformer.FieldValueTransformerWithAvailability; @@ -140,9 +141,15 @@ public void duringSetup(DuringSetupAccess access) { referencedKeySetAdd = ReflectionUtil.lookupMethod(concurrentWeakInternSetClass, "add", Object.class); } - var accessImpl = (DuringSetupAccessImpl) access; - substitutionProcessor = new MethodHandleInvokerRenamingSubstitutionProcessor(accessImpl.getBigBang()); - accessImpl.registerSubstitutionProcessor(substitutionProcessor); + if (!SubstrateOptions.UseOldMethodHandleIntrinsics.getValue()) { + /* + * Renaming is not crucial with old method handle intrinsics, so if those are requested + * explicitly, disable renaming to offer a fallback in case it causes problems. + */ + var accessImpl = (DuringSetupAccessImpl) access; + substitutionProcessor = new MethodHandleInvokerRenamingSubstitutionProcessor(accessImpl.getBigBang()); + accessImpl.registerSubstitutionProcessor(substitutionProcessor); + } } @Override From 35634fae3f3eb851ac9a8b72ae755889790b067a Mon Sep 17 00:00:00 2001 From: Peter Hofer Date: Wed, 30 Aug 2023 19:13:28 +0200 Subject: [PATCH 5/5] [GR-48285] More stable renaming for method handle invokers. --- ...eInvokerRenamingSubstitutionProcessor.java | 36 ++++++++++++++++--- 1 file changed, 32 insertions(+), 4 deletions(-) diff --git a/substratevm/src/com.oracle.svm.hosted/src/com/oracle/svm/hosted/methodhandles/MethodHandleInvokerRenamingSubstitutionProcessor.java b/substratevm/src/com.oracle.svm.hosted/src/com/oracle/svm/hosted/methodhandles/MethodHandleInvokerRenamingSubstitutionProcessor.java index ac025a32f451..2e1b4208c8c7 100644 --- a/substratevm/src/com.oracle.svm.hosted/src/com/oracle/svm/hosted/methodhandles/MethodHandleInvokerRenamingSubstitutionProcessor.java +++ b/substratevm/src/com.oracle.svm.hosted/src/com/oracle/svm/hosted/methodhandles/MethodHandleInvokerRenamingSubstitutionProcessor.java @@ -24,6 +24,7 @@ */ package com.oracle.svm.hosted.methodhandles; +import java.lang.reflect.Field; import java.lang.reflect.Method; import java.util.HashSet; import java.util.List; @@ -47,8 +48,11 @@ * the {@code LambdaForm} which they were compiled from. */ public class MethodHandleInvokerRenamingSubstitutionProcessor extends SubstitutionProcessor { - private static final Class LAMBDA_FORM_CLASS = ReflectionUtil.lookupClass(false, "java.lang.invoke.LambdaForm"); private static final Method CLASS_GET_CLASS_DATA_METHOD = ReflectionUtil.lookupMethod(Class.class, "getClassData"); + private static final Class LAMBDA_FORM_CLASS = ReflectionUtil.lookupClass(false, "java.lang.invoke.LambdaForm"); + private static final Field LAMBDA_FORM_CUSTOMIZED_FIELD = ReflectionUtil.lookupField(LAMBDA_FORM_CLASS, "customized"); + private static final Class DIRECT_METHOD_HANDLE_CLASS = ReflectionUtil.lookupClass(false, "java.lang.invoke.DirectMethodHandle"); + private static final Method DIRECT_METHOD_HANDLE_INTERNAL_MEMBER_NAME_METHOD = ReflectionUtil.lookupMethod(DIRECT_METHOD_HANDLE_CLASS, "internalMemberName"); /* * We currently only replace the invokers of direct method handles which have a simpler @@ -88,15 +92,39 @@ public ResolvedJavaType resolve(ResolvedJavaType type) { private ResolvedJavaType getSubstitution(ResolvedJavaType type) { return typeSubstitutions.computeIfAbsent(type, original -> { + Object lambdaForm; + Object customizedMemberName = null; try { Class clazz = OriginalClassProvider.getJavaClass(original); Object classData = CLASS_GET_CLASS_DATA_METHOD.invoke(clazz); - VMError.guarantee(LAMBDA_FORM_CLASS.isInstance(classData)); - int hash = classData.hashCode(); - return new MethodHandleInvokerSubstitutionType(original, findUniqueName(hash)); + if (LAMBDA_FORM_CLASS.isInstance(classData)) { + lambdaForm = classData; + } else if (classData instanceof List list && list.size() == 2) { + lambdaForm = list.get(0); + Object customizedHandle = list.get(1); + VMError.guarantee(LAMBDA_FORM_CLASS.isInstance(lambdaForm) && DIRECT_METHOD_HANDLE_CLASS.isInstance(customizedHandle) && + LAMBDA_FORM_CUSTOMIZED_FIELD.get(lambdaForm) == customizedHandle, "Expected classData to contain LambdaForm and its customization: %s", classData); + customizedMemberName = DIRECT_METHOD_HANDLE_INTERNAL_MEMBER_NAME_METHOD.invoke(customizedHandle); + } else { + throw VMError.shouldNotReachHere("Unexpected classData: %s", classData); + } } catch (ReflectiveOperationException e) { throw VMError.shouldNotReachHere(e); } + /* + * LambdaForm.hashCode() is not stable between image builds because it incorporates + * identity hash codes of objects such as those of Class that don't override + * hashCode(). For that reason, we compute a hash code from LambdaForm.toString(). It + * might also not be perfectly unique because the string contains unqualified class + * names and can contain string representations of constraints that may be arbitrary + * objects, but it should typically be distinct and stable. + */ + int hash = lambdaForm.toString().hashCode(); + if (customizedMemberName != null) { + /* MemberName.hashCode() also includes identity hash codes of Class objects. */ + hash = hash * 31 + customizedMemberName.toString().hashCode(); + } + return new MethodHandleInvokerSubstitutionType(original, findUniqueName(hash)); }); }