[vm] Check hash-based caches in the DRT_InstanceOf runtime entry.

We added a hash-based cache check in DRT_TypeCheck until the
SubtypeNTestCache stubs handle hash-based caches to avoid doing
the full runtime check when unnecessary, but did not do so for
DRT_InstanceOf. This fixes that until an upcoming CL that adds
support for hash-based caches in the SubtypeNTestCacheStubs lands.

TEST=Manual inspection of profiles before and after change. Not
adding an automated test as this change is a temporary bandaid
similar to the tested path in DRT_TypeCheck that will be removed soon.

Change-Id: I905a0fc6f501c68d66ae1ee848631c40cad04af2
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/310882
Reviewed-by: Martin Kustermann <[email protected]>
Commit-Queue: Tess Strickland <[email protected]>

Author: sstrickl

runtime/vm/runtime_entry.cc

@@ -1068,6 +1068,22 @@ DEFINE_RUNTIME_ENTRY(Instanceof, 5) {
   ASSERT(type.IsFinalized());
   ASSERT(!type.IsDynamicType());  // No need to check assignment.
   ASSERT(!cache.IsNull());
+  // Handle cases where currently the SubtypeNTestCache stubs return a false
+  // negative and the information is already in the cache.
+  //
+  // TODO(sstrickl): Remove this check and the associated helper function when
+  // the SubtypeNTestCache stubs have been updated to handle hash-based caches.
+  if (cache.IsHash()) {
+    const auto& result = Bool::Handle(
+        zone, ResultForExistingTypeTestCacheEntry(
+                  zone, thread, instance, type, instantiator_type_arguments,
+                  function_type_arguments, cache));
+    if (!result.IsNull()) {
+      // Early exit because an entry already exists in the cache.
+      arguments.SetReturn(result);
+      return;
+    }
+  }
   const Bool& result = Bool::Get(instance.IsInstanceOf(
       type, instantiator_type_arguments, function_type_arguments));
   if (FLAG_trace_type_checks) {