Breaking Change Request: Enable isolate groups by-default - Will result in changes to Performance Characteristics

[mkustermann]

Intended change
We intend to enable isolate group sdk/issues/36097 support in the VM by-default.

This will make isolates spawned via Isolate.spawn run inside the same Isolate Group and therefore operate on the same heap, allowing sharing of various kinds of objects and allowing better communication.

Intended change in behavior:
The intention is to

• make the per-isolate base memory overhead smaller (10x less RAM)
• make isolates faster to spawn (10x faster spawn latency)
• make isolates communicate faster (8x faster round-trip communication)
• make receiver isolate of messages mostly non-blocking (removes O(n) receiver cost)
• allow more rich communication between isolates (see sdk/issues/46623)
• allow sharing of objects (program structure, JITed code, constants as well as any String objects - in the future possibly also user-defined data structures)
• fix long-standing bugs that happen if isolates are used with the (currently non-atomic) hot-reload (e.g. flutter/issues/72195)
• allow flutter to smoothly use multiple engines flutter.dev/docs/development/add-to-app/multiple-flutters

The justification/rationale for making the change
Get the improvements mentioned above to users. Enabling more data sharing of objects across isolates in the future.

The expected impact of this change
There are no functional differences. There will be changes in performance characteristics. Those changes will almost exclusively be positive.

The thread pool onto which all lightweight isolates are multiplexed onto is limited (around 10 atm), in order to ensure all threads executing on different cores have a big enough TLAB (thread local allocation buffer - a free chunk of memory from new space) to ensure fast bump allocation.

This means isolates will collaborate on garbage collections (and some other events like lazy JIT compilations, ...). As a consequence blocking GC operations (such as new space collections) will affect all isolates. The worst case pause time due to new space collections is unchanged, however heavily allocating isolates can impact other isolates.

In the common case where generational hypothesis holds (most objects die young) those collections continue being fast. Furthermore for Flutter specifically, any idle time on the UI thread is used to perform GCs, thereby also avoiding too long pause due to new space GCs. (The old space is mainly collected via concurrent marking & sweeping, thereby not stopping mutators)

The only existing use case that could be negatively impacted is existing apps for which many isolates are executing in parallel on many cores (e.g. big server applications).

3 important Dart customers (including flutter) have been opt'ing into this already for a longer period of time in AOT mode. So far we have only heard positive feedback from them (especially memory footprint reductions).

We expect the only use case that might be actually affected by this change is e.g. server customers that use isolates on many threads at the same time.

Clear steps for mitigating the change

For customers that use isolates on many threads at the same time (such running on large servers), the possible workaround is to use Isolate.spawnUri()on the same application - this will cause the VM to use an independent Isolate Group which gives the old behavior. Though communication is then restricted to json-like types.

(see also go/ig-by-default)

[mkustermann]

/cc @a-siva @mraleph @aam (VM)
/cc @Hixie @xster @gaaclarke (Flutter)
/cc @vsmenon @mit-mit

Feel free to CC anyone else.

[xster]

To clarify, the new lightweight isolate implementation now supports both AOT and JIT right?

[aam]

To clarify, the new lightweight isolate implementation now supports both AOT and JIT right?

Right.

[mkustermann]

The breaking change has been announced on here

@mit-mit Could you help get any necessary approvals (since @franklinyow is out)?

[Jonas-Sander]

We expect the only use case that might be actually affected by this change is e.g. server customers that use isolates on many threads at the same time.

Have there been any measurements done on how much performance this might cost?
Additionally is this just a temporary pain point which will be made more performant in the future? Or is this something where the Dart team doesn't see Dart as an important use-case and thus won't optimize for?

I think espeically with the functions framework and Dart being more and more popular it might actually be become more important in the future (of course this is just my speculation).

[mkustermann]

Have there been any measurements done on how much performance this might cost?

We have done measurements on worst case scenarios. For example when 8 threads are continuously building up data structures that live for a long time (that means the generational hypothesis - which most VMs optimize for - does not hold) on 10+ GB heap can lead to a 2x slowdown.

Additionally is this just a temporary pain point which will be made more performant in the future?

Firstly, it's unclear whether any of our existing users would run into any pain points in practice (so far we are not aware of any).

We do have ideas about how this could be further optimized and might invest in that if we believe it is worthwhile.

Or is this something where the Dart team doesn't see Dart as an important use-case and thus won't optimize for?

Right now the VM team is not optimizing for large server use cases (i.e. 10-100s of cores and large amount of RAM) - because that is not how our users use Dart atm.

That being said, one can use the VM to work well in this setting - e.g. by using many isolate groups (as mentioned in the mitigation section above)

I think especially with the functions framework and Dart being more and more popular it might actually be become more
important in the future (of course this is just my speculation).

To the best of my knowledge cloud functions are often ephemerally executed, requests are independent of each other and little global state is kept. Based on that I wouldn't say this change would negatively impact such cases (it may even benefit such use cases).

[Jonas-Sander]

Thanks for your detailed response! :)

[aam]

Additionally is this just a temporary pain point which will be made more performant in the future? Or is this something where the Dart team doesn't see Dart as an important use-case and thus won't optimize for?

With spawning new isolates being at least 10x faster, potentially what we will see is that applications start start using short-lived/just-in-time-spawned isolates significantly more in addition or instead of larger long-running isolates. So in other words use of isolates could become more functions-oriented, similar to how compute in flutter is built. sendAndExit(sendPort, message) that is to be released in the future will speed up this flow even further.

[mkustermann]

Here are numbers from benchmarks we added specifically to see the impact of this change. It was measured in standalone Dart VM (JIT and AOT) on an Intel CPU at commit 5c0466a:

Benchmark	JIT Before	JIT After	JIT Change	AOT Before	AOT After	AOT Change
IsolateSpawn.Dart2JSToFinishRunningRMS	3137316.00	780292.63	-75.13 %	394717.94	396074.63	0.3437 %
IsolateSpawn.Dart2JSToStartRunningRMS	89621.59	574.28	-99.36 %	17494.70	259.79	-98.52 %
IsolateSpawnMemory.Dart2JSDeltaPeakProcessRss	972947456.00	782774272.00	-19.55 %	381440000.00	258723840.00	-32.17 %
IsolateSpawnMemory.Dart2JSDeltaRssOnStart	27043156.00	3941717.00	-85.42 %	18012842.00	1769472.00	-90.18 %
IsolateSpawnMemory.Dart2JSDeltaRssOnEnd	77070336.00	551594.00	-99.28 %	108882600.00	67689128.00	-37.83 %

We measured how long it takes for a mid-to-large application (dart2js in this case) to spawn a new isolate and what the additional base memory overhead is for such an isolate. (The helper isolate will run dart2js on a dart file)

We can see that

• Spawning latency went from being O(n) in application size to being effectively a constant (JIT: 574 us, AOT: 260 us).
• The spawned isolate can re-use JITed code and runs therefore much faster right from the start (it runs 4x faster).
• The peak memory consumption reduces in JIT as well as AOT
• The additional memory per isolate reduces by around 10x

Benchmark	JIT Before	JIT After	JIT Change	AOT Before	AOT After	AOT Change
SendPort.Receive.BinaryTree.2	5.2478	1.0162	-80.64 %	4.0722	1.2520	-69.25 %
SendPort.Receive.BinaryTree.4	12.357	1.0609	-91.41 %	6.3499	1.2568	-80.21 %
SendPort.Receive.BinaryTree.6	39.917	1.1565	-97.10 %	14.764	1.3500	-90.86 %
SendPort.Receive.BinaryTree.8	150.46	1.2812	-99.15 %	47.475	1.5075	-96.82 %
SendPort.Receive.BinaryTree.10	628.53	1.9064	-99.70 %	194.44	2.1566	-98.89 %
SendPort.Receive.BinaryTree.12	2548.28	4.2936	-99.83 %	794.96	4.3565	-99.45 %
SendPort.Receive.BinaryTree.14	10246.44	4.6576	-99.95 %	3256.48	4.6533	-99.86 %
SendPort.Receive.Json.400B	5.9036	1.1105	-81.19 %	6.3360	1.3150	-79.25 %
SendPort.Receive.Json.5KB	52.174	1.2674	-97.57 %	53.186	1.5660	-97.06 %
SendPort.Receive.Json.50KB	503.79	2.1735	-99.57 %	476.19	2.4452	-99.49 %
SendPort.Receive.Json.500KB	5194.21	5.1109	-99.90 %	4971.65	5.1561	-99.90 %
SendPort.Receive.Json.5MB	5579.90	19.9008	-99.98 %	3991.09	19.9308	-99.98 %
SendPort.Receive.Nop	0.84245	0.83353	-1.059 %	0.94660	0.95422	0.8056 %
SendPort.Send.BinaryTree.2	2.6803	0.94366	-64.79 %	2.7459	0.99268	-63.85 %
SendPort.Send.BinaryTree.4	7.5588	2.1675	-71.33 %	8.2269	2.6841	-67.37 %
SendPort.Send.BinaryTree.6	28.467	7.6131	-73.26 %	31.281	10.248	-67.24 %
SendPort.Send.BinaryTree.8	105.96	30.193	-71.51 %	117.82	30.515	-74.10 %
SendPort.Send.BinaryTree.10	419.48	110.05	-73.77 %	404.91	112.05	-72.33 %
SendPort.Send.BinaryTree.12	1738.89	544.90	-68.66 %	1739.35	523.80	-69.89 %
SendPort.Send.BinaryTree.14	7359.78	2336.84	-68.25 %	7435.21	2295.89	-69.12 %
SendPort.Send.Json.400B	5.3041	1.3875	-73.84 %	5.3811	1.2730	-76.34 %
SendPort.Send.Json.5KB	87.984	18.209	-79.30 %	83.766	18.214	-78.26 %
SendPort.Send.Json.50KB	837.65	177.17	-78.85 %	799.39	160.88	-79.88 %
SendPort.Send.Json.500KB	9134.69	2186.36	-76.07 %	8697.33	1972.85	-77.32 %
SendPort.Send.Json.5MB	113778.09	50449.93	-55.66 %	110797.18	46839.12	-57.73 %
SendPort.Send.Nop	0.30535	0.30177	-1.171 %	0.28601	0.30424	6.376 %

Here we can see that the isolate receiving messages will no longer pay a O(n) cost, it is rather constant single-digit us.
We can also see that sending json has become significantly faster, around 4x.

Together the send-and-receive is around 8x faster.

Benchmark	JIT Before	JIT After	JIT Change	AOT Before	AOT After	AOT Change
Isolate.SendReceiveBytes100KB	12903.66	25176.57	95.11 %	12683.20	25339.64	99.79 %
Isolate.SendReceiveBytes100MB	5.9831	7.3162	22.28 %	7.3251	11.073	51.17 %
Isolate.SendReceiveBytes10KB	36008.31	62362.08	73.19 %	35959.83	59863.60	66.47 %
Isolate.SendReceiveBytes10MB	59.690	55.972	-6.229 %	75.595	110.89	46.68 %
Isolate.SendReceiveBytes1KB	48756.69	80447.05	65.00 %	50621.82	78842.88	55.75 %
Isolate.SendReceiveBytes1MB	212.57	477.19	124.5 %	398.70	745.68	87.03 %

We can observe that sending bytes between isolates became between 1.5-2x faster.

(Metric is runs/second, larger numbers are therefore better)

Benchmark	JIT Before	JIT After	JIT Change	AOT Before	AOT After	AOT Change
IsolateJson.Decode100KBx1	1.7628	41.694	2265 %	19.562	38.126	94.90 %
IsolateJson.Decode100KBx4	1.2302	35.571	2791 %	11.319	34.181	202.0 %
IsolateJson.Decode1MBx1	1.0308	4.0661	294.5 %	2.3259	3.4675	49.08 %
IsolateJson.Decode1MBx4	0.67100	3.3272	395.9 %	1.3408	2.5559	90.64 %
IsolateJson.Decode250KBx1	1.4840	16.600	1019 %	8.3048	14.133	70.18 %
IsolateJson.Decode250KBx4	1.0702	11.765	999.4 %	5.1592	10.635	106.1 %
IsolateJson.Decode50KBx1	1.8633	23.011	1135 %	28.397	70.602	148.6 %
IsolateJson.Decode50KBx4	1.4497	61.380	4134 %	17.875	61.395	243.5 %
IsolateJson.SendAndExit_Decode100KBx1	1.7593	42.210	2299 %	19.747	38.317	94.04 %
IsolateJson.SendAndExit_Decode100KBx4	1.2352	36.599	2863 %	11.922	34.356	188.2 %
IsolateJson.SendAndExit_Decode1MBx1	1.0358	4.1429	300.0 %	2.3442	3.4788	48.40 %
IsolateJson.SendAndExit_Decode1MBx4	0.66169	2.3940	261.8 %	1.3342	2.4545	83.97 %
IsolateJson.SendAndExit_Decode250KBx1	1.4895	16.591	1014 %	8.2664	14.057	70.05 %
IsolateJson.SendAndExit_Decode250KBx4	1.0751	12.811	1092 %	5.2002	11.584	122.8 %
IsolateJson.SendAndExit_Decode50KBx1	1.9263	68.975	3481 %	29.038	68.695	136.6 %
IsolateJson.SendAndExit_Decode50KBx4	1.4529	61.024	4100 %	18.488	62.664	239.0 %

(Metric is runs/second, larger numbers are therefore better)

We also measure json decoding on helper isolates (in x1 or x4 isolates). The isolates are receiving bytes, perform utf-8 decoding followed by json decoding and send the result back.

We can observe

• Due to re-using JITed code, the helper isolate is much faster in decoding json than before (where it had to re-JIT everything).

• The faster isolate communication (which is only part of this benchmark's work) has led to a speedup of the benchmark by 1.5x-3x

Benchmark	JIT Before	JIT After	JIT Change	AOT Before	AOT After	AOT Change
EventLoopLatencyJson.Percentile95	1008.00	13815.0	1271 %	1006.00	11897.0	1083 %
EventLoopLatencyJson.Percentile99	1023.00	19878.0	1843 %	1015.00	18424.0	1715 %
EventLoopLatencyJson350KB.Percentile95	1010.00	1400.00	38.61 %	1005.00	1010.00	0.4975 %
EventLoopLatencyJson350KB.Percentile99	1018.00	2112.00	107.5 %	1012.00	1992.00	96.84 %

This shows us that if helper isolates allocate a lot of objects which die young the pause times in the main isolate are not affected much (1-2 ms). If the generational hypothesis doesn't hold (all objects survive - hypothetical worst case scenario) then the main isolate will have the same pause time as the helper isolate that triggers young collections (since young generation is collected with stop-the-world) - which is between 10-20 ms.

Though for flutter this would look different, since it will use idle time between frames to cause GCs, therefore doing more young space collections (before it's full) therefore reducing the pause times (for this hypothetical worst case scenario).

[gmpassos]

This is a very important "upgrade" to DartVM Isolate.

I want to highlight the context of changes in performance:

• Isolates in the same group will collaborate with the same GC:

  • Since the amount of shared memory between Isolates is increased (what reduces the total number of Objects), the total CPU time of GC is reduced (if compared with multiple Isolates in separated groups).
  • Total JIT time is reduced (if compared with multiple Isolates in separated groups).

• The current DartVM Isolate situation (Dart 2.13.4) makes impractical to spawn a high number of Isolates, due to bottlenecks (naturals to the current situation) in memory, GC and JIT.

  • The new approach points a theoretical decrease in performance for servers with a high number of Isolates, but this is already difficult with the current situation. The share of memory, GC and JIT can actually improve the capacity of Isolates.

• The gain in SendPort performance impact.

  • The main trade-off for message based parallelism (Isolates) is the time to serialize, send, receive and deserialize the messages. Sometimes this trade-off makes impractical to use multiple Isolates, since the time to send a message is near the time to compute the related task.
  • The significant improvement in SendPort increases the cases where is interesting to use a DartVM Isolate. The current Isolate performance actually makes many scenarios impractical for parallelism in the current DartVM.

Some questions?:

• Will be possible to force a new spawned Isolate to be in a different group?
  • This can be interesting to craft a solution that separates GCs.
  • This feature is important not only for spawnUri, but for any kind of entrypoint (normal Isolate.spawn).

[aam]

Will be possible to force a new spawned Isolate to be in a different group?

Isolate spawnUri (unlike Isolate spawn) spawns new isolate in its own isolate group.

[gmpassos]

It will be interesting to spawn a normal entrypoint/function (not spwanUri) in a different group, to allow specific optimizations for some solutions.