[GR-45654] Single compile unit for Java DWARF debug info

[adinn]

This PR reorganizes the DWARF debug info layout so that all info is all contained in a single compile unit (CU) rather than employing a CU for every Java class type.

It profits from that reorganization to achieve several important goals:

• reduce the overall size of debug info by avoiding replication of DIEs shared by multiple classes
• reduce the overall size of debug info by omitting unnecessary CU boiler plate
• decouple compiled method info and definition order from class declaration order, thereby freeing the image builder to order method code ranges without respect to class grouping

The re-implementation also has the benefit that it simplifies many of the DWARF generation steps, both when iterating the generic debug model to write specific debug sections and when cross-indexing location references within and across sections.

[fniephaus]

Thanks a lot for the PR, @adinn. It seems this significantly simplifies the debug info infrastructure, great! In addition to that and the reduced size of the debug info, does this also reduce footprint and/or improve the time to generate debug info? No need to run a full set of benchmarks, a rough estimate is probably good enough. :)

[fniephaus]

It seems we could use records for DirEntry, FileEntry, and MemberEntry in the future?

[adinn]

I'm not sure what you mean by that. Do you mean outside of the com.oracle.objectfile package? They are already used outside of com.oracle.objectfile.debugentry as they are referenced by com.oracle.objectfile.elf.dwarf classes.

[adinn]

@fniephaus I built a simple app (GC tester that turns over memory) with the following results

Before:

[adinn@sputstik substratevm]$ do_native_image +fulldebuginfo -debug -log GCTest
===========================================================================================
GraalVM Native Image: Generating 'gctest' (executable)...
===========================================================================================
[1/8] Initializing...                                                       (9.1s @ 0.20GB)
 Java version: 20+34, vendor: GraalVM Community
 Graal compiler: optimization level: '2', target machine: 'x86-64-v3'
 C compiler: gcc (redhat, x86_64, 12.2.1)
 Garbage collector: Serial GC (max heap size: 80% of RAM)
[2/8] Performing analysis...  [****]                                       (24.0s @ 2.32GB)
   3,188 (73.97%) of  4,310 types reachable
   3,820 (50.57%) of  7,554 fields reachable
  15,220 (45.50%) of 33,454 methods reachable
     958 types,    83 fields, and   491 methods registered for reflection
      57 types,    55 fields, and    52 methods registered for JNI access
       4 native libraries: dl, pthread, rt, z
[3/8] Building universe...                                                  (3.3s @ 2.22GB)
[4/8] Parsing methods...      [**]                                          (4.9s @ 1.84GB)
[5/8] Inlining methods...     [***]                                         (5.0s @ 2.09GB)
[6/8] Compiling methods...    [******]                                     (36.3s @ 4.23GB)
[7/8] Layouting methods...    [***]                                         (7.7s @ 1.46GB)
[8/8] Creating image...       [****]                                       (13.0s @ 0.96GB)
   5.42MB (16.81%) for code area:     8,802 compilation units
   7.62MB (23.60%) for image heap:  112,882 objects and 5 resources
  17.92MB (55.54%) for debug info generated in 4.2s
   1.31MB ( 4.05%) for other data
  32.27MB in total
-------------------------------------------------------------------------------------------
Top 10 origins of code area:                 Top 10 object types in image heap:
   4.11MB java.base                             1.15MB byte[] for code metadata
 937.44KB svm.jar (Native Image)                1.07MB java.lang.String
 113.80KB java.logging                        934.86KB byte[] for general heap data
  66.77KB org.graalvm.nativeimage.base        766.74KB byte[] for java.lang.String
  45.63KB jdk.proxy1                          740.47KB java.lang.Class
  37.40KB jdk.proxy3                          273.97KB c.o.s.core.hub.DynamicHubCompanion
  27.20KB jdk.internal.vm.ci                  263.20KB java.lang.Object[]
  24.73KB org.graalvm.sdk                     251.63KB java.util.HashMap$Node
  10.94KB jdk.proxy2                          192.34KB java.lang.String[]
   8.04KB jdk.internal.vm.compiler            168.05KB j.u.c.ConcurrentHashMap$Node
   6.72KB for 2 more packages                   1.89MB for 900 more object types
-------------------------------------------------------------------------------------------
Recommendations:
 HEAP: Set max heap for improved and more predictable memory usage.
 CPU:  Enable more CPU features with '-march=native' for improved performance.
-------------------------------------------------------------------------------------------
          3.2s (3.0% of total time) in 31 GCs | Peak RSS: 5.38GB | CPU load: 5.36
-------------------------------------------------------------------------------------------
Produced artifacts:
 /home/adinn/redhat/openjdk/graal/graal/substratevm/gctest (executable)
 /home/adinn/redhat/openjdk/graal/graal/substratevm/gctest.debug (debug_info)
 /home/adinn/redhat/openjdk/graal/graal/substratevm/sources (debug_info)
===========================================================================================
Finished generating 'gctest' in 1m 44s.
[adinn@sputstik substratevm]$ ls -l gctest.debug
-rwxr-xr-x. 1 adinn adinn 20010752 Apr 14 10:12 gctest.debug

After:

[adinn@sputstik substratevm]$ do_native_image +fulldebuginfo -debug -log GCTest
===========================================================================================
GraalVM Native Image: Generating 'gctest' (executable)...
===========================================================================================
[1/8] Initializing...                                                       (9.6s @ 0.20GB)
 Java version: 20+34, vendor: GraalVM Community
 Graal compiler: optimization level: '2', target machine: 'x86-64-v3'
 C compiler: gcc (redhat, x86_64, 12.2.1)
 Garbage collector: Serial GC (max heap size: 80% of RAM)
[2/8] Performing analysis...  [*****]                                      (22.0s @ 2.28GB)
   3,188 (73.97%) of  4,310 types reachable
   3,820 (50.57%) of  7,554 fields reachable
  15,220 (45.49%) of 33,456 methods reachable
     958 types,    83 fields, and   491 methods registered for reflection
      57 types,    55 fields, and    52 methods registered for JNI access
       4 native libraries: dl, pthread, rt, z
[3/8] Building universe...                                                  (3.6s @ 2.14GB)
[4/8] Parsing methods...      [***]                                         (5.2s @ 1.67GB)
[5/8] Inlining methods...     [***]                                         (4.1s @ 1.93GB)
[6/8] Compiling methods...    [******]                                     (32.1s @ 1.23GB)
[7/8] Layouting methods...    [***]                                         (6.3s @ 2.86GB)
[8/8] Creating image...       [****]                                       (12.1s @ 3.99GB)
   5.42MB (17.44%) for code area:     8,802 compilation units
   7.62MB (24.50%) for image heap:  112,877 objects and 5 resources
  16.75MB (53.86%) for debug info generated in 5.6s
   1.31MB ( 4.21%) for other data
  31.09MB in total
-------------------------------------------------------------------------------------------
Top 10 origins of code area:                 Top 10 object types in image heap:
   4.11MB java.base                             1.15MB byte[] for code metadata
 937.45KB svm.jar (Native Image)                1.07MB java.lang.String
 113.80KB java.logging                        934.89KB byte[] for general heap data
  66.77KB org.graalvm.nativeimage.base        766.73KB byte[] for java.lang.String
  45.63KB jdk.proxy1                          740.47KB java.lang.Class
  37.40KB jdk.proxy3                          273.97KB c.o.s.core.hub.DynamicHubCompanion
  27.20KB jdk.internal.vm.ci                  263.20KB java.lang.Object[]
  24.73KB org.graalvm.sdk                     251.34KB java.util.HashMap$Node
  10.94KB jdk.proxy2                          192.34KB java.lang.String[]
   8.04KB jdk.internal.vm.compiler            168.05KB j.u.c.ConcurrentHashMap$Node
   6.72KB for 2 more packages                   1.89MB for 900 more object types
-------------------------------------------------------------------------------------------
Recommendations:
 HEAP: Set max heap for improved and more predictable memory usage.
 CPU:  Enable more CPU features with '-march=native' for improved performance.
-------------------------------------------------------------------------------------------
          3.0s (3.1% of total time) in 30 GCs | Peak RSS: 5.26GB | CPU load: 5.57
-------------------------------------------------------------------------------------------
Produced artifacts:
 /home/adinn/redhat/openjdk/graal/graal/substratevm/gctest (executable)
 /home/adinn/redhat/openjdk/graal/graal/substratevm/gctest.debug (debug_info)
 /home/adinn/redhat/openjdk/graal/graal/substratevm/sources (debug_info)
===========================================================================================
Finished generating 'gctest' in 1m 35s.
[adinn@sputstik substratevm]$ ls -l gctest.debug 
-rwxr-xr-x. 1 adinn adinn 18778328 Apr 14 10:08 gctest.debug

n.b. the build script I am using, do_native_image, responds to those arguments by switching on debug info generation, full infopoints and retention of local ELF symbols (-debug means don't debug the generator run and -log means don't generate any logging output). So, it is a worst case for debug info size.

I'm not sure the reported heap sizes at each stage are reliable. As far as I am aware this change could not have reduced the compiler phase heap size by such a large amount nor could it have produced such a large increase in the heap sizes for layout and image generation. The peak RSS is perhaps the most reliable indicator -- but it only drops by ~2% so that may be noise.

The times for the layout and image create and the overall time do go down after this change by a couple of seconds, which might perhaps a bit more reliable, but the drop of ~4 seconds in compile time does not make much sense. So maybe any difference after this is also noise.

The one thing that I am fairly confident is reliable is the size of the debug info. It has gone down from 20010752 to 18778328 bytes i.e. ~6% or ~1.2Mb. Even though this is a small app it contains many hundreds of classes so the percentage saving is probably going to be a representative for all apps.

[fniephaus]

Thanks, @adinn! This LGTM but let's wait for @olpaw's review.