[GR-48956] Add stable option for debug options OmitInlinedMethodDebugLineInfo, TrackNodeSourcePosition, DeleteLocalSymbols and SourceLevelDebug

[adinn]

The ability to debug some aspects of GraalVM Native program execution relies on configuration of the following command line options:

-H:+SourceLevelDebug
-H:-OmitInlinedMethodDebugLineInfo
-H:+TrackNodeSourcePosition

In addition it is necessary to configure the following command line option in order for Linux tools (perf, kcachegrind) to be able to use the debug info provide execution performance analysis:

-H:-DeleteLocalSymbols

All 4 options are currently being flagged as experimental options. Since these flags are needed to support important operations that may be needed both during development and after deployment it seems inappropriate to classify their use as 'experimental'. So, they all merit being recognized as 'stable'.

[adinn]

n.b. There is an inconsistency in reporting of option -H:+TrackNodeSourcePosition. It is listed in the individual warnings:

Warning: The option '-H:-DeleteLocalSymbols' is experimental and must be enabled via '-H:+UnlockExperimentalVMOptions' in the future.
Warning: The option '-H:+TrackNodeSourcePosition' is experimental and must be enabled via '-H:+UnlockExperimentalVMOptions' in the future.
Warning: The option '-H:-OmitInlinedMethodDebugLineInfo' is experimental and must be enabled via '-H:+UnlockExperimentalVMOptions' in the future.
Warning: The option '-H:+SourceLevelDebug' is experimental and must be enabled via '-H:+UnlockExperimentalVMOptions' in the future.

but not in the later summary:

 3 experimental option(s) unlocked:
 - '-H:-DeleteLocalSymbols' (origin(s): command line)
 - '-H:-OmitInlinedMethodDebugLineInfo' (origin(s): command line)
 - '-H:+SourceLevelDebug' (origin(s): command line)

[olpaw]

-H:+SourceLevelDebug
-H:-OmitInlinedMethodDebugLineInfo
-H:+TrackNodeSourcePosition

@adinn do you see actual users make use of those options or are you concerned that we use them in mx_substratevm._debuginfotest (which is not a problem at all)?

[adinn]

We actually recommend them to Quarkus Native users in order for them to be able to debug and perform perf analysis on their deployed programs. That will also cover the case where we help them debug the code as part of our support role -- as soon as proper Quarkus Native support rolls in in the next few months. Without using these settings it is frequently much harder -- and sometimes impossible -- to debug certain classes of problem that might manifest in the compiled code.

Of course, I also use all these flags regularly when developing and testing debug info. I understand that I can always pass the flags to unlock experimental options.

I'm not clear what is meant to be experimental about these options and am concerned that our users may be concerned by the warnings. I certainly don't view any of them as being 'transiently available' options (or meriting being treated as such).

If the implication is that any of these options might be removed then that would be a serious limit on the ability to debug a program when it manifested unexpected behaviour. If it is merely that they might be replaced with different option names and/or some different combination of options to obtain the same debug capability then I can see how 'experimental' might be appropriate.

[olpaw]

If it is merely that they might be replaced with different option names and/or some different combination of options to obtain the same debug capability then I can see how 'experimental' might be appropriate.

Exactly that. Experimental for us means: "Do not use this option for images that you use in production". We have no intention of getting rid of any of those options that you listed above.

Regarding -H:-DeleteLocalSymbols I do see a point in having that option stable because it's need for proper profiling images with perf. An alternative to making it stable would be to introduce an API option -gperf that builds an image that will work well for getting profiled with perf, WDYT? cc @jovanstevanovic

[adinn]

Exactly that. Experimental for us means: "Do not use this option for images that you use in production". We have no intention of getting rid of any of those options that you listed above.

Well, as far as Quarkus Native is concerned we definitely intend to recommend employing all those flags when producing production images. Our view is that should a native image display different behaviour to that manifested on the JVM then omission of those flags will make it impossible to debug the fault in many (probably most) cases.

[adinn]

I am happy to go along with use of -gperf to allow perf compatible symbols to be generated. However, note that high quality analysis from perf or kcachegrind also relies on the other 3 options being enabled. Without them the tools provide limited and misleading reports.

I am also not happy with the idea that you can debug production images without the other options being enabled.

Omitting inline info is a complete non-starter given the inlining depth that the Graal compiler employs. It makes it almost impossible to relate generated code back to the original source, especially code that is inserted by the GraalVM runtime. There is an (experimental ;-) flag to allow inline info up to a certain depth which looks like a useful alternative but my experience is that this forces an arbitrary cutoff, frequently cutting off access to GraalVM runtime code and leaving it very much up to chance as to whether it is possible to debug a problem.

Omitting SourceLevelDebug means that a great deal of debug info that will be critical to understanding a JVM vs Native behavioural disparity will not be not available. I accept that this sort of disparity ought only to occur rarely. However, if/when it does happen my existing experience of debugging similar problems (where a change to Substrate code has caused an invalid outcome) suggests it will be impossible to debug the problem without full debug info.

Likewise, omitting TrackNodeSourcePosition also means that a great deal of debug info that is critical to understanding a JVM vs Native behavioural disparity will not be not available.

I'd be happy to see this triad of options made available via a stable '-g' option e.g. -gfull or some such.

[fniephaus]

Well, as far as Quarkus Native is concerned we definitely intend to recommend employing all those flags when producing production images.

Do you mean employing all those flags individually? With -g -O0, users can already control the majority of the hosted options you would like to be marked as stable with public API options. BTW: -H:-OmitInlinedMethodDebugLineInfo this is the default, so there's not really a need to mark that stable either.

[fniephaus]

I am happy to go along with use of -gperf to allow perf compatible symbols to be generated. However, note that high quality analysis from perf or kcachegrind also relies on the other 3 options being enabled.

Nothing keeps -gperf from enabling a group of hosted options. Users can either turn the group on or off, controlling options individually can probably remain experimental, right?

[olpaw]

Omitting inline info is a complete non-starter given the inlining depth that the Graal compiler employs.

Since I merged #6504 the default value of OmitInlinedMethodDebugLineInfo is now false. This was possible because I made CompilationResultFrameTree generation to only use infopoints per default (i.e. DebugCodeInfoUseSourceMappings is false per default) and now the tree generation scales to huge graphs. This has the effect that we can afford having OmitInlinedMethodDebugLineInfo with default value false. If you look at the PR you see how I removed '-H:-OmitInlinedMethodDebugLineInfo', from the _debuginfotest in mx_substratevm.py because it is now redundant.

So nowadays -g is actually pretty close to the OOTB debug-experience that want to have for users.

[adinn]

Do you mean employing all those flags individually? With -g -O0, users can already control the majority of the hosted options you would like to be marked as stable with public API options.

I'm not sure I follow what you are saying here. Why would anyone generate a production image using -O0?