Julia level miscompile in 1.10 and 1.11

[oscardssmith]

On 1.10 and 1.11, f(x) = isnan(x) ? NaN : f(x) returns NaN instead of the expected StackOverflowError that it returns on 1.6-1.9 and nightly. The @code_typed is

CodeInfo(
1 ─ %1 = Base.ne_float(x, x)::Bool
└──      goto #3 if not %1
2 ─      return Main.NaN
3 ─      return NaN
) => Float64

while before optimization we get

julia> @code_typed optimize=false f(1.0)
CodeInfo(
1 ─ %1 = Main.isnan::Core.Const(isnan)
│   %2 = (%1)(x)::Bool
└──      goto #3 if not %2
2 ─ %4 = Main.NaN::Core.Const(NaN)
└──      return %4
3 ─ %6 = Main.f::Core.Const(f)
│   %7 = (%6)(x)::Core.Const(NaN)
└──      return %7
) => Float64

so this appears to be a bug in Julia level optimization.

[Seelengrab]

Is this conceptually related to forward progress assumptions?

[vtjnash]

It seems more likely to be related to effects inference. It is known that recursive functions may give incorrect results in v1.10, since the relevant fixes didn't get backported there (#51092), but that was fixed in v1.11, so there might have been a second cause also

[oscardssmith]

@Seelengrab yes if we had a forward progress assumption (which we don't), this would be a valid way to compile this function.

Surprisingly enough, I think the effects here are correct.

julia> Base.infer_effects(f, (Float64,))
(+c,+e,+n,!t,+s,+m,+u,+o,+r)

[oscardssmith]

bisecting points to 2cea685 and f10c461 for the 1.11 and 1.10 regressions of this (although weirdly neither seem to cause the issue in the associated master)

[oscardssmith]

so this was fixed by #51092 (which makes sense)

[oscardssmith]

And it looks like #50767 is the culprit that broke it.

[oscardssmith]

After discussing with @Keno and @aviatesk, this will likely not be backportable (it was fixed by ~3 months of effects work in 1.12), so closing as wontfix.

[KristofferC]

Also, this is not considered a bug in 1.10 and 1.11 due to lack of forward progress being undefined behavior (or at least not specified).