Warp/composite mesh U/V coordinates cause issues with some preset shaders

[kblaschke]

After the renderer rewrite, some presets still do not render properly, leaving parts or the whole the image black. A few presets where this is quite obvious are:

• martin - stormy sea
• Goody + martin - crystal palace - Aqua Lumens
• Goody + martin - crystal palace - Ocular Anima

The reason for this is that some presets use U/V coordinates in the exponent part of the pow() function, which is not allowed to be negative. Since texture coordinates - at least in OpenGL - can be negative, this will cause the result to become NaN, breaking any subsequent calculations on the value. It might affect other presets as well, not causing clearly visible back areas but other less obvious artifacts as the values may not be in the range expetected by the preset author.

One possible fix might be to wrap the U/V coordinates in the warp mesh vertex shader to always be in the range [0...1[.

[kblaschke]

Note this isn't directly related to issue #725. While the other issue fixed some now superfluous negations in the u/v coordinates of the warp mesh, the issue here still remains unsolved.

[kblaschke]

@dpw13 has identified the issue to be undocumented, undefined and unexpected behavior in HLSL, causing some math functions like pow, sqrt and log to produce results even with negative input values, which - according to the HLSL documentation - should return NaN or Inf, but in reality seemingly take the absolute value of the parameter in question and thus return some kind of mirrored value.

Dane is fixing it in the hlslparser by adding abs() around the problematic parameters to make the shader behave in a similar fashion.

[dpw13]

The following intrinsics have a limited domain where their output is defined and meaningful. Documentation for both HLSL shader model 3 and GLSL say that the output of sqrt(x) where x < 0, for instance, say the output is NaN. However, in reality this function returns sqrt(abs(x)) if x < 0 in DX9's shader model 3 implementation. This has been verified using BeatDrop 1.3.1 after installing DX9 (note that BeatDrop requires DX9 specifically; otherwise you'll get errors about shader model 3 being reported as supported but non-functional).

By creating a shader that plots the output of these intrinsics, we can easily verify the behavior of each intrinsic. Here are the presets I used to plot these intrinsics. The R+G channels plot the unmodified intrinsic. The B channel plots the modified function to be used in projectM. If the plot has no yellow or blue regions, these functions overlap perfectly.

math_presets.zip

The functions include:

• acos: Defined only on the range [-1,1]. Shader model 3 appears to implement this using hyperbolic functions (this is just a guess) that remain valid outside of this range, but the equation is unknown. In an attempt to qualitatively replicate this behavior in GLSL, I've created a simple polynomial function and piecewise extended the function outside of the range [-1:1].
• asin: Same as the acos implementation. The same polynomial is used to mimic the SM3 behavior.
• pow: This function actually behaves identically in HLSL and GLSL. If the first argument is negative, the output should be NaN according to the documentation. The actual output is pow(abs(x),y) unless y is actually a constant integer, in which case the result is actually x^y. Note the difference in behavior between pow.milk and pow_constant.milk in the attached presets.
• rsqrt: The inverse square root is defined on the range [0,inf). SM3 appears to take the absolute value before taking the inverse square root, so this is replaced with rsqrt(abs(x)) in GLSL.
• sqrt: Same as rsqrt.
• log, log2, log10: Same as rsqrt; shader model 3 appears to take the absolute value of the argument before taking the logarithm (note I forgot to include the log presets but you can easily create them).

The trigonometric functions are not perfect, but qualitatively mimic the shader model 3 implementation until the actual function can be found:

The sqrt and rsqrt functions clearly show the abs behavior: