atomix and oneAPI under KernelAbstractions

[OsKnoth]

I run my code under oneAPI on dawn (Cambridge, UK) and

for a kernel with

@atomic :monotonic M[Iz,ind,1] += JJ[ID,1,Iz,IF]

we get the error

ERROR: LoadError: Failed to translate LLVM code to SPIR-V.
RequiresExtension: Feature requires the following SPIR-V extension:
SPV_EXT_shader_atomic_float_add

[OsKnoth]

After some more insights to oneAPI

I tried now the following short program on dawn (University of Cambridge)

using KernelAbstractions
using Atomix
using Adapt
using oneAPI

@kernel function mykernel(x, y)
i = @index(Global)
Atomix.@atomic x[1] += y[i]
end

backend = oneAPIBackend()
@show backend

x_ = zeros(Float64, 1)
y_ = rand(Float64, 100)
x = Adapt.adapt(backend, x_)
y = Adapt.adapt(backend, y_)

mykernel(backend)(x, y, ndrange=length(y))

@info "" sum(x) sum(y)

and get the following traceback

ERROR: LoadError: InvalidIRError: compiling MethodInstance for gpu_mykernel(::KernelAbstractions.CompilerMetadata{…}, ::oneDeviceVector{…}, ::oneDeviceVector{…}) resulted in invalid LLVM IR
Reason: unsupported dynamic function invocation (call to atomic_add!)
Stacktrace:
[1] modify!
@ ~/.julia/packages/Atomix/UFuJD/ext/AtomixoneAPIExt.jl:38
[2] gpu_mykernel
@ ~/.julia/packages/KernelAbstractions/HqVoT/src/macros.jl:324
[3] gpu_mykernel
@ ./none:0
Reason: unsupported call to an unknown function (call to gpu_malloc)
Stacktrace:
[1] malloc
@ ~/.julia/packages/GPUCompiler/Ecaql/src/runtime.jl:85
[2] gc_pool_alloc
@ ~/.julia/packages/GPUCompiler/Ecaql/src/runtime.jl:116
[3] modify!
@ ~/.julia/packages/Atomix/UFuJD/ext/AtomixoneAPIExt.jl:38
[4] gpu_mykernel
@ ~/.julia/packages/KernelAbstractions/HqVoT/src/macros.jl:324
[5] gpu_mykernel
@ ./none:0
Hint: catch this exception as err and call code_typed(err; interactive = true) to introspect the erroneous code with Cthulhu.jl
Stacktrace:
Stacktrace:
[1] check_ir(job::GPUCompiler.CompilerJob{GPUCompiler.SPIRVCompilerTarget, oneAPI.oneAPICompilerParams}, args::LLVM.Module)
@ GPUCompiler ~/.julia/packages/GPUCompiler/Ecaql/src/validation.jl:167
[2] macro expansion
@ ~/.julia/packages/GPUCompiler/Ecaql/src/driver.jl:385 [inlined]
[3] macro expansion
@ ~/.julia/packages/Tracy/slmNc/src/tracepoint.jl:163 [inlined]
[4] macro expansion
@ ~/.julia/packages/GPUCompiler/Ecaql/src/driver.jl:384 [inlined]
[5] emit_llvm(job::GPUCompiler.CompilerJob; kwargs::@kwargs{})
@ GPUCompiler ~/.julia/packages/GPUCompiler/Ecaql/src/utils.jl:116
[6] emit_llvm(job::GPUCompiler.CompilerJob)
@ GPUCompiler ~/.julia/packages/GPUCompiler/Ecaql/src/utils.jl:114
[7] compile_unhooked(output::Symbol, job::GPUCompiler.CompilerJob; kwargs::@kwargs{})
@ GPUCompiler ~/.julia/packages/GPUCompiler/Ecaql/src/driver.jl:95
[8] compile_unhooked
@ ~/.julia/packages/GPUCompiler/Ecaql/src/driver.jl:80 [inlined]
[9] compile(target::Symbol, job::GPUCompiler.CompilerJob; kwargs::@kwargs{})
@ GPUCompiler ~/.julia/packages/GPUCompiler/Ecaql/src/driver.jl:67
[10] compile
@ ~/.julia/packages/GPUCompiler/Ecaql/src/driver.jl:55 [inlined]
[11] #37
@ ~/.julia/packages/oneAPI/eeEyI/src/compiler/compilation.jl:81 [inlined]
[12] JuliaContext(f::oneAPI.var"#37#38"{GPUCompiler.CompilerJob{…}}; kwargs::@kwargs{})
@ GPUCompiler ~/.julia/packages/GPUCompiler/Ecaql/src/driver.jl:34
[13] JuliaContext(f::Function)
@ GPUCompiler ~/.julia/packages/GPUCompiler/Ecaql/src/driver.jl:25
[14] compile(job::GPUCompiler.CompilerJob)
@ oneAPI ~/.julia/packages/oneAPI/eeEyI/src/compiler/compilation.jl:80
[15] actual_compilation(cache::Dict{…}, src::Core.MethodInstance, world::UInt64, cfg::GPUCompiler.CompilerConfig{…}, compiler::typeof(oneAPI.compile), linker::typeof(oneAPI.link))
@ GPUCompiler ~/.julia/packages/GPUCompiler/Ecaql/src/execution.jl:245
[16] cached_compilation(cache::Dict{…}, src::Core.MethodInstance, cfg::GPUCompiler.CompilerConfig{…}, compiler::Function, linker::Function)
@ GPUCompiler ~/.julia/packages/GPUCompiler/Ecaql/src/execution.jl:159
[17] macro expansion
@ ~/.julia/packages/oneAPI/eeEyI/src/compiler/execution.jl:203 [inlined]
[18] macro expansion
@ ./lock.jl:273 [inlined]
[19] zefunction(f::typeof(gpu_mykernel), tt::Type{Tuple{…}}; kwargs::@kwargs{})
@ oneAPI ~/.julia/packages/oneAPI/eeEyI/src/compiler/execution.jl:198
[20] zefunction(f::typeof(gpu_mykernel), tt::Type{Tuple{KernelAbstractions.CompilerMetadata{…}, oneDeviceVector{…}, oneDeviceVector{…}}})
@ oneAPI ~/.julia/packages/oneAPI/eeEyI/src/compiler/execution.jl:195
[21] macro expansion
@ ~/.julia/packages/oneAPI/eeEyI/src/compiler/execution.jl:66 [inlined]
[22] (::KernelAbstractions.Kernel{…})(::oneArray{…}, ::Vararg{…}; ndrange::Int64, workgroupsize::Nothing)
@ oneAPI.oneAPIKernels ~/.julia/packages/oneAPI/eeEyI/src/oneAPIKernels.jl:89
[23] top-level scope
@ ~/GitHub/CliMA/CGDycore.jl/TestKernels/OneAPIAtomic.jl:19
[24] include(fname::String)
@ Main ./sysimg.jl:38
[25] top-level scope
@ REPL[1]:1
in expression starting at /home/dn-knot1/GitHub/CliMA/CGDycore.jl/TestKernels/OneAPIAtomic.jl:19
Some type information was truncated. Use show(err) to see complete types.