Invalid Computation Fusion in GT Backends

[fthaler]

The following code does not validate using the gtx86 and gtmc backends (and probably gtcuda):

import numpy as np
import gt4py
from gt4py.gtscript import stencil, computation, BACKWARD, Field


def foo(out: Field[float]):
    with computation(BACKWARD), interval(...):
        tmp = 1.0

    with computation(BACKWARD):
        with interval(-1, None):
            out = tmp + tmp[0, 0, -1]
        with interval(0, -1):
            out = out[0, 0, 1]


def validate_with_backend(backend):
    data = gt4py.storage.zeros(backend=backend,
                               default_origin=(0, 0, 0),
                               shape=(1, 1, 10),
                               dtype=float)
    stencil(backend=backend, definition=foo)(data)
    result = 'validates' if np.allclose(np.asarray(data), 2) else 'fails'
    print(f'{result} with backend {backend}')


validate_with_backend('debug')
validate_with_backend('numpy')
validate_with_backend('gtx86')
validate_with_backend('gtmc')

In the GT backends, the first and second computation are merged into a single backward loop, introducing a read before write on tmp[0, 0, -1] in the second computation.

[jdahm]

@fthaler - Good find. We now believe the problem goes further than that. Even when gt4py generates "correct" GridTools code, we get incorrect results. Here is an example:

from gt4py import storage
from gt4py.gtscript import stencil, Field

backend = "gtmc"
nx = 6
nk = 3
shape = (nx, nx, nk)


@stencil(backend)
def example(field: Field[float]):
    with computation(PARALLEL), interval(...):
        tmp = 1
    with computation(PARALLEL):
        with interval(0, -1):
            field = tmp[0, 0, 1]
        with interval(-1, None):
            field = tmp[0, 0, 0]


storage = storage.zeros(backend, default_origin=(0, 0, 0), shape=shape, dtype=float)
example(storage)
print(storage[:, :, 0].T)

Despite gt4py correctly generating GT C++ code with two parallel multistages, it seems like GridTools is not adding the proper synchronization between adjacent multistages of the same type. Here it is failing because tmp[0, 0, 1] is not yet written to when referenced.

[jdahm]

cc: @anstaf

[anstaf]

@fthaler - Good find. We now believe the problem goes further than that. Even when gt4py generates "correct" GridTools code, we get incorrect results. Here is an example:

from gt4py import storage
from gt4py.gtscript import stencil, Field

backend = "gtmc"
nx = 6
nk = 3
shape = (nx, nx, nk)


@stencil(backend)
def example(field: Field[float]):
    with computation(PARALLEL), interval(...):
        tmp = 1
    with computation(PARALLEL):
        with interval(0, -1):
            field = tmp[0, 0, 1]
        with interval(-1, None):
            field = tmp[0, 0, 0]


storage = storage.zeros(backend, default_origin=(0, 0, 0), shape=shape, dtype=float)
example(storage)
print(storage[:, :, 0].T)

Despite gt4py correctly generating GT C++ code with two parallel multistages, it seems like GridTools is not adding the proper synchronization between adjacent multistages of the same type. Here it is failing because tmp[0, 0, 1] is not yet written to when referenced.

GT C++ assumes that within the parallel execution it should be no vertical dependencies. It is not the case in this example.

[jdahm]

GT C++ assumes that within the parallel execution it should be no vertical dependencies. It is not the case in this example.

But "within the parallel execution"s there indeed is no vertical dependency. The dependency is "across" parallel multistages where the execution model indicates there should be a global synchronization

From https://gridtools.github.io/gridtools/latest/user_manual/user_manual.html#execution-model

More concretely, a multistage is a list of stages, implemented with stencil operators, to be executed with a certain execution policy. A computation combines several multistages and executes one multistage after the other.

So GridTools C++ merges every pair of parallel multistages without checking for data dependencies? In that way, they behave differently from FORWARD and BACKWARD multistages?

[jdahm]

I realized after I wrote my message initially that this is not to the issue that @fthaler originally posted about above, so we should discuss elsewhere.

[fthaler]

Seems to be fixed in current master.