Python API support for CuPy.arrays and torch.Tensor

[anagainaru]

On the write side:

# define the torch tensor or the cupy array
cuda0 = torch.device('cuda:0')
gpuArray = torch.tensor([[0, 1.0, 2.0], [3.0, 4.0, 5.0]], dtype=torch.float32, device=cuda0)
# or 
gpuArray = cupy.array([[0, 1.0, 2.0], [3.0, 4.0, 5.0]], dtype=np.float32)

# write data directly from the python object
stream.write("gpuArray", gpuArray, gpuArray.shape, [0] * len(gpuArray.shape), gpuArray.shape)

On the read side, I added a function for receiving a pre-allocated buffer for the data

cpuBuffer = numpy.zeros(cpuShape, dtype=np.float32)
gpuBuffer = cupy.zeros(gpuShape, dtype=np.float32)
tensorBuffer = torch.zeros(tensorShape, dtype=torch.float32, device=cuda0)

stream.read_in_buffer("cpuArray", cpuBuffer)
stream.read_in_buffer("gpuArray", gpuBuffer)
stream.read_in_buffer("tensorArray", tensorBuffer)

Testing was added for the new function.

[anagainaru]

The Windows2022 CI fails with:

2025-05-21T15:10:39.3440570Z 1035: Step 8 shape (2, 9)
2025-05-21T15:10:39.3440761Z 1035: Step 9 shape (2, 10)
2025-05-21T15:10:39.3440983Z 1035: Reading as stream with BeginStep/EndStep:
2025-05-21T15:10:39.3441232Z 1035: Step 0 shape (2, 1)
2025-05-21T15:10:39.3441429Z 1035: Step 1 shape (2, 2)
2025-05-21T15:10:39.3441601Z 1035: Step 2 shape (2, 3)
2025-05-21T15:10:39.3441779Z 1035: Step 3 shape (2, 4)
2025-05-21T15:10:39.3443221Z 1035: unknown file: error: C++ exception with description "�[1;36m[Wed May 21 15:10:39 2025]�[1;34m [ADIOS2 EXCEPTION]�[0m <Engine> <BP5Reader> <ParseMetadataIndex> : Record 2 (id = 115) has invalid length 40. We parsed 265864 bytes for this record�[0m
2025-05-21T15:10:39.3443998Z 1035: " thrown in the test body.
2025-05-21T15:10:39.3445177Z 1035: unknown file: error: C++ exception with description "�[1;36m[Wed May 21 15:10:39 2025]�[1;34m [ADIOS2 EXCEPTION]�[0m <Engine> <BP5Reader> <ParseMetadataIndex> : Record 2 (id = 115) has invalid length 40. We parsed 265864 bytes for this record�[0m
2025-05-21T15:10:39.3446689Z 1035: " thrown in the test body.
2025-05-21T15:10:39.3447148Z 1035: [  FAILED  ] TimeSeries.WriteReadShape2D (487 ms)
2025-05-21T15:10:39.3447706Z 1035: [  FAILED  ] TimeSeries.WriteReadShape2D (487 ms)
2025-05-21T15:10:39.3448395Z 1035: [----------] 1 test from TimeSeries (487 ms total)
2025-05-21T15:10:39.3448841Z 1035: 
2025-05-21T15:10:39.3449159Z 1035: [----------] 1 test from TimeSeries (487 ms total)
2025-05-21T15:10:39.3449577Z 1035: 
2025-05-21T15:10:39.3449882Z 1035: [----------] Global test environment tear-down
2025-05-21T15:10:39.3450389Z 1035: [----------] Global test environment tear-down
2025-05-21T15:10:39.3450875Z 1035: [==========] 1 test from 1 test suite ran. (487 ms total)
2025-05-21T15:10:39.3451382Z 1035: [  PASSED  ] 0 tests.
2025-05-21T15:10:39.3451723Z 1035: [  FAILED  ] 1 test, listed below:
2025-05-21T15:10:39.3452166Z 1035: [  FAILED  ] TimeSeries.WriteReadShape2D
2025-05-21T15:10:39.3452570Z 1035: 
2025-05-21T15:10:39.3452805Z 1035:  1 FAILED TEST
2025-05-21T15:10:39.3453157Z 1035: [==========] 1 test from 1 test suite ran. (491 ms total)
2025-05-21T15:10:39.3453597Z 1035: [  PASSED  ] 0 tests.
2025-05-21T15:10:39.3453959Z 1035: [  FAILED  ] 1 test, listed below:
2025-05-21T15:10:39.3454420Z 1035: [  FAILED  ] TimeSeries.WriteReadShape2D
2025-05-21T15:10:39.3454830Z 1035: 
2025-05-21T15:10:39.3455061Z 1035:  1 FAILED TEST

This doesn't have anything to do with the python API, but we should try to look at it. @eisenhauer @pnorbert do you have an idea why this fails?

[eisenhauer]

Huh. Never seen that before. That's an error in ParseMetadataIndex, which is pretty simple and bulletproof. Only thing I can think of is data corruption on the write...

[anagainaru]

Hmm, I checked and the new test I added for the python API does not use the same BP file name. I guess let's see if it persists.