How to create/read additional variables in an operator?

[pnorbert]

Some operator needs to create additional information and save that along the output of the operator (compressed data). Then, the uncompression call also needs access to this additional information. It does not fit into a header of the actual variable.

One idea is to create new ADIOS variables in the output. How could one do that inside an Operator, how could it get access to the Engine object? On the read side, it is even more tricky, as it would want to read more variables within the PerformGets() loop of the engine.

A simpler idea is to write a new file under the output .bp file so that when users move the data, the additional data is also safely moves around. (The discussion is started exactly because the current solution that writes a separate file in a separate location does work but will be soon lost). In this case, how can the Operator know the name of the current output so that it could create a new output under it?

@gqian-coder

[eisenhauer]

Just how big is this new data? So large that we can't just make a larger (possibly variably sized) header? (Both new files and new variables sound unpleasant...)

[gqian-coder]

The additional metadata is something like the following: uint64_t GridDim [96]*{3} = 80 / 653 char GridSparsity [96]*{1} uint64_t MeshGridCluster [96]*{__} = 1 / 90 uint64_t MeshGridMap [96]*{__} = 0 / 289660 The "96" is the number of ADIOS blocks. The {__} of MeshGridMap is the number of mesh nodes per block. The {__} of MeshGridCluster has a smaller size, usually 10x smaller than the {__} of MeshGridMap. This metadata is one copy used across all variables and time steps during decompression. The size of variables will be {timestep}x{blocks}x{number of mesh nodes}. Qian

…

On Wed, Jun 4, 2025 at 11:44 AM Greg Eisenhauer ***@***.***> wrote: Just how big is this new data? So large that we can't just make a larger (possibly variably sized) header? (Both new files and new variables sound unpleasant...) — Reply to this email directly, view it on GitHub <#4542 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AMOUPYP25Q7IHZF4MNR3Q333B4H6ZAVCNFSM6AAAAAB6S4USU2VHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTGMZWHE3DIOA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[eisenhauer]

OK, more, maybe stupid questions... I'm assuming that the "x / y" stuff is just min/max from bpls, but let me know if that has some other meaning I should be aware of.

So, if 96 is the number of blocks (for this variable and on this timestep?), then presumably this can change? Or is there some assumption that there's always going to be the same number of blocks written?
When you say "one copy used across all variables and timesteps", is the not per-variable? Or is that one copy of these values for each variable and for each timestep?

[gqian-coder]

Yes for min and max from bpls. 96 is the number of blocks used for mesh partition, i.e., the whole unstructured meshes are partitioned into 96 regions. This number is fixed after metadata generation but is subject to changes of mesh partitions. "one copy used across all variables and timesteps" --- the metadata is one copy of these values for each variable and for each timestep. If we store the metadata as a separate bp file under the same directory as the data.bp, I just need ADIOS to pass the Engine object, i.e., data filename, into the Operator, so I can read the metadata.bp inside the Operator for data decompression. Qian

…

On Thu, Jun 5, 2025 at 3:33 PM Greg Eisenhauer ***@***.***> wrote: OK, more, maybe stupid questions... I'm assuming that the "x / y" stuff is just min/max from bpls, but let me know if that has some other meaning I should be aware of. So, if 96 is the number of blocks (for this variable and on this timestep?), then presumably this can change? Or is there some assumption that there's always going to be the same number of blocks written? When you say "one copy used across all variables and timesteps", is the not per-variable? Or is that one copy of these values for each variable and for each timestep? — Reply to this email directly, view it on GitHub <#4542 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AMOUPYPDUEX4HMCFU6AX55T3CCLRTAVCNFSM6AAAAAB6S4USU2VHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTGMZYGM2DKNQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[pnorbert]

It is a new mesh generated from the original mesh, which is partitioned to one block per process.
The new mesh applies to all compressed variables and it does not change over time.
Therefore this data is generated only once on each process and is reused by subsequent calls to the operator.

On the read side, the operator needs this new mesh, again read into memory once and reused for the all variables and steps.

The problem is where and how to write this new mesh so that it is moved along with the data file and the reader operator surely finds it.

[pnorbert]

Therefore I think the simplest solution would be to give the output file name to the operator, so it can create another, independent adios output under that file name (directory). So the simplest question is, does the operator have access to the Engine->m_Name somehow. Of course, this is a hack, so a more elegant solution would be welcome.