PIXIE3D ADIOS2 restart aborts due to unreasonable amount of memory required (follow up to issue #2042)

[lchacon-LANL]

Describe the bug
ADIOS2 restart (opening bp file read-only) in PIXIE3D overwhelms memory and aborts with error:

std::bad_alloc
adios2_open_new_comm

Memory per MPI rank at fresh start is ~0.6 GB. Memory per MPI rank at restart (after ~4000 time steps on a 64x128x64 mesh) is > 4 GB, independently of the number of MPI ranks. 32 MPI ranks overwhelms node memory (128 GB) and code aborts.

Memory with fresh start with 16 ranks:

Memory at restart (from BP file with ~4000 time slices) with 16 ranks:

Memory at restart per MPI rank seems independent of the number of MPI ranks. See for 8 ranks:

To Reproduce
I am at latest master-branch commit ff8326c. Very hard to provide "simple" example as both PIXIE3D and (very large) PIXIE3D BP restart file are required. Seems like problem is related to the number of time steps in restart file, and therefore providing a smaller restart file will not be very helpful. I am happy to provide access to both upon request.

Expected behavior
Memory at restart should be inversely proportional to the number of MPI ranks to avoid overwhelming node memory.

Desktop (please complete the following information):

• OS/Platform: Linux ba170.localdomain 3.10.0-1160.45.1.1chaos.ch6.x86_64
• Build: gcc 9.4.0, openmpi 3.1.6, shared libs

Additional context
Seems like problem is related to the number of time steps stored in the restart file, as PIXIE3D has no issue restarting from BP files with the same underlying 3D mesh but fewer time steps.

Following up
Was the issue fixed? Please report back.

[williamfgc]

@lchacon-LANL thanks for reporting this.

Memory at restart per MPI rank seems independent of the number of MPI ranks

Yes, metadata is broadcasted from rank 0 to all processes and "deserialized" into more memory expensive structures (binary trees) when reading in parallel. On a memory distributed system this shouldn't be an issue, but it scales up linearly on a single node as you're showing (it would still be interesting to find out the small differences).

Memory at restart should be inversely proportional to the number of MPI ranks to avoid overwhelming node memory.

At some point we discussed a "metadata steps view" implementation when opening a file, but it was never implemented. A workaround is to use rank 0 for all I/O and send data to other ranks as needed, but it would make your workflow more convoluted and would add network cost in memory distributed systems. I think @pnorbert also implemented a streaming (step-by-step) approach, I don't know if that fits your use-case. Hope it helps.

[lchacon-LANL]

@williamfgc,

I performed the reported tests on a single node to illustrate the issue, but this is not the way we are running PIXIE3D. In fact, this particular 3D run was originally performed on 512 cores using 16 nodes with 32 cores/node in a local LANL cluster. This run cannot be restarted anymore on 16 nodes because of the memory issue I reported. To restart, we would need to double the number of nodes and halve the number of cores per node (which is our interim solution, but this is undesirable for obvious reasons).

Am I to understand that this is more an ADIOS2 "feature" than a bug?

[williamfgc]

@lchacon-LANL thanks for the additional context and for narrowing down the issue. Yes, performance in adios2 is obtained by trading memory with the application (it is "faster" to pull from disk once and broadcast the metadata to all ranks). Since your use case becomes memory-bound due to the metadata-size-per-process, it hits an upper limit rather quickly. Perhaps, restricting the initial metadata view would be justified by your use case. That would have be a new feature, but only supported by file engines. @pnorbert any plans for this with BP5? It shouldn't be too invasive in other BP engines, but still requires some effort. Hope it helps.

[lchacon-LANL]

@williamfgc Thanks for your reply. I am currently using file engines exclusively (BP4, to be precise). It would not be hard at all to upgrade to BP5 for new runs. @pnorbert, please advise.

[pnorbert]

We don't have a solution for this inside adios right now but I made a quick fix. This script can hack the metadata of a BP4 output so that only a single step is visible.

$ cat ~/bin/adios2_select_step.sh
#!/bin/bash

if [ -z "$2" ]; then
    echo "Usage: adios2_select_step.sh <bp4file> <step>"
    echo " bp4file : BP4 format"
    echo " step    : 0..N-1, where file has N steps"
    exit 1
fi

FILE=$1
STEP=$2

if [ ! -d $FILE -o ! -f $FILE/md.idx  -o ! -f $FILE/md.0 ]; then
    echo "Error: file is not a valid BP4 directory"
    exit 2
fi

# First time call, save the full metadata
if [ ! -f $FILE/md.idx.orig ]; then
    cp -p $FILE/md.idx $FILE/md.idx.orig
fi

if [ ! -f $FILE/md.0.orig ]; then
    cp -p $FILE/md.0 $FILE/md.0.orig
fi

IDXSIZE=`stat $FILE/md.idx.orig | awk '/Size/{print $2}'`
((NSTEPS = IDXSIZE/64 -1));
echo "Found $NSTEPS steps in $FILE"

if (( $STEP >= $NSTEPS )); then
    echo "Error: step $STEP is not available"
    exit 3
fi


#
# Work from saved full metadata and create a subset
#
cd $FILE

# copy md.idx and md.0 header (64 bytes)
dd if=md.idx.orig of=md.idx bs=64 count=1 status=none
dd if=md.0.orig of=md.0 bs=64 count=1 status=none

# get offsets in md.0 for record STEP
((SKIP = STEP*64+64))
RECORD=`od -td8 md.idx.orig -Ad -v -j $SKIP -w64 | head -1`
#echo "record = $RECORD"
PGPTR=`echo $RECORD | awk '{print $4}'`
VARPTR=`echo $RECORD | awk '{print $5}'`
ATTRPTR=`echo $RECORD | awk '{print $6}'`
ENDPTR=`echo $RECORD | awk '{print $7}'`
TIMESTAMP=`echo $RECORD | awk '{print $8}'`
echo "Metadata offsets for step $STEP in md.0: $PGPTR..$ENDPTR"

# copy metadata (from offset PGPTR to 64)
((COUNT = ENDPTR - PGPTR))
dd if=md.0.orig of=md.0 bs=1 count=$COUNT skip=$PGPTR seek=64 status=none

function write64bit () {
    local OUTPUT=$1
    local N64=$2
    local v=`awk -v n=$N64 'BEGIN{printf "%016X", n;}'`;
    echo -n -e "\\x${v:14:2}\\x${v:12:2}\\x${v:10:2}\\x${v:8:2}\\x${v:6:2}\\x${v:4:2}\\x${v:2:2}\\x${v:0:2}" >> $OUTPUT
}

# Create record with new offsets (PGPTR -> 64)
# copy md.idx record STEP (to bytes 64..127)
((VARPTR = VARPTR - PGPTR + 64))
((ATTRPTR = ATTRPTR - PGPTR + 64))
((ENDPTR = ENDPTR - PGPTR + 64))
write64bit md.idx 1
write64bit md.idx 0
write64bit md.idx 64
write64bit md.idx $VARPTR
write64bit md.idx $ATTRPTR
write64bit md.idx $ENDPTR
write64bit md.idx $TIMESTAMP
write64bit md.idx 0

cd -

It makes a copy of the full metadata so you can call this script repeatedly to make another step visible, and also to recover the original status.

cd restart.bp
cp -p md.idx.orig md.idx
cp -p md.0.orig md.0
cd -

Here is how it works for my test file with 3 steps:

$ bpls -la test.bp/ -d -n 8
  float    r32   3*{8} = 0 / 13.1
    (0,0)    2.1 3.1 4.1 5.1 6.1 7.1 8.1 9.1
    (1,0)    4.1 5.1 6.1 7.1 8.1 9.1 10.1 11.1
    (2,0)    6.1 7.1 8.1 9.1 10.1 11.1 12.1 13.1

$ ~/bin/adios2_select_step.sh test.bp 0
Found 3 steps in test.bp
Metadata offsets for step 0 in md.0: 64..262
/home/adios/ADIOS2/build.debug

$ bpls -la test.bp/ -d -n 8
  float    r32   {8} = 2.1 / 9.1
    (0)    2.1 3.1 4.1 5.1 6.1 7.1 8.1 9.1

$ ~/bin/adios2_select_step.sh test.bp 1
Found 3 steps in test.bp
Metadata offsets for step 1 in md.0: 262..521
/home/adios/ADIOS2/build.debug

$ bpls -la test.bp/ -d -n 8
  float    r32   {8} = 0 / 11.1
    (0)    4.1 5.1 6.1 7.1 8.1 9.1 10.1 11.1

$ ~/bin/adios2_select_step.sh test.bp 2
Found 3 steps in test.bp
Metadata offsets for step 2 in md.0: 521..780
/home/adios/ADIOS2/build.debug

$ ./bin/bpls -la test.bp/ -d -n 8
  float    r32   {8} = 0 / 13.1
    (0)    6.1 7.1 8.1 9.1 10.1 11.1 12.1 13.1

$ ~/bin/adios2_select_step.sh test.bp 3
Found 3 steps in test.bp
Error: step 3 is not available

$ cd test.bp/
$ cp -p md.idx.orig md.idx
$ cp -p md.0.orig md.0
$ cd -
$ bpls -la test.bp/ -d -n 8
  float    r32   3*{8} = 0 / 13.1
    (0,0)    2.1 3.1 4.1 5.1 6.1 7.1 8.1 9.1
    (1,0)    4.1 5.1 6.1 7.1 8.1 9.1 10.1 11.1
    (2,0)    6.1 7.1 8.1 9.1 10.1 11.1 12.1 13.1

[lchacon-LANL]

@pnorbert Thanks for the hack! I will test momentarily.

I do not understand why the metadata file md.0 is 4GB large, though. For 4000 time steps, this means 4MB per step, which seems like a lot... Is there something I can do on my end to reduce the metadata memory footprint per tilmestep?

[pnorbert]

send me a bpls -latD output...

[williamfgc]

Is there something I can do on my end to reduce the metadata memory footprint per tilmestep?

@lchacon-LANL if the data range (min/max) per block is not needed at read time (or with bpls), you can set at write time:

in code: io.SetParameter("StatsLevel", "0"); savings depend on the frequency of this operation from your data (number of Put using arrays). If you're using XML or YAML config file it's just another key/value Parameter. It would be interesting if it results in any savings if appropriate.

[lchacon-LANL]

Thanks, @williamfgc. I will give explore this option.

@pnorbert I have sent you the requested output by LANL file transfer (200MB file compressed, 1.3GB uncompressed). I have limited output for arrays to one element per rank. Let me know if you need anything else.

[pnorbert]

The BP file looks okay to me, unfortunately metadata grows linearly with the number of blocks (variables * processes * blocks-per-process * steps).

I don't understand though why the metadata is 4MB per step, I expect about 1MB per step for your file. Can you print

$ od -td8 -w64 record-2.5.0.bp/md.idx

Just the first few lines is enough, no need for all steps

[lchacon-LANL]

@pnorbert Here are the first few lines of the command

od -td8 -w64 record-2.5.0.bp/md.idx

0000000 5783234707332875329 2319404614039008800 7013266221191949897 6646882 285873026381106 0 0 0
0000100 1 0 64 15440 977272 978260 1639825158 0
0000200 2 0 978260 993636 1955468 1955480 1639825158 0
0000300 3 0 1955480 1970856 2932688 2932700 1639825193 0
0000400 4 0 2932700 2948076 3909908 3909920 1639825247 0
0000500 5 0 3909920 3925296 4887128 4887140 1639825299 0
0000600 6 0 4887140 4902516 5864348 5864360 1639825380 0
0000700 7 0 5864360 5879736 6841568 6841580 1639825460 0
0001000 8 0 6841580 6856956 7818788 7818800 1639825545 0
0001100 9 0 7818800 7834176 8796008 8796020 1639825643 0

[pnorbert]

Thanks, so the metadata is indeed ~1MB per step, which I expected from the block list. So the memory runs out when the BP4 engine on every process parses that metadata and builds its own data structures. The only remedy is to cut the number of steps within one output, or use that script above to temporarily limit the access to a single step.

BP5 is expected to decrease the metadata size a bit, and use less memory when processing it, but eventually the same problem holds.

[lchacon-LANL]

Thanks, Norbert. Noted.