Recommended ADIOS2 XML configuration for Lustre system with only 32 OSTs (LUMI-G)

[sangjun11316]

My application produces a ~25 TB ADIOS2 (2.10.2) checkpoint in a test run with 512 GPU nodes.

With the same settings (ADIOS2 at defaults), checkpoint writing is much slower on LUMI-G than Frontier. Both systems share similar architecture with AMD MI250X GPUs so the computing performance is more or less identical, but the key differences regarding IO seem to be the Lustre OST count. lfs df -h . on Scratch reports 1349 OSTs on Frontier and only 32 on LUMI-G.

ADIOS2 by default tries to write 1 subfile per node (so 512 subfiles for 512 nodes), which likely overloads LUMI-G's 32 OSTs. Changing striping size (from default 1 MB) to 1G or 2G didn't help much.

I then tried to reduced the number of subfiles wirtten by ADIOS2 setting SubStreams to 32, but the checkpoint still comes out with 512 subfiles rather than 32.

Could I get some advice on:

1. Why my XML below fails to produce 32 subfiles?
2. Are there recommended ADIOS2 configurations for multi-TB outputs on systems with few OSTs like LUMI-G?
3. Any Lustre best practices (like striping) we should pair with the ADIOS2 settings?

I did contact LUMI-G support, but understandably it seems that they can't offer application-specific tuning so I'll need to determine the optimal settings myself.

  1 <?xml version="1.0"?>
  2 <!-- XGC Config XML file for ADIOS2 -->
  3 
  4 <adios-config>
  5 
  6     <!--==========================================
  7                     Configuration for the Particle Writer
  8         ==========================================-->
  9     <!--
 10    <io name="particles">
 11         <engine type="BPFile">
 12             <parameter key="collectivemetadata" value="off"/>
 13             <parameter key="Profile" value="Off"/>
 14         </engine>
 15     </io>
 16 
 17     <io name="restart">
 18         <engine type="BPFile">
 19             <parameter key="collectivemetadata" value="off"/>
 20             <parameter key="Profile" value="Off"/>
 21             <parameter key="SubStreams" value="1024"/>
 22         </engine>
 23     </io>
 24 
 25     <io name="restartf0">
 26         <engine type="BPFile">
 27             <parameter key="collectivemetadata" value="off"/>
 28             <parameter key="Profile" value="Off"/>
 29         </engine>
 30     </io>
 31     -->
 32 
 33     <io name="restart">
 34         <engine type="BPFile">
 35             <parameter key="collectivemetadata" value="off"/>
 36             <parameter key="Profile" value="Off"/>
 37             <parameter key="SubStreams" value="32"/>
 38         </engine>
 39     </io>
 40 
 41 </adios-config>

[pnorbert]

The parameter name to pick the number of files for BP5 (which BPFile now points to) is NumSubFiles, not SubStreams.

You can also change the aggregation AggregationType to EveryoneWritesSerial, and pick NumAggregators arbitrarily (NumSubFiles equals NumAggregators for this aggregator)

As for LUMI-G, maybe @gregorweiss could advise for a good settings? He ran very big simulations there.

[pnorbert]

See notes on aggregation parameters

[gregorweiss]

Sorry, I don't have experience with running in LUMI-G but on the old HAWK system at HLRS, which also utilized a Lustre file system. As you already figured, choosing the number of files is crucial. Regarding your stripe size you'd have to experiment while considering the actual size of individual IO operations could be a good starting point. I think you are on the right track tuning these parameters. Optimization is left to experimentation with those.

[pnorbert]

1. According to https://lumi-supercomputer.github.io/LUMI-training-materials/2day-20240502/08_Lustre/?utm_source=chatgpt.com#storage-areas a 32OST Lustre partition provides 240GB/s bandwidth. An application is lucky to get half of that. Do you know your current performance?

2. Can you use the Lumi-F Project flash space? You need to request bigger than 2TB space first, of course. It should give you much better bandwidth (2TB/s according to the document).

3. I would set striping count to 1 on the restart_dir/ , adios writing multiple files at once should utilize all OSTs, overlapping them is not beneficial.

4. In my experience, stripe size changes have no effect on adios performance on lustre, not even in general, but of course entirely irrelevant for stripe count 1.

[pnorbert]

@gregorweiss My mistake, I thought exaFOAM was running on Lumi at 500k cores but indeed that was Hawk.
But that machine's Lustre is even slower isn't it, and only in a single partition? So how did you scale 500k writers to that file system?

[sangjun11316]

Thank you @pnorbert, the best I got so far (which was with the default Lustre & ADIOS2 setting) takes ~400 seconds to write a 25 TB checkpoint. So I am at ~ 25% of the full 240 GB/s bandwidth. The striping count has been kept to 1, it seems that I have to consider the Flash space..

[sklasky]

@sangjun11316 . Could you please 1. Tell us the time it takes to write for several outputs. I am asking because of I/O variability, and I would like to have some statistics to know the best and worst time you get. How often are you writing a checkpoint (i.e. once an hour, once every 2 hours, ...)? You could use ADIOS staging (using SST) to move the data from your compute nodes to some other nodes, but this will involve some tricks to keep the number of staging nodes left to buffer the data low. @eisenhauer Can we move some of the data to staging, then write it, and then move more and write again?

[sangjun11316]

@sklasky, the checkpoint frequency is an input parameter. In principle, I can write very few checkpoints (perhaps one per each job), but the trade-off is that a failure before a checkpoint means rerunning a long segment. Of course, I have other outputs than checkpoints, which I kept disabled during this test period due to the limited allocations. But I expect those will not be as problematic as the checkpoints as they are much smaller.

I'm afraid I don't have much statistics due to the limited allocation. I only have one checkpoint sample per tested Lustre/ADIOS2 setting combination. The best was from the default Lustre(count 1, size 1 MB)/ADIOS2 settings taking ~ 400 seconds for 25 TB checkpoint. The worst was when I set ADIOS2 to write 32 subfiles (= # OSTs in LUMI) by EveryoneWritesSerial and NumAggregators=32, taking over 20 minutes.

At the moment, the remaining allocation will only allow me to write one or two checkpoints, so I guess the best option now is just to try out Flash storage. After the main allocation starts, I would be able to do a more systematic sweep.

[gregorweiss]

@gregorweiss My mistake, I thought exaFOAM was running on Lumi at 500k cores but indeed that was Hawk.
But that machine's Lustre is even slower isn't it, and only in a single partition? So how did you scale 500k writers to that file system?

We did it with one-file-per-node policy. That's when we got peak of the filesystem performance, while using the whole cluster. (Also we tried to aggregate with respect to the network topology, which was something like 64 nodes that shared an uplink to the FS. But that didn't improve nor worsened the performance.)

But the amount of data was far from 25TB, which seems to be the challenge here. In our case, it only was roughly 64KB per process, so only on the order of tens of GB at most close to a hundred GB.

In sum, it pretty much worked out-of-the-box. Sorry, if I can't help with any good hints here.

[sangjun11316]

@pnorbert @gregorweiss @sklasky
I'm glad to report that the IO performance on LUMI-F filesystem is much better than the slower LUMI-P.
On LUMI-P, it took 400 and 800 seconds for 512 and 1024 nodes jobs (weak scaling, problem size proportional to # nodes).
On LUMI-F, it took 70 seconds for both jobs (with default Lustre and ADIOS2 settings)

It is still slower than Frontier (~50 seconds), but not as problematic as before. Thank you for suggesting to use LUMI-F :)

[sangjun11316]

This is comparison of LUMI-P/F and Frontier with default ADIOS2 settings. Added for recording.