v2.11.0

[vicentebolea]

What's new in 2.11?

This is a major release with new features and lots of bug fixes. The main
highlights include enhanced derived variables, remote data access, GPU
improvements, campaign management overhaul, and advanced compression operators.

Summary

Derived Variables

Support for reader-side computed variables with expression evaluation. This
release adds comprehensive functionality including new mathematical operations
(trigonometric functions, multiplication, division, power) and the
ability to apply derived variables to aggregated arrays.

Remote Data Access Infrastructure

Implementation of XRootD and remote server capabilities enabling distributed
file access. This includes support for remote streaming, metadata caching, and
optimized query handling across networked systems.

GPU and Memory Space Improvements

Enhanced GPU backend support with memory selection adjustments, Kokkos
integration improvements, and bindings for cuPy pointers and torch tensors.
Addresses layout mismatches between user code and device memory.

Campaign Management System Overhaul

Significant updates to campaign tracking using SQLite databases replacing JSON
files, configuration file support, and attribute handling. Includes multi-host
support and improved metadata organization.

Compression and Data Format Advances

New operators including BigWhoop compression and Complex MGARD support. ZFP now
supports 4D arrays, and improved MGARD efficiency through zstd compression of
lossless data portions.

OpenSSF Best Practices

ADIOS2 now adopts OpenSSF (Open Source Security Foundation) best practices
including enhanced security measures, automated vulnerability scanning, and
adherence to industry-standard development practices.

What's Changed

• python,cmake: update version script to support rc by @vicentebolea in python,cmake: update version script to support rc #3979
• BP5internal - Switch to name based record lookup by @eisenhauer in BP5internal - Switch to name based record lookup #3981
• Fix compilation errors on strict compiler by @eisenhauer in Fix compilation errors on strict compiler #3983
• Update the tutorial with links to external tutorials by @anagainaru in Update the tutorial with links to external tutorials #3978
• ASAN cleanup by @eisenhauer in ASAN cleanup #3982
• Fix read request length on BP5 Local arrays with operators by @eisenhauer in Fix read request length on BP5 Local arrays with operators #3989
• Backport master 3980 3994 by @pnorbert in Backport master 3980 3994 #3996
• Backport 3984 to master: Python API: Adios() new argument 'config_file', Stream() new argument… by @pnorbert in Backport 3984 to master: Python API: Adios() new argument 'config_file', Stream() new argument… #3999
• update CODEOWNERS by @vicentebolea in update CODEOWNERS #4003
• update readthedocs deps by @vicentebolea in update readthedocs deps #4001
• Merge upstream dill changes with master by @eisenhauer in Merge upstream dill changes with master #4006
• Implementing sqlite database instead of json file. Writing part. by @dmitry-ganyushin in Implementing sqlite database instead of json file. Writing part. #4000
• A simpler way to deal with the mismatching layouts between the user code and the memory space by @anagainaru in A simpler way to deal with the mismatching layouts between the user code and the memory space #3976
• fix BP5 parameter parsing for size values with units by @pnorbert in fix BP5 parameter parsing for size values with units #4012
• Merge 3769 to master by @pnorbert in Merge 3769 to master #4011
• Replace the only PERFSTUBS_SCOPED_TIMER_FUNC() call in BP5 to get rid… by @pnorbert in Replace the only PERFSTUBS_SCOPED_TIMER_FUNC() call in BP5 to get rid… #4013
• Python updates by @pnorbert in Python updates #4014
• FFS upstream by @eisenhauer in FFS upstream #4016
• Posix transport should fail if you try to read past EOF by @eisenhauer in Posix transport should fail if you try to read past EOF #3997
• Make POSIX transport work on Windows, default to it by @eisenhauer in Make POSIX transport work on Windows, default to it #4023
• Campaign fix by @dmitry-ganyushin in Campaign fix #4024
• readthedocs: enable query docs by @vicentebolea in readthedocs: enable query docs #4030
• Don't force initialization for buffers whose contents we will immediately destroy by @eisenhauer in Don't force initialization for buffers whose contents we will immediately destroy #4031
• fix IO.available_attributes(), remove a debug print from Stream, fix … by @pnorbert in fix IO.available_attributes(), remove a debug print from Stream, fix … #4021
• campaign management: update sqlite database after another run by @dmitry-ganyushin in campaign management: update sqlite database after another run #4038
• Fixed campaign python script. by @dmitry-ganyushin in Fixed campaign python script. #4040
• Fix Windows Testing build by @eisenhauer in Fix Windows Testing build #4027
• Tweak POSIX fail on EOF to kill heisenbug by @eisenhauer in Tweak POSIX fail on EOF to kill heisenbug #4035
• ci: add macos static build by @vicentebolea in ci: add macos static build #4041
• Python write scalars by @pnorbert in Python write scalars #4036
• Add a static windows build to CI by @eisenhauer in Add a static windows build to CI #4037
• Add arguments to IO.AvailableAttributes() and io.available_attributes… by @pnorbert in Add arguments to IO.AvailableAttributes() and io.available_attributes… #4043
• Return a 0-dim numpy array of size 1 when reading a scalar variable … by @pnorbert in Return a 0-dim numpy array of size 1 when reading a scalar variable … #4044
• Add timeout parameter to stream.steps(). Necessary for controlling ti… by @pnorbert in Add timeout parameter to stream.steps(). Necessary for controlling ti… #4045
• blosc2: do not export blosc2 lib by @vicentebolea in blosc2: do not export blosc2 lib #4055
• Fix segfault when asking for BlocksInfo for a variable name that does… by @pnorbert in Fix segfault when asking for BlocksInfo for a variable name that does… #4047
• Reading of campaign configuration parameters from ~/.config/adios2 by @dmitry-ganyushin in Reading of campaign configuration parameters from ~/.config/adios2 #4048
• do not flush io and adios in read mode. BP5 reader does not like it. by @pnorbert in do not flush io and adios in read mode. BP5 reader does not like it. #4065
• Allow plugin operators to take advantage of the estimated size API by @rupertnash in Allow plugin operators to take advantage of the estimated size API #4046
• Add minmax and shape functions to CampaignReader, so that per-block i… by @pnorbert in Add minmax and shape functions to CampaignReader, so that per-block i… #4067
• fixes to still be able to build with gcc 4.8.2. by @pnorbert in fixes to still be able to build with gcc 4.8.2.  #4063
• added support to read back from H5T_STRING VARIABLES by @guj in added support to read back from H5T_STRING VARIABLES #4060
• Using the correct flag to detect the CUDA backend in ZFP by @anagainaru in Using the correct flag to detect the CUDA backend in ZFP #4066
• Fix Reord to use MinBlocksInfo where appropriate by @eisenhauer in Fix Reord to use MinBlocksInfo where appropriate #4071
• • Restructure python API doc in separate main topic, add working exam… by @pnorbert in - Restructure python API doc in separate main topic, add working exam… #4075
• C bindings for the memory space related functions by @anagainaru in C bindings for the memory space related functions #4074
• Actually look at return value for read in remote_server by @eisenhauer in Actually look at return value for read in remote_server #4078
• Fortran bindings for memory space related functions by @anagainaru in Fortran bindings for memory space related functions #4077
• Raise an exception if remote open fails by @eisenhauer in Raise an exception if remote open fails #4069
• Python: add the same treatment to attributes as to variables before: … by @pnorbert in Python: add the same treatment to attributes as to variables before: … #4076
• Update documentation for 2.10 changes to the GPU-backend by @anagainaru in Update documentation for 2.10 changes to the GPU-backend #4083
• BlockIndex.Evaluate() made branch based on BP5 vs BP4. To support Cam… by @pnorbert in BlockIndex.Evaluate() made branch based on BP5 vs BP4. To support Cam… #4084
• Fix typo in the links to external tutorials by @anagainaru in Fix typo in the links to external tutorials  #4086
• adios2_define_derived_variable C/Fortran API. C is compiled condition… by @pnorbert in adios2_define_derived_variable C/Fortran API. C is compiled condition… #4088
• Add setup for Aurora (load adios2 as e4s package) by @pnorbert in Add setup for Aurora (load adios2 as e4s package) #4089
• Do not create adios-campaign/ unless there is something to record by @pnorbert in Do not create adios-campaign/ unless there is something to record #4096
• Bison 3.8 Parser by @lizdulac in Bison 3.8 Parser #4062
• WIP: Make Fortran tests fail with a non-zero exit code by @eisenhauer in WIP: Make Fortran tests fail with a non-zero exit code #4097
• Fix typo in Fortran API docs by @JamieJQuinn in Fix typo in Fortran API docs #4102
• ci: add ccache job summary by @vicentebolea in ci: add ccache job summary #4101
• Fix static blosc2 build by @vicentebolea in Fix static blosc2 build #4093
• Don't run derived test in MPI mode, it's not written for that by @eisenhauer in Don't run derived test in MPI mode, it's not written for that #4104
• Pull in DILL Upstream that avoids libFFI except on demand. by @eisenhauer in Pull in DILL Upstream that avoids libFFI except on demand. #4091
• Update campaign manager script to handle config file, ... by @pnorbert in Update campaign manager script to handle config file, ... #4111
• Use EXPECT_EQ instead of assert() in tests by @eisenhauer in Use EXPECT_EQ instead of assert() in tests #4108
• Fix warnings that happen in Release build by @eisenhauer in Fix warnings that happen in Release build #4113
• EVpath upstream to make NO_RDMA more robust by @eisenhauer in EVpath upstream to make NO_RDMA more robust #4116
• Different names for MPI and Serial tests by @eisenhauer in Different names for MPI and Serial tests #4118
• cmake: add sqlite3 and zlib dep in adios2 cmake pkg by @vicentebolea in cmake: add sqlite3 and zlib dep in adios2 cmake pkg #4120
• Add support for user options in ~/.config/adios2/adios2.yaml by @pnorbert in Add support for user options in ~/.config/adios2/adios2.yaml #4119
• Updates to campaign management by @pnorbert in Updates to campaign management #4122
• Add attribute support to campaign by @pnorbert in Add attribute support to campaign #4124
• DataPlane Configuration changes by @eisenhauer in DataPlane Configuration changes #4121
• Add -f file1 [file2...] option to process adios files from a list ins… by @pnorbert in Add -f file1 [file2...] option to process adios files from a list ins… #4126
• Add the CURL function to derived variables by @lizdulac in Add the CURL function to derived variables #4114
• Setting the derived variable support OFF by default by @anagainaru in Setting the derived variable support OFF by default #4127
• Fixes for FreeBSD, including upstream by @eisenhauer in Fixes for FreeBSD, including upstream #4138
• Compile FileHTTP on Windows by @eisenhauer in Compile FileHTTP on Windows #4132
• Option in bpls to show the expression string for derived variables by @anagainaru in Option in bpls to show the expression string for derived variables #4145
• pugixml,cmake: fix linkage when pugixml is external by @vicentebolea in pugixml,cmake: fix linkage when pugixml is external #4142
• ReadModeFlatten by @eisenhauer in ReadModeFlatten #4147
• Minor docs tweaks by @eisenhauer in Minor docs tweaks #4153
• add line back to gs-pipeline.py by @caitlinross in add line back to gs-pipeline.py #4155
• Upstream EVPath and SST updates to make the Visual Studio compiler happy. by @eisenhauer in Upstream EVPath and SST updates to make the Visual Studio compiler happy. #4154
• adios2-config: set the policy directly by @mathstuf in adios2-config: set the policy directly #4157
• Adding hooks to gather performance for derived variables by @anagainaru in Adding hooks to gather performance for derived variables #4161
• Remove special ReadFlattenSteps Mode, switch to control by writer-sid… by @eisenhauer in Remove special ReadFlattenSteps Mode, switch to control by writer-sid… #4160
• Reorg Remote as base class in preparation for more remote subclasses by @eisenhauer in Reorg Remote as base class in preparation for more remote subclasses #4162
• Update CURL implementation to allow the compiler to optimize the code by @anagainaru in Update CURL implementation to allow the compiler to optimize the code #4163
• Dmitry's Xrootd implementation, with minor updates by @eisenhauer in Dmitry's Xrootd implementation, with minor updates #4164
• Typo in documentation by @anagainaru in Typo in documentation #4166
• Basic Xrootd remote fixes by @eisenhauer in Basic Xrootd remote fixes #4167
• Compress the expression string into one line for bpls by @lizdulac in Compress the expression string into one line for bpls #4158
• Add server-side ADIOS file pools to Xrootd by @eisenhauer in Add server-side ADIOS file pools to Xrootd #4170
• Add API call to reset memory selection by @franzpoeschel in Add API call to reset memory selection #4169
• Close more FDs because XRootd plays games in library init by @eisenhauer in Close more FDs because XRootd plays games in library init #4178
• EVPath Upstream by @eisenhauer in EVPath Upstream #4181
• Xrootd Server Cleanup by @eisenhauer in Xrootd Server Cleanup #4173
• Kill outdated reference to bindings.Mode.ReadFlattenSteps by @eisenhauer in Kill outdated reference to bindings.Mode.ReadFlattenSteps #4185
• Add macro defines for BP3, BP4 and BP5 by @franzpoeschel in Add macro defines for BP3, BP4 and BP5 #4191
• cmake: add ADIOS2_LIBADIOS_MODE flag by @vicentebolea in cmake: add ADIOS2_LIBADIOS_MODE flag #4196
• Add random port/report port in background, switch to malloc for data buffers by @eisenhauer in Add random port/report port in background, switch to malloc for data buffers #4183
• Patch for xrootd with derived variables by @anagainaru in Patch for xrootd with derived variables #4200
• FFS tweaks by @eisenhauer in FFS tweaks #4203
• Dill upstream in master by @eisenhauer in Dill upstream in master #4197
• Read profiler fix by @guj in Read profiler fix #4206
• Do Basename for storing reader profiling by @eisenhauer in Do Basename for storing reader profiling #4204
• Limit the exposed symbols related to derived variables by @anagainaru in Limit the exposed symbols related to derived variables #4207
• include to be able to compile std::function<> with GCC c… by @pnorbert in include <functional> to be able to compile std::function<> with GCC c… #4209
• ci: added rocm gha build by @vicentebolea in ci: added rocm gha build #4213
• print and flush last line in remote server to ensure it shows up on t… by @pnorbert in print and flush last line in remote server to ensure it shows up on t… #4215
• Backport PR 4189 and 4174 to master by @pnorbert in Backport PR 4189 and 4174 to master #4217
• ci: add sync script workflow by @vicentebolea in ci: add sync script workflow #4224
• Fix some DimsArray issues and add unit testing by @eisenhauer in Fix some DimsArray issues and add unit testing #4221
• [Bug fix] Switch from ROCM_VERSION_MAJOR to HIP_VERSION_MAJOR by @anagainaru in [Bug fix] Switch from ROCM_VERSION_MAJOR to HIP_VERSION_MAJOR #4222
• ci: remove macos11 build by @vicentebolea in ci: remove macos11 build #4229
• Extend Expression Library: Subtract by @lizdulac in Extend Expression Library: Subtract #4227
• ADIOS2 User Side KVCache to accelerate the remote query access. by @Change72 in ADIOS2 User Side KVCache to accelerate the remote query access. #4210
• Fixing C++ and Fortran example to correctly read in files (not master branch this time) by @adrianjhpc in Fixing C++ and Fortran example to correctly read in files (not master branch this time) #4239
• ci: add frontier pipelines by @vicentebolea in ci: add frontier pipelines #4223
• ci: do not test frontier commit prior to frontier support by @vicentebolea in ci: do not test frontier commit prior to frontier support #4242
• Update ReadMe.md by @vicentebolea in Update ReadMe.md #4243
• ATL and EVPath Upstream by @eisenhauer in ATL and EVPath Upstream #4238
• Remove reference to xarray backend by @JamesMcClung in Remove reference to xarray backend #4235
• First cut at reader-side derived variables by @eisenhauer in First cut at reader-side derived variables #4228
• Update pybind11 to 2.13.1 by @vicentebolea in Update pybind11 to 2.13.1  #4245
• Connect to tunnel server to launch remote server by @pnorbert in Connect to tunnel server to launch remote server #4234
• Fix VarShape() for Joined Arrays by @eisenhauer in Fix VarShape() for Joined Arrays #4246
• [Bug fix] Read side computation of derived variables should not be triggered when data is stored by @anagainaru in [Bug fix] Read side computation of derived variables should not be triggered when data is stored #4247
• change to ADIOS2_USE_KVCACHE from ADIOS2_USE_Cache to be consistent i… by @pnorbert in change to ADIOS2_USE_KVCACHE from ADIOS2_USE_Cache to be consistent i… #4250
• cmake,ci: disable kvcache by @vicentebolea in cmake,ci: disable kvcache #4256
• cmake: use MPI_CXX for targets of CXX sources by @vicentebolea in cmake: use MPI_CXX for targets of CXX sources #4255
• Fix errors and warning related to derived variables by @anagainaru in Fix errors and warning related to derived variables #4252
• Don't print kvcache connection closed message when it was not opened. by @pnorbert in Don't print kvcache connection closed message when it was not opened. #4251
• Fix CampaignReader when a campaign has multiple hosts. Directory indi… by @pnorbert in Fix CampaignReader when a campaign has multiple hosts. Directory indi… #4259
• cmake: only remove werror|wall install test by @vicentebolea in cmake: only remove werror|wall install test #4260
• ci: update gcc11 images to ubuntu22.04 by @vicentebolea in ci: update gcc11 images to ubuntu22.04 #4262
• ci,windows: Use conda for all deps by @vicentebolea in ci,windows: Use conda for all deps #4266
• KVCache ctest and cache key optimization by @Change72 in KVCache ctest and cache key optimization #4267
• Add package matrix to readme by @vicentebolea in Add package matrix to readme #4265
• ci,macos: use conda for all deps by @vicentebolea in ci,macos: use conda for all deps #4269
• Replace lexer.l with unformatted lexer.l by @lizdulac in Replace lexer.l with unformatted lexer.l #4271
• ci,python: enable python in windows builds by @vicentebolea in ci,python: enable python in windows builds #4139
• ci: address gha warnings by @vicentebolea in ci: address gha warnings #4268
• EVpath upstream by @eisenhauer in EVpath upstream #4277
• Add io.define_derived_variable(name, expression, type) function to Py… by @pnorbert in Add io.define_derived_variable(name, expression, type) function to Py… #4275
• Remote SST streaming by @eisenhauer in Remote SST streaming #4280
• FFS upstream by @eisenhauer in FFS upstream #4281
• FFS and ENET upstream by @eisenhauer in FFS and ENET upstream #4286
• FFS include fixup from Upstream by @eisenhauer in FFS include fixup from Upstream #4288
• cmake: dont set version/soversion in interface lib by @vicentebolea in cmake: dont set version/soversion in interface lib #4290
• EVPath upstream for mingw by @eisenhauer in EVPath upstream for mingw #4289
• Fixing parser/lexer related errors by @anagainaru in Fixing parser/lexer related errors #4293
• Relaxing the restriction of the derived variables to have the same type as the operands by @anagainaru in Relaxing the restriction of the derived variables to have the same type as the operands #4291
• ADIOS-level tweaks for mingw by @eisenhauer in ADIOS-level tweaks for mingw #4295
• Rationalize SST_HAVE_MPI_DP by @eisenhauer in Rationalize SST_HAVE_MPI_DP #4298
• Extend Derived Variable Expressions: Multiplication, Division, Pow, Sqrt by @lizdulac in Extend Derived Variable Expressions: Multiplication, Division, Pow, Sqrt #4248
• KVCache use MinBlocksInfo to get start/count in query by @Change72 in KVCache use MinBlocksInfo to get start/count in query #4294
• Documentation and fixing the API for derived variables by @anagainaru in Documentation and fixing the API for derived variables #4301
• Adding the derived variables documentation to the advance section by @anagainaru in Adding the derived variables documentation to the advance section #4302
• Try MacOS gfortran fixup by @eisenhauer in Try MacOS gfortran fixup #4305
• Restore the derived variable default to OFF by @eisenhauer in Restore the derived variable default to OFF #4306
• Uniquify filenames in HeatTransfer tests by @eisenhauer in Uniquify filenames in HeatTransfer tests #4307
• Update the script to build adios with Kokkos for Frontier by @anagainaru in Update the script to build adios with Kokkos for Frontier #4303
• update codeowners by @vicentebolea in update codeowners #4309
• Fixing formatting in the derived documentation by @anagainaru in Fixing formatting in the derived documentation #4310
• Extend Derived Variable Expressions: Trig functions by @lizdulac in Extend Derived Variable Expressions: Trig functions #4282
• Update .gitattributes by @vicentebolea in Update .gitattributes #4314
• Add CTest for XRootd Remote support by @eisenhauer in Add CTest for XRootd Remote support #4199
• Open file with in-memory metadata by @pnorbert in Open file with in-memory metadata  #4315
• Implementing reader-side Get()s for ExpressionString derived variables by @eisenhauer in Implementing reader-side Get()s for ExpressionString derived variables #4249
• ci: fix pypi artifacts by @vicentebolea in ci: fix pypi artifacts #4316
• Introducing aggregated derived operators by @anagainaru in Introducing aggregated derived operators  #4253
• Updating very old tests by @eisenhauer in Updating very old tests #4317
• Tweak readme after test changes by @eisenhauer in Tweak readme after test changes #4322
• ci: only use conda-forge channel by @vicentebolea in ci: only use conda-forge channel #4325
• Extend Expressions: 3D Cross product by @lizdulac in Extend Expressions: 3D Cross product #4319
• cmake: refactor sst mpi vars by @vicentebolea in cmake: refactor sst mpi vars #4299
• Fix in python API: by @pnorbert in Fix in python API:  #4331
• Fix an issue when reading blocks in streaming mode by @pnorbert in Fix an issue when reading blocks in streaming mode #4332
• Basic script to compare two BP files given a tolerance by @anagainaru in Basic script to compare two BP files given a tolerance #4308
• Korvo upstream - enable building without perl installed by @eisenhauer in Korvo upstream - enable building without perl installed #4336
• Look for tunnel server elsewhere if indicated by @eisenhauer in Look for tunnel server elsewhere if indicated #4335
• Compress the lossless part of MGARD data with Zstd. by @pnorbert in Compress the lossless part of MGARD data with Zstd.  #4339
• If there's an exception in GenerateReadRequests, clear state and rethrow by @eisenhauer in If there's an exception in GenerateReadRequests, clear state and rethrow #4338
• Backport bp5dbg parse records and check sizes during it for mmd.0 and md.0. No… #3759 to master. by @pnorbert in Backport #3759 to master. #4340
• Setting the memory space for Spans to Host by @anagainaru in Setting the memory space for Spans to Host #4343
• New API to allow storing only stats for primary variables by @anagainaru in New API to allow storing only stats for primary variables #4328
• Updating the testing for the GPU backend by @anagainaru in Updating the testing for the GPU backend #4345
• Checks and changes in Get(variable, datum) by @eisenhauer in Checks and changes in Get(variable, datum) #4352
• Support for 4D arrays in ZFP compressor by @gregorweiss in Support for 4D arrays in ZFP compressor #4344
• Update the memory selection layout when there is a layout mismatch by @anagainaru in Update the memory selection layout when there is a layout mismatch #4346
• Fix the code that copies wrongly from host to device GPU buffers instead of device to host by @anagainaru in Fix the code that copies wrongly from host to device GPU buffers instead of device to host #4359
• Stop C/TestBPWriteStatsOnly from running with MPI, it's not suitable by @eisenhauer in Stop C/TestBPWriteStatsOnly from running with MPI, it's not suitable #4361
• Add logging to server by @eisenhauer in Add logging to server #4362
• ci(frontier): revert to use Core/24.00 by @vicentebolea in ci(frontier): revert to use Core/24.00 #4364
• Fix missing int64_t definition by @eisenhauer in Fix missing int64_t definition #4365
• Fix the Kokkos testing compilation error on Frontier by @anagainaru in Fix the Kokkos testing compilation error on Frontier #4366
• Replacing DefApps with Core/24.00 for the frontier-cray CI by @anagainaru in Replacing DefApps with Core/24.00 for the frontier-cray CI #4367
• Add testing for structs using Kokkos Views by @anagainaru in Add testing for structs using Kokkos Views #4369
• Some BP5 Serialization documentation, mostly writer-side perspective. by @eisenhauer in Some BP5 Serialization documentation, mostly writer-side perspective. #4372
• Possibly fix leak when an ondemand request comes in just before close by @eisenhauer in Possibly fix leak when an ondemand request comes in just before close #4373
• fix(cmake): increase min required cmake when using bloscs2 by @vicentebolea in fix(cmake): increase min required cmake when using bloscs2 #4379
• ci(msan): fine tune test timeouts by @vicentebolea in ci(msan): fine tune test timeouts #4380
• Start developer_docs directory with some content by @eisenhauer in Start developer_docs directory with some content #4385
• FFS Upstream by @eisenhauer in FFS Upstream #4391
• Add New SelectiveMetadatAggregation method by @eisenhauer in Add New SelectiveMetadatAggregation method #4387
• Fix bug in reader side derived vars by @eisenhauer in Fix bug in reader side derived vars #4393
• [Bug fix] SegFault when a variable that is part of a derived expression doesn't have data by @anagainaru in [Bug fix] SegFault when a variable that is part of a derived expression doesn't have data #4394
• [Bug fix] Change the default expression string for derived variables by @anagainaru in [Bug fix] Change the default expression string for derived variables #4396
• VarRec->PerWriterBlockStart is initialized, but never used, remove it by @eisenhauer in VarRec->PerWriterBlockStart is initialized, but never used, remove it #4386
• Separate the derived variable testing in MPI and not MPI functions by @anagainaru in Separate the derived variable testing in MPI and not MPI functions #4395
• Fix name clashes in concurrent tests by @eisenhauer in Fix name clashes in concurrent tests #4397
• Add the VariableExprStr function in the CampaignReader by @anagainaru in Add the VariableExprStr function in the CampaignReader #4401
• Fixed typo in GPU-aware docs by @lahwaacz in Fixed typo in GPU-aware docs #4403
• Fix DESTDIR installation by @lahwaacz in Fix DESTDIR installation #4402
• docs: add ADIOS2_USE_PIP by @vicentebolea in docs: add ADIOS2_USE_PIP #4408
• added more details in profiling collection by @guj in added more details in profiling collection #4342
• Update cmake version used for the Frontier build script by @anagainaru in Update cmake version used for the Frontier build script #4400
• Added note to PerformDataWrite (it is collective) by @guj in Added note to PerformDataWrite (it is collective) #4409
• ci: add spack ci sync job by @vicentebolea in ci: add spack ci sync job #4420
• ci: add spack ci job by @vicentebolea in ci: add spack ci job #4421
• fix FLEX arguments for cmake by @pnorbert in fix FLEX arguments for cmake #4423
• ci: recover cdash status by @vicentebolea in ci: recover cdash status #4425
• Testing the python API for derived variables (Stats and Expression) by @anagainaru in Testing the python API for derived variables (Stats and Expression) #4399
• Make adios2_remote_server multithreaded and compress responses using … by @pnorbert in Make adios2_remote_server multithreaded and compress responses using … #4407
• cmake: refactor generate-adios2-config.sh by @vicentebolea in cmake: refactor generate-adios2-config.sh #4405
• ci: rehaul sanitizer builds by @vicentebolea in ci: rehaul sanitizer builds #4424
• Fix for cxx23 by @pnorbert in Fix for cxx23 #4416
• Fix effectively ignored XML parameters for SZ operator by @gregorweiss in Fix effectively ignored XML parameters for SZ operator #4404
• thirdparty: Update kwsys by @vicentebolea in thirdparty: Update kwsys #4412
• Update pugixml by @vicentebolea in Update pugixml #4437
• Update pybind11 by @vicentebolea in Update pybind11 #4436
• EVPath upstream by @eisenhauer in EVPath upstream #4428
• Update pugixml by @vicentebolea in Update pugixml #4439
• Operator for BigWhoop compression by @gregorweiss in Operator for BigWhoop compression #4398
• docs: update readthedocs.yml to new requirements by @vicentebolea in docs: update readthedocs.yml to new requirements #4445
• Fix buffer overflow by @eisenhauer in Fix buffer overflow #4447
• ReadMe.md: 2.10.2 is available by @eschnett in ReadMe.md: 2.10.2 is available #4448
• GTKorvo Upstream by @eisenhauer in GTKorvo Upstream #4450
• Add support for key-encrypted metadata in campaign files. Ask the con… by @pnorbert in Add support for key-encrypted metadata in campaign files. Ask the con… #4451
• Remote server test fix by @eisenhauer in Remote server test fix #4456
• Modify remote requests to include step count in a single request ... by @pnorbert in Modify remote requests to include step count in a single request ... #4455
• ci: set ubuntu host to be ubuntu22.04 by @vicentebolea in ci: set ubuntu host to be ubuntu22.04 #4464
• Cache data blocks larger than 1MB in files. by @pnorbert in Cache data blocks larger than 1MB in files. #4461
• Retry on server connection failure by @eisenhauer in Retry on server connection failure #4465
• Fixup Shape for FlattenSteps by @eisenhauer in Fixup Shape for FlattenSteps #4467
• Bindings for cuPy pointers by @anagainaru in Bindings for cuPy pointers #4459
• Kill Modifiable info item for Attributes by @eisenhauer in Kill Modifiable info item for Attributes #4468
• [BugFix] Set the correct dimensions and type for a derived expression by @anagainaru in [BugFix] Set the correct dimensions and type for a derived expression #4473
• Daos Engine Merge by @eisenhauer in Daos Engine Merge #4477
• Cleanup CMake a bit by @eisenhauer in Cleanup CMake a bit #4479
• ci,frontier: do not load hdf5 by @vicentebolea in ci,frontier: do not load hdf5 #4481
• ci,macos: specify lib shared type in gh cache by @vicentebolea in ci,macos: specify lib shared type in gh cache #4484
• Catch Shape.size() change by @eisenhauer in Catch Shape.size() change #4487
• Make filenames not conflict in TestForEOF by @eisenhauer in Make filenames not conflict in TestForEOF #4490
• python: stream.write accept np array w/o extra args by @vicentebolea in python: stream.write accept np array w/o extra args #4491
• Fix joined array output in bpls by @eisenhauer in Fix joined array output in bpls #4494
• mpi_dp: build always, heuristics determine priority by @vicentebolea in mpi_dp: build always, heuristics determine priority #4498
• ci: delegate cdash status into the cdash-status action by @vicentebolea in ci: delegate cdash status into the cdash-status action #4501
• ci: codeql ignore thirdparties by @vicentebolea in ci: codeql ignore thirdparties #4500
• ci: codeql ignore thirdparties by @vicentebolea in ci: codeql ignore thirdparties #4503
• ci: set conda classic solver by @vicentebolea in ci: set conda classic solver #4507
• ci: update frank cluster tags by @vicentebolea in ci: update frank cluster tags #4499
• Simpler Joined Array Size mod, backout prior fix by @eisenhauer in Simpler Joined Array Size mod, backout prior fix #4508
• Hopefully fix a heisen bug by @eisenhauer in Hopefully fix a heisen bug #4509
• ci: fix codeql warnings by @vicentebolea in ci: fix codeql warnings #4506
• issues: stale issues that have been open been inactive by @vicentebolea in issues: stale issues that have been open been inactive #4495
• Make the committed DAOS engines actually work by @eisenhauer in Make the committed DAOS engines actually work #4510
• Suppress persistent fortran warnings by @eisenhauer in Suppress persistent fortran warnings #4511
• cmake: expose ADIOS2_HAVE_SST in cmake pkgs by @vicentebolea in cmake: expose ADIOS2_HAVE_SST in cmake pkgs #4512
• Kill heisen failures on xrootd start by @eisenhauer in Kill heisen failures on xrootd start #4515
• Support -D -d -t for BP5 files, i.e. dump a subset of each block of … by @pnorbert in Support -D -d -t for BP5 files, i.e. dump a subset of each block of … #4519
• Use Pread() instead of read() in Posix file transport to eliminate kernel state dependency by @eisenhauer in Use Pread() instead of read() in Posix file transport to eliminate kernel state dependency #4517
• New reading engine "TimeSeries" that can read .ats text file ... by @pnorbert in New reading engine "TimeSeries" that can read .ats text file ...  #4520
• ci: bump UO Frank sanitizer kernel ver by @vicentebolea in ci: bump UO Frank sanitizer kernel ver #4521
• ci: show black formatter diff by @vicentebolea in ci: show black formatter diff #4527
• Loading and fetching data to and from torch tensors directly by @anagainaru in Loading and fetching data to and from torch tensors directly #4525
• Add old output files and ctests to see if we can still read them by @eisenhauer in Add old output files and ctests to see if we can still read them #4523
• Python API support for CuPy.arrays and torch.Tensor by @anagainaru in Python API support for CuPy.arrays and torch.Tensor #4528
• Change buffer size restriction to fixed rather than compiler/arch dependent value. by @eisenhauer in Change buffer size restriction to fixed rather than compiler/arch dependent value. #4529
• Python API: Support Deferred mode for read by @pnorbert in Python API: Support Deferred mode for read #4530
• Fixing the GPU examples for the Fortran and C bindings by @anagainaru in Fixing the GPU examples for the Fortran and C bindings #4480
• Handle HDF5 and TEXT files in from ACA 0.4 by @pnorbert in Handle HDF5 and TEXT files in from ACA 0.4 #4532
• Enable checking of new archive output by @eisenhauer in Enable checking of new archive output #4531
• Gray scott examples - fix memory leak by @ChristosT in Gray scott examples - fix memory leak #4533
• init_fabric: use Trace verbosity instead of Critical for success message by @Lgt2x in init_fabric: use Trace verbosity instead of Critical for success message #4537
• Tweaks to remove DP name dependency from SST by @eisenhauer in Tweaks to remove DP name dependency from SST #4539
• WaitForOpen in EVPath remote was a no-op, should throw if an open fails by @eisenhauer in WaitForOpen in EVPath remote was a no-op, should throw if an open fails #4538
• Summarized notes on SST options by @guj in Summarized notes on SST options #4541
• Make sst tutorial doc be reachable by @eisenhauer in Make sst tutorial doc be reachable #4543
• GoogleTest version update, move to require C++14. by @eisenhauer in GoogleTest version update, move to require C++14. #4544
• Kill orphan files by @eisenhauer in Kill orphan files #4547
• Fix mishandling m_AvailableShapes map in TimeSeries engine. by @pnorbert in Fix mishandling m_AvailableShapes map in TimeSeries engine.  #4546
• Support for shared memory provider of libfabric in SST + simple support for manual data progress via threading by @franzpoeschel in Support for shared memory provider of libfabric in SST + simple support for manual data progress via threading #3964
• Operator Parameters by @eisenhauer in Operator Parameters #4549
• Dill tweaks by @eisenhauer in Dill tweaks #4550
• Fix some new Windows warnings by @eisenhauer in Fix some new Windows warnings #4552
• Fixes for 32-bit builds on Visual Studio 17 (Windows Server 2022) by @franzpoeschel in Fixes for 32-bit builds on Visual Studio 17 (Windows Server 2022) #4551
• Add profiler/ for parsing and plotting adios json files by @guj in Add profiler/ for parsing and plotting adios json files #4526
• Add OpenReadSimpleFile() and Close() to the remote server by @eisenhauer in Add OpenReadSimpleFile() and Close() to the remote server #4553
• Fix leaks if server has already exited by @eisenhauer in Fix leaks if server has already exited #4557
• Suppress windows warning about enums by @eisenhauer in Suppress windows warning about enums #4555
• Output testing: zip warning reduced to notice, disable new output gen on Windows by @eisenhauer in Output testing: zip warning reduced to notice, disable new output gen on Windows #4564
• ci: use windows 2022 and 2025 by @vicentebolea in ci: use windows 2022 and 2025 #4568
• Add zip ci images by @vicentebolea in Add zip ci images #4567
• Fix segfault in BP5 writer when using append mode, everyone writes serial, and num aggregators not set by @caitlinross in Fix segfault in BP5 writer when using append mode, everyone writes serial, and num aggregators not set #4570
• Campaign v0.5 by @pnorbert in Campaign v0.5 #4571
• Operator::AddExtraParameters() to allow engine pass parameters before… by @pnorbert in Operator::AddExtraParameters() to allow engine pass parameters before… #4576
• Fix column-major reading with memory selection by @ohlmann in Fix column-major reading with memory selection #4575
• Fix GCC 15 compatibility: Add missing #include headers by @Copilot in Fix GCC 15 compatibility: Add missing #include <cstdint> headers #4569
• yaml-cpp: Add missing include by @qtpowell in yaml-cpp: Add missing <cstdint> include #4578
• Enable operator info in bpls by @eisenhauer in Enable operator info in bpls #4574
• Tweak Operator Name Query to avoid op application by @eisenhauer in Tweak Operator Name Query to avoid op application #4583
• Fix a problem with multiple spans on a single variable in BP5 by @eisenhauer in Fix a problem with multiple spans on a single variable in BP5 #4587
• Simplify test temporarily to avoid bug by @eisenhauer in Simplify test temporarily to avoid bug #4589
• Fix issue with GCC 7. by @kmorel in Fix issue with GCC 7. #4591
• EVPath upstream by @eisenhauer in EVPath upstream #4595
• ci,conda: remove default channels by @vicentebolea in ci,conda: remove default channels #4594
• Shared remote connection by @pnorbert in Shared remote connection #4593
• Restore remote test and fix bug by @eisenhauer in Restore remote test and fix bug #4590
• Pull in ai_canonname fix from EVPath upstream by @eisenhauer in Pull in ai_canonname fix from EVPath upstream #4598
• Campaign threaded performgets by @pnorbert in Campaign threaded performgets #4599
• Make logging append, kill spurious by @eisenhauer in Make logging append, kill spurious #4600
• Move decompression from ReadResponseHandler() that is single threaded… by @pnorbert in Move decompression from ReadResponseHandler() that is single threaded… #4601
• Fix Python, restore DataMan tests by @eisenhauer in Fix Python, restore DataMan tests #4602
• Fix posix transport offset by @pnorbert in Fix posix transport offset #4603
• ci: fix conda windows error by @vicentebolea in ci: fix conda windows error #4606
• largeblocks: Skip test if not enough space by @vicentebolea in largeblocks: Skip test if not enough space #4605
• ci,frontier: add maintainance job for frontier ci by @vicentebolea in ci,frontier: add maintainance job for frontier ci #4608
• Check for bad value in LookupVarByKey by @eisenhauer in Check for bad value in LookupVarByKey #4604
• Remove system('cat file') by @eisenhauer in Remove system('cat file') #4607
• HDF5 engine / remote, small arrays by @pnorbert in HDF5 engine / remote, small arrays #4609
• Fix in TimeSeries engine for variables without min/max (like char arr… by @pnorbert in Fix in TimeSeries engine for variables without min/max (like char arr… #4581
• ci: remove gcc11 uo job by @vicentebolea in ci: remove gcc11 uo job #4610
• python API for Engine::GetMetadata and IO.Open(...metadata) by @pnorbert in python API for Engine::GetMetadata and IO.Open(...metadata) #4613
• Modify ATS file format to accomodate for remote files in the series. … by @pnorbert in Modify ATS file format to accomodate for remote files in the series. … #4614
• FFS Upstream by @eisenhauer in FFS Upstream #4618
• Free the MPI_Info object created in GroupBySHM(). by @pnorbert in Free the MPI_Info object created in GroupBySHM().  #4626
• • Delete adios2_campaign_manager.py since it is obsolete by @pnorbert in - Delete adios2_campaign_manager.py since it is obsolete #4628
• libfabric/RDMA dataplane of SST: Fix detection of CXI key by @franzpoeschel in libfabric/RDMA dataplane of SST: Fix detection of CXI key #4629
• Use proper dynamic vector construction to fix segfault with gcc-15 by @mr-c in Use proper dynamic vector construction to fix segfault with gcc-15 #4632
• Fix a typo. by @mr-c in Fix a typo. #4631
• cmake: allow building with newer CXX standards by @vicentebolea in cmake: allow building with newer CXX standards #4633
• Add verbosity and tweak some termination conditions by @eisenhauer in Add verbosity and tweak some termination conditions #4635
• FFS upstream by @eisenhauer in FFS upstream #4636
• Data size based aggregation by @scottwittenburg in Data size based aggregation #4548
• Update docs to mention DataSizeBased aggregation by @scottwittenburg in Update docs to mention DataSizeBased aggregation #4639
• Changing location of campaign config files by @pnorbert in Changing location of campaign config files #4640
• Parallelize portions of metadata installation by @eisenhauer in Parallelize portions of metadata installation #4623
• Campaign doc by @pnorbert in Campaign doc #4641
• generate-adios2-config: do not hide custom compiler environment settings by @mathstuf in generate-adios2-config: do not hide custom compiler environment settings #4634
• Changes to the build system for the bpcmp utility by @anagainaru in Changes to the build system for the bpcmp utility #4637
• Allowing constants in the expressions of derived variables by @anagainaru in Allowing constants in the expressions of derived variables #4644
• CURL derived variable applied on an aggregated array by @anagainaru in CURL derived variable applied on an aggregated array #4650
• Making build with HDF5 more informative by @guj in Making build with HDF5 more informative #4649
• Fixing a bunch of small bugs/typos for the derived variable engine by @anagainaru in Fixing a bunch of small bugs/typos for the derived variable engine #4651
• Python API fixes and examples: by @pnorbert in Python API fixes and examples: #4653
• XRootD development by @eisenhauer in XRootD development #4655
• Clean up DAOS Cmake conditions by @eisenhauer in Clean up DAOS Cmake conditions #4656
• Complex MGARD operator by @gqian-coder in Complex MGARD operator #4654
• Allow building with DAOS without Caliper by @eisenhauer in Allow building with DAOS without Caliper #4658
• Fix Warnings by @eisenhauer in Fix Warnings #4660
• Bug fix for derived variables when used on Joined Arrays by @anagainaru in Bug fix for derived variables when used on Joined Arrays #4662
• Fix leak in CommImplMPI::GroupByShm by @eisenhauer in Fix leak in CommImplMPI::GroupByShm #4665
• ci: fix readthedocs job by speeding up job by @vicentebolea in ci: fix readthedocs job by speeding up job #4681
• ci,frontier: do not fetch in computing nodes by @vicentebolea in ci,frontier: do not fetch in computing nodes #4678
• ci: make build_and_test dep on everything by @vicentebolea in ci: make build_and_test dep on everything #4684
• Configure Dependabot for GitHub Actions and Docker by @vicentebolea in Configure Dependabot for GitHub Actions and Docker #4685
• Restrict workflow token permissions to minimal required access by @Copilot in Restrict workflow token permissions to minimal required access #4679
• Stopgap measure for deprecated libfabric macros by @eisenhauer in Stopgap measure for deprecated libfabric macros #4690
• Pin CI GitHub action dependencies by hash by @sbelsk in Pin CI GitHub action dependencies by hash #4691
• Bump NumPy minimum required version to 1.22 by @sbelsk in Bump NumPy minimum required version to 1.22 #4692
• Aca 0.6 https again by @pnorbert in Aca 0.6 https again #4693
• H5VolReadWrite: Do not use variable after memory is freed by @scottwittenburg in H5VolReadWrite: Do not use variable after memory is freed #4698
• Raise an exception when an XRootD get fails. by @eisenhauer in Raise an exception when an XRootD get fails. #4699
• ci: Update os and spack/mirror versions for ubuntu docker images by @scottwittenburg in ci: Update os and spack/mirror versions for ubuntu docker images #4666
• CampaignReader include/exclude patterns (regular expressions) by @pnorbert in CampaignReader include/exclude patterns (regular expressions) #4701
• docs: add OpenSSF best practices badge by @vicentebolea in docs: add OpenSSF best practices badge #4703
• ci: restrict file scope perms in workflows by @vicentebolea in ci: restrict file scope perms in workflows #4702
• Documentation for profiling performance with ADIOS by @anagainaru in Documentation for profiling performance with ADIOS #4524
• Revise supported versions in SECURITY.md by @vicentebolea in Revise supported versions in SECURITY.md #4680
• CODEOWNERS: use infra group instead of vicente by @vicentebolea in CODEOWNERS: use infra group instead of vicente #4704
• Add OpenSSF scorecard report action by @sbelsk in Add OpenSSF scorecard report action #4706
• remove untrue documentation for adios2.Stream.available_variables() by @pnorbert in remove untrue documentation for adios2.Stream.available_variables() #4708
• Bump the all-actions group across 1 directory with 6 updates by @dependabot[bot] in Bump the all-actions group across 1 directory with 6 updates #4694
• Fix: add missing cstdint include in adiosNetwork.h by @Lgt2x in Fix: add missing cstdint include in adiosNetwork.h #4709
• bindings: renamed cxx11 bindings to cxx by @vicentebolea in bindings: renamed cxx11 bindings to cxx #4707
• CI: use gfortran-13 in macos xcode-15 jobs by @vicentebolea in CI: use gfortran-13 in macos xcode-15 jobs #4712
• Bump version to v2.11.0-rc1 by @vicentebolea in Bump version to v2.11.0-rc1 #4713
• docs: add whatsnew section for 2.11 by @vicentebolea in docs: add whatsnew section for 2.11 #4715
• Fix logic error in BP5 selective metadata aggregation (release branch) by @eisenhauer in Fix logic error in BP5 selective metadata aggregation (release branch) #4718
• RDMA dataplane: Share fi_fabric() and don't close until all are done (Release Branch) by @eisenhauer in RDMA dataplane: Share fi_fabric() and don't close until all are done (Release Branch) #4719
• Do not install example binaries in adios2 deployments by @vicentebolea in Do not install example binaries in adios2 deployments #4722
• Merge pull request Potential fix for CMake logic in using an external EVPath build #4697 from Crivella/faeture-fix_external_evpath by @vicentebolea in Merge pull request #4697 from Crivella/faeture-fix_external_evpath #4720
• Bump version to v2.11.0 by @vicentebolea in Bump version to v2.11.0 #4725

New Contributors

• @JamieJQuinn made their first contribution in Fix typo in Fortran API docs #4102
• @Change72 made their first contribution in ADIOS2 User Side KVCache to accelerate the remote query access. #4210
• @adrianjhpc made their first contribution in Fixing C++ and Fortran example to correctly read in files (not master branch this time) #4239
• @JamesMcClung made their first contribution in Remove reference to xarray backend #4235
• @gregorweiss made their first contribution in Support for 4D arrays in ZFP compressor #4344
• @lahwaacz made their first contribution in Fixed typo in GPU-aware docs #4403
• @ChristosT made their first contribution in Gray scott examples - fix memory leak #4533
• @Lgt2x made their first contribution in init_fabric: use Trace verbosity instead of Critical for success message #4537
• @ohlmann made their first contribution in Fix column-major reading with memory selection #4575
• @Copilot made their first contribution in Fix GCC 15 compatibility: Add missing #include <cstdint> headers #4569
• @qtpowell made their first contribution in yaml-cpp: Add missing <cstdint> include #4578
• @kmorel made their first contribution in Fix issue with GCC 7. #4591
• @mr-c made their first contribution in Use proper dynamic vector construction to fix segfault with gcc-15 #4632
• @gqian-coder made their first contribution in Complex MGARD operator #4654
• @sbelsk made their first contribution in Pin CI GitHub action dependencies by hash #4691

Full Changelog: v2.10.2...v2.11.0

---

This discussion was created from the release v2.11.0.