Spuriously large and unphysical increments in EnKF on release/rrfs.v1.0.0 #960
Description
When running EnKF retros in the dev-sci branch of rrfs-workflow, we found that the increments can be extremely large:
Here are increments from the first cycle of a conus13km retro using 30 members and only assimilating aircar temperature and wind observations:
ens. mean anal. increment min/max u -400.986419678 526.739135742
ens. mean anal. increment min/max v -299.293304443 286.644195557
ens. mean anal. increment min/max t -50.3178100586 261.687805176
ens. mean anal. increment min/max oz -0.537249161425E-05 0.100919532997E-04
ens. mean anal. increment min/max ps -101.541290283 21.7783260345
These increments are unphysical and crash the subsequent ensemble forecasts.
Updates to src/enkf/mpi_readobs.f90 can resolve this problem:
ens. mean anal. increment min/max u -24.8986816406 27.3974285126
ens. mean anal. increment min/max v -30.1263866425 35.6076431274
ens. mean anal. increment min/max t -4.40757656097 6.25625514984
ens. mean anal. increment min/max oz -0.423902207558E-06 0.442797130518E-06
ens. mean anal. increment min/max ps -1.66400146484 2.14062714577
Here is the ChatGPT summary of the changes made to mpi_readobs.f90: "this update fixes the large spurious increments because it removes a fragile shared memory plus “root only” reduction pattern and replaces it with a straightforward, globally consistent reduction of a normal Fortran array. That prevents corrupted or incomplete ensemble priors in observation space, which were manifesting as giant outlier perturbations and thus giant increments". However, we are still not sure if this problem is specifically due to GSI or something with the configuration in rrfs-workflow. We will need more testing once mpi_readobs.f90 is updated.