Update the ULFM Readme

Comment indentation

Signed-off-by: Aurelien Bouteiller <[email protected]>

Author: abouteiller

docs/features/ulfm.rst

@@ -6,6 +6,18 @@ User-Level Fault Mitigation (ULFM)
 This chapter documents the features and options specific to the **User
 Level Failure Mitigation (ULFM)** Open MPI implementation.
 
+Quick Start
+-----------
+
+This is an extremely terse summary of how to use ULFM:
+
+.. code-block::
+
+  shell$ ./configure --with-ft=ulfm [...options...]
+  shell$ make [-j N] all install
+  shell$ mpicc my-ft-program.c -o my-ft-program
+  shell$ mpiexec -n 4 --with-ft ulfm my-ft-program
+
 Features
 --------
 
@@ -100,11 +112,12 @@ Available from: https://journals.sagepub.com/doi/10.1177/1094342013488238.
 Building ULFM support in Open MPI
 ---------------------------------
 
-In Open MPI |ompi_ver|, ULFM support is **enabled by default** |mdash|
-when you build Open MPI, unless you specify ``--without-ft``, ULFM
-support will automatically be built.
+In Open MPI |ompi_ver|, ULFM support is **built-in by default** |mdash|
+that is, when you build Open MPI, unless you specify ``--without-ft``, ULFM
+support is automatically available (but is inactive unless enabled at
+runtime).
 
-Optionally, you can specify ``--with-ft`` to ensure that ULFM support
+Optionally, you can specify ``--with-ft ulfm`` to ensure that ULFM support
 is definitely built.
 
 Support notes
@@ -114,89 +127,6 @@ Support notes
   that if you are going to use ULFM, you should disable building
   OpenSHMEM with ``--disable-oshmem``.
 
-* SLURM is tested and supported with fault tolerance.
-
-  .. important:: Do not use ``srun``, or your application gets killed
-                 by the scheduler upon the first failure.  Instead,
-                 use ``mpirun`` in an ``salloc/sbatch`` allocation.
-
-* LSF is untested with fault tolerance.
-
-* PBS/Torque is tested and supported with fault tolerance.
-
-  .. important:: Be sure to use ``mpirun`` in a ``qsub`` allocation.
-
-Modified, Untested and Disabled Components
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-Frameworks and components which are not listed in the following list
-are unmodified and support fault tolerance. Listed frameworks may be
-**modified** (and work after a failure), **untested** (and work before
-a failure, but may malfunction after a failure), or **disabled** (they
-cause unspecified behavior all around when FT is enabled).
-
-All runtime disabled components are listed in the ``ft-mpi`` aggregate
-MCA param file
-``$installdir/share/openmpi/amca-param-sets/ft-mpi``. You can tune the
-runtime behavior with ULFM by either setting or unsetting variables in
-this file (or by overiding the variable on the command line (e.g.,
-``--mca btl ofi,self``). Note that if fault tolerance is disabled at
-runtime, these components will load normally (this may change observed
-performance when comparing with and without fault tolerance).
-
-* ``pml``: MPI point-to-point management layer
-
-  * ``monitoring``, ``v``: **untested** (they have not been modified
-    to handle faults)
-  * ``cm``, ``crcpw``, ``ucx``: **disabled**
-
-* ``btl``: Point-to-point Byte Transfer Layer
-
-  * ``ofi``, ``portals4``, ``smcuda``, ``usnic``, ``sm(+knem)``:
-    **untested** (they may work properly, please report)
-
-* ``mtl``: Matching transport layer Used for MPI point-to-point messages on
-  some types of networks
-
-  * All ``mtl`` components are **disabled**
-
-* ``coll``: MPI collective algorithms
-
-  * ``cuda``, ``inter``, ``sync``, ``sm``: **untested** (they have not
-    been modified to handle faults, but we expect correct post-fault
-    behavior)
-  * ``hcoll``, ``portals4`` **disabled** (they have not been modified
-    to handle faults, and we expect unspecified post-fault behavior)
-
-* ``osc``: MPI one-sided communications
-
-  * All ``osc`` components are **untested** (they have not been
-    modified to handle faults, and we expect unspecified post-fault
-    behavior)
-
-* ``io``: MPI I/O and dependent components
-
-  * ``fs``: File system functions for MPI I/O
-  * ``fbtl``: File byte transfer layer: abstraction for individual
-    read/write operations for OMPIO
-  * ``fcoll``: Collective read and write operations for MPI I/O
-  * ``sharedfp``: Shared file pointer operations for MPI I/O
-  * All components in these frameworks are unmodified, **untested**
-    (we expect clean post-failure abort)
-
-* ``vprotocol``: Checkpoint/Restart components
-
-  * These components have not been modified to handle faults, and are
-    **untested**.
-
-* ``threads``, ``wait-sync``: Multithreaded wait-synchronization
-  object
-
-  * ``argotbots``, ``qthreads``: **disabled** (these components have
-    not been modified to handle faults; we expect post-failure
-    deadlock)
-
-
 Running ULFM Open MPI
 ---------------------
 
@@ -215,12 +145,14 @@ Running your application
 
 You can launch your application with fault tolerance by simply using
 the normal Open MPI ``mpiexec`` launcher, with the
-``--with-ft ulfm`` CLI option:
+``--with-ft ulfm`` CLI option (or its synonym ``--with-ft mpi``):
 
 .. code-block::
 
    shell$ mpirun --with-ft ulfm ...
 
+  .. important:: by default, fault tolerance is not active.
+
 Running under a batch scheduler
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
@@ -231,6 +163,18 @@ process fails. In order to avoid this problem, it is preferred that
 you use ``mpiexec`` within an allocation (e.g., ``salloc``,
 ``sbatch``, ``qsub``) rather than a direct launch (e.g., ``srun``).
 
+* SLURM is tested and supported with fault tolerance.
+
+  .. important:: Do not use ``srun``, or your application gets killed
+                 by the scheduler upon the first failure.  Instead,
+                 use ``mpirun`` in an ``salloc/sbatch`` allocation.
+
+* LSF is untested with fault tolerance.
+
+* PBS/Torque is tested and supported with fault tolerance.
+
+  .. important:: Be sure to use ``mpirun`` in a ``qsub`` allocation.
+
 Run-time tuning knobs
 ^^^^^^^^^^^^^^^^^^^^^
 
@@ -240,12 +184,21 @@ most cases. You can change the default settings with ``--mca
 mpi_ft_foo <value>`` for Open MPI options, and with ``--prtemca
 errmgr_detector_bar <value>`` for PRTE options.
 
+  .. important:: The main control for enabling/disabling fault tolerance
+                 at runtime is the ``--with-ft ulfm`` (or its synomym
+                 ``--with-ft mpi``) ``mpiexec`` CLI option. This option
+                 setup multiple subsystems of Open MPI to enable fault
+                 tolerance. The options described below are best used to
+                 overide the default behavior after the ``--with-ft ulfm``
+                 opion is used.
+
 PRTE level options
 ~~~~~~~~~~~~~~~~~~
 
-* ``prrte_enable_recovery <true|false> (default: false)`` controls
+* ``prrte_enable_ft <true|false> (default: false)`` controls
   automatic cleanup of apps with failed processes within
-  mpirun. Enabling this option also enables ``mpi_ft_enable``.
+  mpirun. This option is automatically set to ``true`` when using
+  ``--with-ft ulfm``.
 * ``errmgr_detector_priority <int> (default 1005``) selects the
   PRRTE-based failure detector. Only available when
   ``prte_enable_recovery`` is ``true``. You can set this to ``0`` when
@@ -263,17 +216,33 @@ PRTE level options
 Open MPI level options
 ~~~~~~~~~~~~~~~~~~~~~~
 
-* ``mpi_ft_enable <true|false> (default: same as
-  prrte_enable_recovery)`` permits turning on/off fault tolerance at
-  runtime. When false, failure detection is disabled; Interfaces
-  defined by the fault tolerance extensions are substituted with dummy
-  non-fault tolerant implementations (e.g., ``MPIX_Comm_agree`` is
-  implemented with ``MPI_Allreduce``); All other controls below become
-  irrelevant.
+Some default values are applied to some Open MPI parameters when using
+``mpiexec --with-ft ulfm``. These defaults are obtained from the ``ft-mpi``
+aggregate MCA param file
+``$installdir/share/openmpi/amca-param-sets/ft-mpi``. You can tune the
+runtime behavior with ULFM by either setting or unsetting variables in
+this file, or by overiding the variable on the command line (e.g.,
+``--mca btl ofi,self``).
+
+  .. important:: Note that if fault tolerance is disabled at runtime,
+                 that is, when not using ``--with-ft ulfm``), the
+                 ``ft-mpi`` MCA param file is not loaded, thus
+                 components that are unsafe for fault tolerance will
+                 load normally (this may change observed performance
+                 when comparing with and without fault tolerance).
+
+* ``mpi_ft_enable <true|false> (default: false)``
+  permits turning on/off fault tolerance at runtime. This option is
+  automatically set to ``true`` from the aggregate MCA param file
+  ``ft-mpi`` loaded when using ``--with-ft ulfm``. When false, failure
+  detection is disabled; Interfaces defined by the fault tolerance extensions
+  are substituted with dummy non-fault tolerant implementations (e.g.,
+  ``MPIX_Comm_agree`` is implemented with ``MPI_Allreduce``); All other
+  controls below become irrelevant.
 * ``mpi_ft_verbose <int> (default: 0)`` increases the output of the
   fault tolerance activities. A value of 1 will report detected
   failures.
-* ``mpi_ft_detector <true|false> (default: false)``, **EXPERIMENTAL**
+* ``mpi_ft_detector <true|false> (default: false)``, **DEPRECATED**
   controls the activation of the Open MPI level failure detector. When
   this detector is turned off, all failure detection is delegated to
   PRTE (see above).  The Open MPI level fault detector is
@@ -291,22 +260,99 @@ Open MPI level options
   latency (typically 1us increase).* You may want to **enable this
   option if you experience false positive** processes incorrectly
   reported as failed with the Open MPI failure detector.
+  This option is only relevant when `mpi_ft_detector` is `true`.
 * ``mpi_ft_detector_period <float> (default: 3e0 seconds)`` heartbeat
   period. Recommended value is 1/3 of the timeout. _Values lower than
   100us may impart a noticeable effect on latency (typically a 3us
   increase)._
+  This option is only relevant when `mpi_ft_detector` is `true`.
 * ``mpi_ft_detector_timeout <float> (default: 1e1 seconds)`` heartbeat
   timeout (i.e. failure detection speed). Recommended value is 3 times
   the heartbeat period.
+  This option is only relevant when `mpi_ft_detector` is `true`.
 
 Known Limitations in ULFM
-^^^^^^^^^^^^^^^^^^^^^^^^^
+-------------------------
 
 * InfiniBand support is provided through the UCT BTL; fault tolerant
   operation over the UCX PML is not yet supported for production runs.
 * TOPO, FILE, RMA are not fault tolerant. They are expected to work
   properly before the occurence of the first failure.
 
+Modified, Untested and Disabled Components
+------------------------------------------
+
+Frameworks and components which are not listed in the following list
+are unmodified and support fault tolerance. Listed frameworks may be
+**modified** (and work after a failure), **untested** (and work before
+a failure, but may malfunction after a failure), or **disabled** (they
+cause unspecified behavior all around when FT is enabled).
+
+All runtime disabled components are listed in the ``ft-mpi`` aggregate
+MCA param file
+``$installdir/share/openmpi/amca-param-sets/ft-mpi``. You can tune the
+runtime behavior with ULFM by either setting or unsetting variables in
+this file (or by overiding the variable on the command line (e.g.,
+``--mca btl ofi,self``).
+
+  .. important:: Note that if fault tolerance is disabled at runtime,
+                 the ``ft-mpi`` MCA param file is not loaded, thus
+                 components that are unsafe for fault tolerance will
+                 load normally (this may change observed performance
+                 when comparing with and without fault tolerance).
+
+* ``pml``: MPI point-to-point management layer
+
+  * ``monitoring``, ``v``: **untested** (they have not been modified
+    to handle faults)
+  * ``cm``, ``crcpw``, ``ucx``: **disabled**
+
+* ``btl``: Point-to-point Byte Transfer Layer
+
+  * ``ofi``, ``portals4``, ``smcuda``, ``usnic``, ``sm(+knem)``:
+    **untested** (they may work properly, please report)
+
+* ``mtl``: Matching transport layer Used for MPI point-to-point messages on
+  some types of networks
+
+  * All ``mtl`` components are **disabled**
+
+* ``coll``: MPI collective algorithms
+
+  * ``cuda``, ``inter``, ``sync``, ``sm``: **untested** (they have not
+    been modified to handle faults, but we expect correct post-fault
+    behavior)
+  * ``hcoll``, ``portals4`` **disabled** (they have not been modified
+    to handle faults, and we expect unspecified post-fault behavior)
+
+* ``osc``: MPI one-sided communications
+
+  * All ``osc`` components are **untested** (they have not been
+    modified to handle faults, and we expect unspecified post-fault
+    behavior)
+
+* ``io``: MPI I/O and dependent components
+
+  * ``fs``: File system functions for MPI I/O
+  * ``fbtl``: File byte transfer layer: abstraction for individual
+    read/write operations for OMPIO
+  * ``fcoll``: Collective read and write operations for MPI I/O
+  * ``sharedfp``: Shared file pointer operations for MPI I/O
+  * All components in these frameworks are unmodified, **untested**
+    (we expect clean post-failure abort)
+
+* ``vprotocol``: Checkpoint/Restart components
+
+  * These components have not been modified to handle faults, and are
+    **untested**.
+
+* ``threads``, ``wait-sync``: Multithreaded wait-synchronization
+  object
+
+  * ``argotbots``, ``qthreads``: **disabled** (these components have
+    not been modified to handle faults; we expect post-failure
+    deadlock)
+
 Changelog
 ---------
 

ompi/runtime/ompi_mpi_params.c

@@ -406,10 +406,10 @@ int ompi_mpi_register_params(void)
 
 #if OPAL_ENABLE_FT_MPI
     /* Before loading any other part of the MPI library, we need to load
- *      * the ft-mpi tune file to override default component selection when
- *           * FT is desired ON; this does override openmpi-params.conf, but not
- *                * command line or env.
- *                     */
+     * the ft-mpi tune file to override default component selection when
+     * FT is desired ON; this does override openmpi-params.conf, but not
+     * command line or env.
+     */
     if( ompi_ftmpi_enabled ) {
         mca_base_var_load_extra_files("ft-mpi", false);
     }