ulfm readme: Indentation of the 'important' and lists

Signed-off-by: Aurelien Bouteiller <[email protected]>

Author: abouteiller

docs/features/ulfm.rst

@@ -120,12 +120,9 @@ runtime).
 Optionally, you can specify ``--with-ft ulfm`` to ensure that ULFM support
 is definitely built.
 
-Support notes
-^^^^^^^^^^^^^
-
-* ULFM Fault Tolerance does not apply to OpenSHMEM.  It is recomended
-  that if you are going to use ULFM, you should disable building
-  OpenSHMEM with ``--disable-oshmem``.
+.. note:: ULFM Fault Tolerance does not apply to OpenSHMEM.  It is recomended
+   that if you are going to use ULFM, you should disable building OpenSHMEM
+   with ``--disable-oshmem``.
 
 Running ULFM Open MPI
 ---------------------
@@ -151,30 +148,33 @@ the normal Open MPI ``mpirun`` launcher, with the
 
    shell$ mpirun --with-ft ulfm ...
 
-  .. important:: By default, fault tolerance is not active at run time.
-                 It must be enabled via `--with-ft ulfm`.
+.. important:: By default, fault tolerance is not active at run time.
+   It must be enabled via ``--with-ft ulfm``.
 
 Running under a batch scheduler
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 ULFM can operate under a job/batch scheduler, and is tested routinely
 with ALPS, PBS, and Slurm. One difficulty comes from the fact that
-many job schedulers will "cleanup" the application as soon as any
-process fails. In order to avoid this problem, it is preferred that
-you use ``mpirun`` within an allocation (e.g., ``salloc``,
-``sbatch``, ``qsub``) rather than a direct launch (e.g., ``srun``).
+many job schedulers handle failures by triggering an immediate "cleanup"
+of the application as soon as any process fails. In addition, failure
+detection subsystems integrated into PRTE are not active in direct launch
+scenarios and may not have a launcher specific alternative. This may cause
+the application to not detect failures and lock. In order to avoid these
+problems, it is preferred that you use ``mpirun`` within an allocation
+(e.g., ``salloc``, ``sbatch``, ``qsub``) rather than a direct launch.
 
 * SLURM is tested and supported with fault tolerance.
 
-  .. important:: Do not use ``srun``, or your application gets killed
-                 by the scheduler upon the first failure.  Instead,
-                 use ``mpirun`` in an ``salloc`` or ``sbatch`` allocation.
-
-* LSF is untested with fault tolerance.
+  .. important:: Use ``mpirun`` in an ``salloc`` or ``sbatch`` allocation.
+     Direct launch with ``srun`` is not supported.
 
 * PBS/Torque is tested and supported with fault tolerance.
 
-  .. important:: Be sure to use ``mpirun`` in a ``qsub`` allocation.
+  .. important:: Use ``mpirun`` in a ``qsub`` allocation. Direct launch
+     with ``aprun`` is not supported.
+
+* LSF is untested with fault tolerance.
 
 Run-time tuning knobs
 ^^^^^^^^^^^^^^^^^^^^^
@@ -185,13 +185,12 @@ most cases. You can change the default settings with ``--mca
 mpi_ft_foo <value>`` for Open MPI options, and with ``--prtemca
 errmgr_detector_bar <value>`` for PRTE options.
 
-  .. important:: The main control for enabling/disabling fault tolerance
-                 at runtime is the ``--with-ft ulfm`` (or its synomym
-                 ``--with-ft mpi``) ``mpirun`` CLI option. This option
-                 sets up multiple subsystems in Open MPI to enable fault
-                 tolerance. The options described below are best used to
-                 overide the default behavior after the ``--with-ft ulfm``
-                 opion is used.
+.. important:: The main control for enabling/disabling fault tolerance
+   at runtime is the ``--with-ft ulfm`` (or its synomym ``--with-ft mpi``)
+   ``mpirun`` CLI option. This option sets up multiple subsystems in
+   Open MPI to enable fault tolerance. The options described below are
+   best used to overide the default behavior after the ``--with-ft ulfm``
+   opion is used.
 
 PRTE level options
 ~~~~~~~~~~~~~~~~~~
@@ -225,12 +224,11 @@ runtime behavior of ULFM by either setting or unsetting variables in
 this file, or by overiding the variable on the command line (e.g.,
 ``--mca btl ofi,self``).
 
-  .. important:: Note that if fault tolerance is disabled at runtime,
-                 (that is, when not using ``--with-ft ulfm``), the
-                 ``ft-mpi`` AMCA param file is not loaded, thus
-                 components that are unsafe for fault tolerance will
-                 load normally (this may change observed performance
-                 when comparing with and without fault tolerance).
+.. important:: Note that if fault tolerance is disabled at runtime,
+   (that is, when not using ``--with-ft ulfm``), the ``ft-mpi`` AMCA
+   param file is not loaded, thus components that are unsafe for fault
+   tolerance will load normally (this may change observed performance
+   when comparing with and without fault tolerance).
 
 * ``mpi_ft_enable <true|false> (default: false)``
   permits turning on/off fault tolerance at runtime. This option is
@@ -254,23 +252,33 @@ this file, or by overiding the variable on the command line (e.g.,
   ``MPI_COMM_WORLD`` exclusively.  Processes connected from
   ``MPI_COMM_CONNECT``/``ACCEPT`` and ``MPI_COMM_SPAWN`` may
   occasionally not be detected when they fail.
+
+  .. caution:: This component is deprecated. Failure detection is now
+     performed at the PRTE level. See the section above on controlling
+     PRTE behavior for information about how to tune the failure detector.
+
 * ``mpi_ft_detector_thread <true|false> (default: false)`` controls
   the use of a thread to emit and receive failure detector's
   heartbeats. *Setting this value to "true" will also set
   MPI_THREAD_MULTIPLE support, which has a noticeable effect on
   latency (typically 1us increase).* You may want to **enable this
   option if you experience false positive** processes incorrectly
   reported as failed with the Open MPI failure detector.
-  This option is only relevant when ``mpi_ft_detector`` is ``true``.
+
+  .. important:: This option is only relevant when ``mpi_ft_detector`` is ``true``.
+
 * ``mpi_ft_detector_period <float> (default: 3e0 seconds)`` heartbeat
   period. Recommended value is 1/3 of the timeout. _Values lower than
   100us may impart a noticeable effect on latency (typically a 3us
   increase)._
-  This option is only relevant when ``mpi_ft_detector`` is ``true``.
+
+  .. important:: This option is only relevant when ``mpi_ft_detector`` is ``true``.
+
 * ``mpi_ft_detector_timeout <float> (default: 1e1 seconds)`` heartbeat
   timeout (i.e. failure detection speed). Recommended value is 3 times
   the heartbeat period.
-  This option is only relevant when ``mpi_ft_detector`` is ``true``.
+
+  .. important:: This option is only relevant when ``mpi_ft_detector`` is ``true``.
 
 Known Limitations in ULFM
 -------------------------
@@ -287,27 +295,27 @@ Frameworks and components are listed below and categorized into one of
 three classifications:
 
 1. **Modified:** This framework/component has been specifically modified
-                 such that it will continue to work after a failure.
+   such that it will continue to work after a failure.
 2. **Untested:** This framework/component has not been modified and/or
-                 tested with fault tolerance scenarios, and _may_ malfunction
-                 after a failure.
+   tested with fault tolerance scenarios, and _may_ malfunction
+   after a failure.
 3. **Disabled:** This framework/component will cause unspecified behavior when
-                 fault tolerance is enabled.
+   fault tolerance is enabled.
 
 Any framework or component not listed below are categorized as **Unmodified**,
 meaning that it is unmodified for fault tolerance, but will continue to work
 correctly after a failure.
 
 * ``pml``: MPI point-to-point management layer
 
-  * ``monitoring``, ``v``: **untested** (they have not been modified
-    to handle faults)
+  * ``monitoring``, ``v``: **untested** (they have not been modified to handle
+    faults)
   * ``cm``, ``crcpw``, ``ucx``: **disabled**
 
 * ``btl``: Point-to-point Byte Transfer Layer
 
-  * ``ofi``, ``portals4``, ``smcuda``, ``usnic``, ``sm(+knem)``:
-    **untested** (they may work properly, please report)
+  * ``ofi``, ``portals4``, ``smcuda``, ``usnic``, ``sm(+knem)``: **untested**
+    (they may work properly, please report)
 
 * ``mtl``: Matching transport layer Used for MPI point-to-point messages on
   some types of networks