Document which pytest features work with unittest

Fix #2626

Author: nicoddemus

changelog/2626.doc

@@ -0,0 +1 @@
+Explicitly document which pytest features work with ``unittest``.

doc/en/mark.rst

@@ -10,6 +10,7 @@ By using the ``pytest.mark`` helper you can easily set
 metadata on your test functions. There are
 some builtin markers, for example:
 
+* :ref:`skip <skip>` - always skip a test function
 * :ref:`skipif <skipif>` - skip a test function if a certain condition is met
 * :ref:`xfail <xfail>` - produce an "expected failure" outcome if a certain
   condition is met

doc/en/skipping.rst

@@ -27,6 +27,7 @@ corresponding to the "short" letters shown in the test progress::
 (See :ref:`how to change command line options defaults`)
 
 .. _skipif:
+.. _skip:
 .. _`condition booleans`:
 
 Skipping test functions

doc/en/unittest.rst

@@ -2,58 +2,80 @@
 .. _`unittest.TestCase`:
 .. _`unittest`:
 
-Support for unittest.TestCase / Integration of fixtures
-=====================================================================
-
-.. _`unittest.py style`: http://docs.python.org/library/unittest.html
-
-``pytest`` has support for running Python `unittest.py style`_ tests.
-It's meant for leveraging existing unittest-style projects
-to use pytest features.  Concretely, pytest will automatically 
-collect ``unittest.TestCase`` subclasses and their ``test`` methods in
-test files.  It will invoke typical setup/teardown methods and 
-generally try to make test suites written to run on unittest, to also 
-run using ``pytest``.  We assume here that you are familiar with writing
-``unittest.TestCase`` style tests and rather focus on 
-integration aspects.
-
-Note that this is meant as a provisional way of running your test code 
-until you fully convert to pytest-style tests. To fully take advantage of
-:ref:`fixtures <fixture>`, :ref:`parametrization <parametrize>` and 
-:ref:`hooks <writing-plugins>` you should convert (tools like `unittest2pytest 
-<https://pypi.python.org/pypi/unittest2pytest/>`__ are helpful). 
-Also, not all 3rd party pluging are expected to work best with 
-``unittest.TestCase`` style tests.
-
-Usage
--------------------------------------------------------------------
-
-After :ref:`installation` type::
-
-    pytest
-
-and you should be able to run your unittest-style tests if they
-are contained in ``test_*`` modules.  If that works for you then
-you can make use of most :ref:`pytest features <features>`, for example
-``--pdb`` debugging in failures, using :ref:`plain assert-statements <assert>`,
-:ref:`more informative tracebacks <tbreportdemo>`, stdout-capturing or 
-distributing tests to multiple CPUs via the ``-nNUM`` option if you 
-installed the ``pytest-xdist`` plugin.  Please refer to
-the general ``pytest`` documentation for many more examples.
+unittest.TestCase Support
+=========================
 
-.. note::
+.. _`unittest.py style`: https://docs.python.org/3/library/unittest.html
 
-    Running tests from ``unittest.TestCase`` subclasses with ``--pdb`` will
-    disable tearDown and cleanup methods for the case that an Exception
-    occurs. This allows proper post mortem debugging for all applications
-    which have significant logic in their tearDown machinery. However,
-    supporting this feature has the following side effect: If people
-    overwrite ``unittest.TestCase`` ``__call__`` or ``run``, they need to 
-    to overwrite ``debug`` in the same way  (this is also true for standard
-    unittest).
+``pytest`` supports running Python `unittest.py style`_ tests out of the box.
+It's meant for leveraging existing ``unittest``-style test suites
+to use pytest as a test runner and also allow to incrementally adapt
+the test suite to take full advantage of pytest features.
+
+To run an existing ``unittest``-style test suite using ``pytest``, simply type::
+
+    pytest tests
+
+
+pytest will automatically collect ``unittest.TestCase`` subclasses and
+their ``test`` methods in ``test_*.py`` or ``*_test.py`` files.
+
+Almost all ``unittest`` features are supported, like ``@unittest.skip`` style decorators, ``setUp/tearDown``,
+``setUpClass/tearDownClass()``, etc (see :ref:`unittest-limitations`).
+
+Benefits
+--------
+
+You can make use of several pytest features, most without changing any existing code:
+
+* Use :ref:`plain assert-statements <assert>` instead of ``self.assert*`` functions (`unittest2pytest
+  <https://pypi.python.org/pypi/unittest2pytest/>`__ is immensely helpful in this);
+* Obtain :ref:`more informative tracebacks <tbreportdemo>`;
+* :ref:`stdout and stderr <captures>` capturing;
+* :ref:`Test selection options <select-tests>` using ``-k`` and ``-m`` flags;
+* :ref:`maxfail`;
+* :ref:`--pdb <pdb-option>` command-line option for debugging on test failures
+  (see :ref:`note <pdb-unittest-note>` below);
+* Distribute tests to multiple CPUs using the `pytest-xdist <http://pypi.python.org/pypi/pytest-xdist>`_ plugin;
+
+.. _unittest-limitations:
+
+Limitations
+-----------
+
+.. _`load_tests protocol`: https://docs.python.org/3/library/unittest.html#load-tests-protocol
+.. _`setUpModule/tearDownModule`: https://docs.python.org/3/library/unittest.html#setupmodule-and-teardownmodule
+.. _`subtests`: https://docs.python.org/3/library/unittest.html#distinguishing-test-iterations-using-subtests
+
+Pytest currently does not support the following ``unittest`` features:
+
+* `load_tests protocol`_;
+* `setUpModule/tearDownModule`_;
+* `subtests`_;
 
-Mixing pytest fixtures into unittest.TestCase style tests
------------------------------------------------------------
+
+pytest features in ``unittest.TestCase`` subclasses
+---------------------------------------------------
+
+The following pytest features work in ``unittest.TestCase`` subclasses:
+
+* :ref:`Marks <mark>`: :ref:`skip <skip>`, :ref:`skipif <skipif>`, :ref:`xfail <xfail>`;
+* :ref:`Auto-use fixtures <mixing-fixtures>`;
+
+The following pytest features **do not** work, and probably
+never will due to different design philosophies:
+
+* :ref:`Fixtures <fixture>` (except for ``autouse`` fixtures, see :ref:`below <mixing-fixtures>`);
+* :ref:`Parametrization <parametrize>`;
+* :ref:`Custom hooks <writing-plugins>`;
+
+
+Third party plugins may or may not work well, depending on the plugin and the test suite.
+
+.. _mixing-fixtures:
+
+Mixing pytest fixtures into ``unittest.TestCase`` subclasses using marks
+------------------------------------------------------------------------
 
 Running your unittest with ``pytest`` allows you to use its
 :ref:`fixture mechanism <fixture>` with ``unittest.TestCase`` style
@@ -143,8 +165,8 @@ share the same ``self.db`` instance which was our intention
 when writing the class-scoped fixture function above.
 
 
-autouse fixtures and accessing other fixtures
--------------------------------------------------------------------
+Using autouse fixtures and accessing other fixtures
+---------------------------------------------------
 
 Although it's usually better to explicitly declare use of fixtures you need
 for a given test, you may sometimes want to have fixtures that are 
@@ -165,6 +187,7 @@ creation of a per-test temporary directory::
     import unittest
 
     class MyTest(unittest.TestCase):
+
         @pytest.fixture(autouse=True)
         def initdir(self, tmpdir):
             tmpdir.chdir() # change to pytest-provided temporary directory
@@ -200,3 +223,16 @@ was executed ahead of the ``test_method``.
 
    You can also gradually move away from subclassing from ``unittest.TestCase`` to *plain asserts*
    and then start to benefit from the full pytest feature set step by step.
+
+.. _pdb-unittest-note:
+
+.. note::
+
+    Running tests from ``unittest.TestCase`` subclasses with ``--pdb`` will
+    disable tearDown and cleanup methods for the case that an Exception
+    occurs. This allows proper post mortem debugging for all applications
+    which have significant logic in their tearDown machinery. However,
+    supporting this feature has the following side effect: If people
+    overwrite ``unittest.TestCase`` ``__call__`` or ``run``, they need to
+    to overwrite ``debug`` in the same way  (this is also true for standard
+    unittest).

doc/en/usage.rst

@@ -41,6 +41,8 @@ Getting help on version, option names, environment variables
     pytest -h | --help # show help on command line and config file options
 
 
+.. _maxfail:
+
 Stopping after the first (or N) failures
 ---------------------------------------------------
 
@@ -49,6 +51,8 @@ To stop the testing process after the first (N) failures::
     pytest -x            # stop after first failure
     pytest --maxfail=2    # stop after two failures
 
+.. _select-tests:
+
 Specifying tests / selecting tests
 ---------------------------------------------------
 
@@ -135,6 +139,9 @@ with Ctrl+C to find out where the tests are *hanging*. By default no output
 will be shown (because KeyboardInterrupt is caught by pytest). By using this
 option you make sure a trace is shown.
 
+
+.. _pdb-option:
+
 Dropping to PDB_ (Python Debugger) on failures
 -----------------------------------------------