Testing Suite
========================================================================================
|coverage|

.. include:: ../README.rst
   :start-after: begin_badges
   :end-before:  end_badges

Exhale uses `pytest <https://docs.pytest.org/en/latest/>`_ for its testing suite.  To
run the tests, Exhale uses `tox <https://tox.wiki/en/latest/>`_.

.. code-block:: console

   $ cd /path/to/exhale
   $ pip install tox
   $ tox

.. note::

   See ``tox.ini`` in the root of the repository.  You only need to install ``tox`` to
   be able to run the tests, which will internally download the additional dependencies
   such as ``pytest``.  The tests will be run in a virtual environment.

Running Specific Tests
----------------------------------------------------------------------------------------

By default, ``tox`` will run the python tests and linting checks with
`flake8 <https://flake8.pycqa.org/en/latest/>`_.  More formally, the default environment
list is defined as:

.. code-block:: ini

   [testenv]
   envlist = py, flake8

This means that the version of python the tests are run with are the interpreter that
you installed ``tox`` for.  To run a specific test:

``tox -e py``
    Run the python unit tests.

``tox -e flake8``
    Run the lint tests.

``tox -e docs``
    Build the sphinx documentation using the *html* builder (in nitpicky mode).  You can
    view the generated html website with ``open .tox/docs/tmp/html/index.html``.

``tox -e linkcheck``
    Build the sphinx documentation using the *linkcheck* builder.

.. tip::

   If you need to debug a test case to discern why it is failing, the vibrantly colorful
   `ipdb <https://pypi.org/project/ipdb/>`_ debugger is already installed in the
   environment running the tests.  It is nearly identical to ``pdb``, and although it
   is designed for IPython, it works just as well in "regular" code.  Just like with
   ``pdb``, set a trace in the test case you are debugging:

   .. code-block:: py

      import ipdb
      ipdb.set_trace()

   In order to be able to achieve the break-point, you would launch the tests with

   .. code-block:: console

      $ tox -e py -- -s

   The ``--`` signals that the command line arguments for ``tox`` are complete, and
   everything afterward should be forwarded to the specified environment (in this case
   to ``pytest`` since we launched ``-e py``).  The ``-s`` switch for ``pytest`` enables
   the debugger to stop the test, without this flag the test will fail at the
   break-point.

   The same goes for other test cases, if you need to specify a flag to ``flake8`` for
   some reason you would run ``tox -e flake8 -- --ignore XYYY`` where ``XYYY`` is the
   linter check you want to ignore.

Writing Project Test Cases
----------------------------------------------------------------------------------------

``exhale`` provides tools to build tests that need to generate output files (= ``*.rst``
files) from actual test projects. To create such tests:

1. Create a new test-case folder in ``testing/projects/{new project name}``.

2. Inherit from :class:`testing.base.ExhaleTestCase` in a module you create
   in the ``testing/tests`` folder, setting the ``test_project`` class-level variable
   to ``{new project name}``

3. Add some ``test_{something}`` methods. :class:`testing.base.ExhaleTestCase` will
   work its magic to apply the necessary pytest fixtures to all these methods so that
   the configuration is applied, ``sphinx`` is bootstrapped and all ``*.rst`` files
   generated by ``exhale`` are in the specified output directory at the start of your
   test. Note that the ``sphinx`` application is made available as ``self.app``.

4. In case you need to run tests with a configuration differing from the default
   configuration, you can use the :func:`testing.decorators.confoverrides` decorator
   to tweak the configuration for a test method or the whole test class::

      @confoverrides(conf_var1=conv_val1, conf_var2=conf_val2)
      class MyTestCase(ExhaleTestCase):
          test_project = 'my_project'

   or::

      @confoverrides(conf_var1=conv_val1, conf_var2=conf_val2)
      def test_something(self):
          ...

   Note that ``@confoverrides`` applied to a test method are combined with
   ``@confoverrides`` applied to test classes and have precedence.

5. In case you don't want ``exhale`` to generate any files and just need to test the
   configuration, you can use the :func:`testing.decorators.no_run` on a test
   method or test class::

      @no_run
      class MyTestCase(ExhaleTestCase):
          test_project = 'my_project'

   or::

      @no_run
      def test_something_else(self):
          ...

For more examples, just have a look at the existing tests in the ``testing/tests``
folder.


Full Testing Suite Documentation
----------------------------------------------------------------------------------------

..

    A **HUGE** thank you to Thomas Khyn for his thorough help figuring out how to
    coordinate ``pytest``, ``sphinx``, metaclasses, and more -- this testing framework
    would never have come to fruition without his
    `epic pull request <https://github.com/svenevs/exhale/pull/31>`_.

.. automodule:: testing
   :members:

.. toctree::
   :maxdepth: 2

   testing/base
   testing/conftest
   testing/decorators
   testing/fixtures
   testing/hierarchies
   testing/projects
   testing/utils
   testing/tests
