doc/test_api.rst

Test API
========

Overview
--------
This document describes how to write new factory tests and integrate
them into the testing framework.

Basics
------
Factory tests are implemented as Python unit tests. Each factory test
is a subclass of the `unittest.TestCase
<https://docs.python.org/2/library/unittest.html#unittest.TestCase>`_
class, and may make full use of the Python ``unittest`` API (e.g., all
the ``assert`` methods, ``fail``, ``setUp``, ``tearDown``, etc.). The
factory SDK also provides APIs that can be used to parse test
arguments (see `Test arguments`_), and to interact with the test UI
running in the browser (see :ref:`test-ui-api`).

Each factory test should at minimum contain

* a ``runTest`` method
  containing the test implementation (see `Test implementation`_), and
* an ``ARGS`` attribute describing the arguments that may be used
  by the test list when invoking the test (see `Test arguments`_).

Where to put your test
----------------------
Each factory has a name (like ``bad_blocks`` or ``lcd_backlight``).
For simple tests containing only a Python file, put the test code
in

  :samp:`py/test/pytests/{name}.py`

Alternatively, if your test has accompanying files (such as HTML or
JavaScript files), you may create a directory for it and put the code
in

  :samp:`py/test/pytests/{name}/{name}.py`.

Within this file, there should be a single subclass of `unittest.TestCase`.
There can be only one test class per module.

Test implementation
-------------------
Your test should contain a ``runTest`` method containing the body of
the test.  (As with all Python unittests, it may also contain a
``setUp`` and ``tearDown`` method.  ``setUp`` is run first, and
``tearDown`` is run after the test passes or fails; although note
that ``tearDown`` may not be run if the test is forcibly stopped.)

The test is considered to have succeeded if the ``runTest`` method
returns.  The test will fail if:

* your test calls any of the ``unittest.TestCase.assertXXX`` methods,
  such as `assertTrue
  <https://docs.python.org/2/library/unittest.html#unittest.TestCase.assertTrue>`_,
  and the assertion fails. For example, this will cause your test to
  fail with a good error message if a file does not exist as expected:

    self.assertTrue(
      os.path.exists(self.args.path),
      'File %r does not exist' % self.args.path)

* your test calls `self.fail
  <https://docs.python.org/2/library/unittest.html#unittest.TestCase.fail>`,
  which tells the Python unit test framework to raise an exception.

* your test directly raises an exception with ``raise``.

* code that you call directly or indirectly raises an exception. For
  example, if you call ``subprocess.check_call``, and ``check_call``
  raises a ``CalledProcessError`` that you do not catch, your test
  will fail. If you want such a failure in ``check_call`` to cause
  your test to fail, you can choose to not catch the exception and let
  it propagate out of your ``runTest`` method.

There are several key APIs you will need to understand to write your
test:

* `Test arguments`_ allow your test to handle arguments specified
  in test lists.

* :ref:`test-ui-api` allows your test to provide a UI to interact with
  the operator or show status messages.  If your test is simple and
  does not require interaction with the user, you may choose not to
  provide a UI.

Test arguments
--------------
Test lists need to customize the behavior of tests in various
situations, e.g., to specify different limits or parameters or to
enable/disable various checks. To allow this sort of customization,
you can declare test arguments in your test case by adding an ``ARGS``
attribute describing the supported set of arguments. ``ARGS`` is a
list of items of type :py:class:`cros.factory.test.args.Arg`.

For example::

  import unittest
  from cros.factory.test.args import Arg

  class BadBlocksTest(unittest.TestCase):
    ARGS = [
      Arg('path', str, 'The path to a temporary file to use for testing.'),
      Arg('max_bytes', int, 'Maximum size to test, in bytes.',
          default=16*1024*1024),
    ]

This declares two arguments: ``path`` is a required string, and
``max_bytes`` is an optional number defaulting to 16 megabytes.

The test list might contain an entry like::

  FactoryTest(
      id='BadBlocks',
      pytest_name='bad_blocks',
      dargs={'path': '/usr/local/foo',
             'max_bytes': 8*1024*1024}
  )

The factory test runner will check that:

* all the arguments in the test list are valid arguments (e.g.,
  you don't accidentally specify a ``filename`` argument, since
  ``filename`` is not declared in ``ARGS``).
* all required arguments (in this case ``path``) are specified.
* all the arguments are of the correct type (e.g., you don't say
  ``max_bytes='foo'``, since ``max_bytes`` must be an ``int``).

.. py:module:: cros.factory.test.args

.. autoclass:: Arg

   .. automethod:: __init__

Once you have declared the arguments used by your test,
you can use ``self.args`` anywhere in your test implementation
to refer to the value of that argument.  For example::

  class BadBlocksTest(unittest.TestCase)::
    ...  # see above for ARGS = [...] declaration

    def runTest(self):
      logging.info('path=%s, max_bytes=%d',
                   self.args.path, self.args.max_bytes)