| Test API |
| ======== |
| |
| Overview |
| -------- |
| This document describes how to write new factory tests and integrate |
| them into the testing framework. |
| |
| Basics |
| ------ |
| Factory tests are implemented as Python unit tests. Each factory test |
| is a subclass of the `unittest.TestCase |
| <https://docs.python.org/2/library/unittest.html#unittest.TestCase>`_ |
| class, and may make full use of the Python ``unittest`` API (e.g., all |
| the ``assert`` methods, ``fail``, ``setUp``, ``tearDown``, etc.). The |
| factory SDK also provides APIs that can be used to parse test |
| arguments (see `Test arguments`_), and to interact with the test UI |
| running in the browser (see :ref:`test-ui-api`). |
| |
| Each factory test should at minimum contain |
| |
| * a ``runTest`` method |
| containing the test implementation (see `Test implementation`_), and |
| * an ``ARGS`` attribute describing the arguments that may be used |
| by the test list when invoking the test (see `Test arguments`_). |
| |
| Where to put your test |
| ---------------------- |
| Each factory has a name (like ``bad_blocks`` or ``lcd_backlight``). |
| For simple tests containing only a Python file, put the test code |
| in |
| |
| :samp:`py/test/pytests/{name}.py` |
| |
| Alternatively, if your test has accompanying files (such as HTML or |
| JavaScript files), you may create a directory for it and put the code |
| in |
| |
| :samp:`py/test/pytests/{name}/{name}.py`. [*]_ |
| |
| Within this file, there should be a single subclass of `unittest.TestCase`. |
| There can be only one test class per module, so if you need to have different |
| test cases, you need to separate them into different modules. But you can create |
| a directory for them, for example: |
| |
| - :samp:`py/test/pytests/{dir_name}/{test1}.py` |
| - :samp:`py/test/pytests/{dir_name}/{test2}.py` |
| - :samp:`py/test/pytests/{dir_name}/{test3}.py`. |
| |
| In this case, the name of your tests are :samp:`{dir_name}.{test1}`, |
| :samp:`{dir_name}.{test2}` and :samp:`{dir_name}.{test3}`. |
| |
| To know more about how we load a pytest from name, please refer to |
| :py:func:`cros.factory.test.utils.pytest_utils.LoadPytestModule`. |
| |
| .. [*] Since the ``__init__.py`` of a package will be loaded whenever its |
| submodule is imported. We recommand keeping ``__init__.py`` empty to |
| prevent longer loading time if ``__init__.py`` is too large. |
| |
| Test implementation |
| ------------------- |
| Your test should contain a ``runTest`` method containing the body of |
| the test. (As with all Python unittests, it may also contain a |
| ``setUp`` and ``tearDown`` method. ``setUp`` is run first, and |
| ``tearDown`` is run after the test passes or fails; although note |
| that ``tearDown`` may not be run if the test is forcibly stopped.) |
| |
| The test is considered to have succeeded if the ``runTest`` method |
| returns. The test will fail if: |
| |
| * your test calls any of the ``unittest.TestCase.assertXXX`` methods, |
| such as `assertTrue |
| <https://docs.python.org/2/library/unittest.html#unittest.TestCase.assertTrue>`_, |
| and the assertion fails. For example, this will cause your test to |
| fail with a good error message if a file does not exist as expected: |
| |
| self.assertTrue( |
| os.path.exists(self.args.path), |
| 'File %r does not exist' % self.args.path) |
| |
| * your test calls `self.fail |
| <https://docs.python.org/2/library/unittest.html#unittest.TestCase.fail>`, |
| which tells the Python unit test framework to raise an exception. |
| |
| * your test directly raises an exception with ``raise``. |
| |
| * code that you call directly or indirectly raises an exception. For |
| example, if you call ``subprocess.check_call``, and ``check_call`` |
| raises a ``CalledProcessError`` that you do not catch, your test |
| will fail. If you want such a failure in ``check_call`` to cause |
| your test to fail, you can choose to not catch the exception and let |
| it propagate out of your ``runTest`` method. |
| |
| There are several key APIs you will need to understand to write your |
| test: |
| |
| * `Test arguments`_ allow your test to handle arguments specified |
| in test lists. |
| |
| * :ref:`test-ui-api` allows your test to provide a UI to interact with |
| the operator or show status messages. If your test is simple and |
| does not require interaction with the user, you may choose not to |
| provide a UI. |
| |
| Test arguments |
| -------------- |
| Test lists need to customize the behavior of tests in various |
| situations, e.g., to specify different limits or parameters or to |
| enable/disable various checks. To allow this sort of customization, |
| you can declare test arguments in your test case by adding an ``ARGS`` |
| attribute describing the supported set of arguments. ``ARGS`` is a |
| list of items of type :py:class:`cros.factory.utils.arg_utils.Arg`. |
| |
| For example:: |
| |
| import unittest |
| from cros.factory.utils.arg_utils import Arg |
| |
| class BadBlocksTest(unittest.TestCase): |
| ARGS = [ |
| Arg('path', str, 'The path to a temporary file to use for testing.'), |
| Arg('max_bytes', int, 'Maximum size to test, in bytes.', |
| default=16*1024*1024), |
| ] |
| |
| This declares two arguments: ``path`` is a required string, and |
| ``max_bytes`` is an optional number defaulting to 16 megabytes. |
| |
| The test list might contain an entry like:: |
| |
| { |
| "pytest_name": "bad_blocks", |
| "args": { |
| "path": "/usr/local/foo", |
| "max_bytes": 8388608 |
| } |
| } |
| |
| The factory test runner will check that: |
| |
| * all the arguments in the test list are valid arguments (e.g., |
| you don't accidentally specify a ``filename`` argument, since |
| ``filename`` is not declared in ``ARGS``). |
| * all required arguments (in this case ``path``) are specified. |
| * all the arguments are of the correct type (e.g., you don't say |
| ``max_bytes='foo'``, since ``max_bytes`` must be an ``int``). |
| |
| .. py:module:: cros.factory.utils.arg_utils |
| |
| .. autoclass:: Arg |
| |
| .. automethod:: __init__ |
| |
| Once you have declared the arguments used by your test, |
| you can use ``self.args`` anywhere in your test implementation |
| to refer to the value of that argument. For example:: |
| |
| class BadBlocksTest(unittest.TestCase):: |
| ... # see above for ARGS = [...] declaration |
| |
| def runTest(self): |
| logging.info('path=%s, max_bytes=%d', |
| self.args.path, self.args.max_bytes) |
| |
| .. py:module:: cros.factory.test.utils.pytest_utils |
| |
| .. autofunction:: LoadPytestModule |