Collecting code coverage data from IGT tests

Introduction

Ensuring that a test plan covers all the driver code is not trivial. Also, as time goes by, changes at both the tests and drivers may badly affect the code coverage. So, some tools are needed in order to be able to verify and improve the driver test coverage. While static analyzers can help checking the driver's code, it is not as effective as runtime tests.

Thankfully gcc has a feature which allows capturing such data in realtime, called gcov. LLVM/clang also has a similar feature (llvm-cov). Such feature is available at the Linux Kernel since 2009.

Building a Kernel with GCOV support

Enabling GCOV at the Linux Kernel requires two steps:

  1. Enable GCOV_KERNEL:

    ./scripts/config -e DEBUG_FS -e GCOV_KERNEL
    
  2. Enable per-driver or per-makefile GCOV support. In order to enable support for all DRM drivers:

    for i in $(find drivers/gpu/drm/ -name Makefile); do
        sed '1 a GCOV_PROFILE := y' -i $i
    done
    

When gcov is enabled for a given driver or directory, GCC will generate some special object files, like:

...
drivers/gpu/drm/drm_probe_helper.gcno
drivers/gpu/drm/drm_dp_dual_mode_helper.gcno
drivers/gpu/drm/drm_plane.gcno
drivers/gpu/drm/drm_lease.gcno
drivers/gpu/drm/drm_mipi_dsi.gcno
drivers/gpu/drm/drm_dsc.gcno
drivers/gpu/drm/drm_property.gcno
drivers/gpu/drm/drm_dp_aux_dev.gcno
drivers/gpu/drm/drm_blend.gcno
...

Those will be stored at the Kernel object directory, which is usually the same as the Kernel source directory, except if the Kernel was built with:

make O=kernel_output_dir

Such compile-time files are compiler-dependent and they're needed in order to properly decode the code coverage counters that will be produced in runtime.

Collecting GCOV data in runtime

Once a GCOV-enabled Kernel boots, the Kernel will keep track of the code monitored via GCOV under sysfs, at /sys/kernel/debug/gcov/.

There is a special file there: /sys/kernel/debug/gcov/reset. When something is written to it, all counters will be cleaned.

There are also driver-related counters and softlinks stored there:

ls -la /basedir/linux/drivers/gpu/drm/
...
-rw------- 1 root root 0 Feb 16 07:03 drm_probe_helper.gcda
lrwxrwxrwx 1 root root 0 Feb 16 07:03 drm_probe_helper.gcno -> /basedir/linux/drivers/gpu/drm/drm_probe_helper.gcno
-rw------- 1 root root 0 Feb 16 07:03 drm_property.gcda
lrwxrwxrwx 1 root root 0 Feb 16 07:03 drm_property.gcno -> /basedir/linux/drivers/gpu/drm/drm_property.gcno
-rw------- 1 root root 0 Feb 16 07:03 drm_rect.gcda
lrwxrwxrwx 1 root root 0 Feb 16 07:03 drm_rect.gcno -> /basedir/linux/drivers/gpu/drm/drm_rect.gcno
...

The actual counters are stored at the *.gcda files on a compiler-dependent format.

calling igt_runner directly

When code coverage support is enabled, the igt_runner tool will internally clean up the counters before starting test(s). Once test(s) finish, it will also run an external script that will be responsible for collecting the data and store on some file.

Enabling code coverage data collect can be done either per test or as a hole for an entire test list, by using those command line options:

  • --collect-code-cov

    Enables gcov-based collect of code coverage for tests.

  • --coverage-per-test

    Stores code coverage results per each test. This option implies --collect-code-cov.

For those options to work, it is mandatory to specifiy what script will be used to collect the data with --collect-script file_name.

calling ./scripts/run-tests.sh script

The run-tests.sh script can used instead as a frontend for igt_runner. It has the following options:

  • -c <capture_script>

    Capture gcov code coverage using the capture_script

  • -P

    Store code coverage results per each test.

  • -k kernel_dir

    Linux Kernel source code directory used to generate code coverage builds. This is passed through the capture script via the IGT_KERNEL_TREE shell environment variable.

So, for instance, if one wans to capture code coverage data from the Kernel that was built at the same machine, at the directory ~/linux, and wants to capture one file per test, it would use:

./scripts/run-tests.sh -T my.testlist -k ~/linux -c code_cov_capture -P

Code Coverage Collect script

While any script could in thesis be used, currently, there are two ones under the IGT's scripts/ source directory:

  • code_cov_capture:

    Assumes that the Kernel was built at the same machine, and uses the lcov tool to generate GCC-independent code coverage data, in the form of *.info files. Internally, it uses an shell environment variable (IGT_KERNEL_TREE), which points to the place where the Kernel source and objects are contained.

    Such script requires lcov tool to be installed at the test machine.

  • code_cov_gather_on_test:

    Generates a gzipped tarbal with the code coverage counters in binary format. Such kind of output should then be parsed at the same machine where the Kernel as built, as its content is not ony dependent on the Kernel source, but also on the Kernel output objects.

For each script, the igt_runner passes just one parameter: the results directory + the test name.

For instance, if it is needed to run a test called debugfs_test (read_all_entries) using code_cov_capture parameter, e. g.:

$ echo "igt@debugfs_test@read_all_entries" > my.testlist
$ ./scripts/run-tests.sh -T my.testlist -k ~/linux -c code_cov_capture -P
Found test list: "/basedir/igt/build/tests/test-list.txt"
[31410.499969] [1/1] debugfs_test (read_all_entries)
[31411.060446] Storing code coverage results...
[31418.01]     Code coverage wrote to /basedir/igt/results/code_cov/debugfs_test_read_all_entries.info
Done.

The script will be called as:

code_cov_capture results/code_cov/debugfs_test_read_all_entries

Please notice that any character that it is not a number nor a letter at the test name will be converted into ‘_’, as other characters are not supported as titles at the lcov files.

Passing extra arguments to the script

If any extra global parameters are needed by the script, those can be sent via shell's environment var.

Parsing data from code coverage *.info files

The *.info files generated by lcov are plain text files that list the tests that were executed in runtime.

The code_cov_parse_info script has some logic on it that allows printing the called functions stored inside the *.info file. It can also optionally apply the following filters. Its main options are:

  • --stat or --statistics

    Prints code coverage statistics.

    It displays function, line, branch and file coverage percentage.

    The statistics report is affected by the applied filters.

  • --print-coverage, --print or -p

    Prints the functions that were executed in runtime and how many times they were reached.

    The function coverage report is affected by the applied filters.

  • --print-unused or -u

    Prints the functions that were never reached.

    The function coverage report is affected by the applied filters.

  • --show-lines or --show_lines

    When printing per-function code coverage data, always output the source file and the line number where the function is defined.

  • --output output file or -o output file

    Produces an output file merging all input files.

    The generated output file is affected by the applied filters.

  • --show-files or --show_files

    Shows the list of files that were useed to produce the code coverage results.

  • It also has a set of parameters that filters the code coverage results: --only-drm, --only-i915, --func-filters, --source-filters, --ignore-unused. When used, all coverage displayed reports, and the stored output file will be affected by such filters.

More details can be seen by calling:

code_cov_parse_info --help

or:

code_cov_parse_info --man

The *.info file format

The *.info files contain several fields on it, grouped into records. An info file looks like:

TN:fbdev_eof
...
SF:/basedir/linux/drivers/gpu/drm/i915/intel_runtime_pm.c
...
FN:158,__intel_runtime_pm_get
FNDA:2,__intel_runtime_pm_get
...
end_of_record
SF:<some other file>
...
end_of_record
...

The main fields at the above record are:

  • TN: Test name
  • SF: Source file
  • FN: line_number function_name
  • FNDA: call_count function_name

So, the above example means that, inside drivers/gpu/drm/i915/intel_runtime_pm.c there's a function __intel_runtime_pm_get() which it was called 2 times.

Generating code coverage documentation

The lcov package contains the needed tools to parse and generate code coverage documentation. It is used by code_cov_capture script to convery from compiler-dependent *.gcno counters into a compiler-independent format (*.info).

Grouping multiple *.info files is as easy as running:

cat core*.info > all_core.info

The lcov package also contains a tool which converts a given *.info file into html patches, called genhtml.

As the output can actually show the code source file, genhtml need access not only to the info file, but also to the Kernel directory with the source files. Some optional arguments can be used at the command line, or can be stored at /etc/lcovrc or ~/.lcovrc files.

As generating the documentation depends wheather the results were generated as with a single or multiple *.info files by code_cov_capture or stored in raw formats inside *.tar.gz file(s) by code_cov_gather_on_test, there's a script that does all the required steps to build the code coverage html reports: code_cov_gen_report. Besides its own command line arguments, it also accepts arguments to be passed to code_cov_parse_info.

If a code_cov_parse_info command line parameter is passed, it will also call the script, in order to use a filtered *.info file to be used when generating the HTML reports.

It requires the following arguments:

  • --read file or dir (or -r file or dir)

    File or directory where the code coverage capture file(s) is(are) located.

  • --kernel-source dir (or -k dir)

    Kernel source directory.

  • --kernel-object dir (or -O dir)

    Kernel object directory. Only needed when Kernel was built with make O=dir.

  • --output-dir dir (or -o _dir)

    Directory where the html output will be stored. By default, the script won't let re-use an already existing directory.

  • --info

    The files specified by --read parameter are at lcov's *.info format.

  • --tar

    The files specified by --read are gzipped tarballs containing all *.gcno files and all *.gcda softlinks from the /sys/kernel/debug/gcov/ directory at the test machine, created by code_cov_gather_on_test script.

  • --force-override

    Allow using a non-empty directory for --output-dir.

  • It also accepts --print, --only-drm, --only-i915 and --ignore-unused options from code_cov_parse_info.

--info and --tar are mutually exclusive and at least one of them should be specified.

Code coverage capture script example

Capture, parse and generate code coverage html data

The script below provides a simple yet powerful script using code coverage capture on a test machine that also contains the Linux Kernel source and objects. It assumes that LGT was installed.

#/bin/bash -e

TESTLIST="my_tests.testlist"
OUT_DIR="${HOME}/results"

mkdir -p $OUT_DIR/html

echo "igt@debugfs_test@read_all_entries" > $TESTLIST
echo "igt@core_auth@basic-auth" >> $TESTLIST
echo "igt@gem_exec_basic@basic" >> $TESTLIST

sudo IGT_KERNEL_TREE="${HOME}/linux" igt_runner -s -o --coverage-per-test \
                  --collect-script code_cov_capture --test-list $TESTLIST \
                  /usr/local/libexec/igt-gpu-tools $OUT_DIR/ | sed s,$HOME/,,

sudo chown -R $(id -u):$(id -g) $OUT_DIR/

for i in $OUT_DIR/code_cov/*.info; do
        echo -e "\n$(basename $i):"
        code_cov_parse_info --only-drm --ignore-unused --stat $i
done
echo -e "\nTOTAL:"
code_cov_parse_info --only-drm --stat --output $OUT_DIR/results.info \
        $OUT_DIR/code_cov/*.info

cd $OUT_DIR/html
genhtml -q -s --legend --branch-coverage $OUT_DIR/results.info

Running such script produces the following output:

[3622.993304] [1/3] debugfs_test (read_all_entries)
[3631.95]     Code coverage wrote to results/code_cov/debugfs_test_read_all_entries.info
[3626.217016] Storing code coverage results...
[3631.957998] [2/3] core_auth (basic-auth)
[3638.03]     Code coverage wrote to results/code_cov/core_auth_basic_auth.info
[3632.116024] Storing code coverage results...
[3638.070869] [3/3] gem_exec_basic (basic)
[3644.24]     Code coverage wrote to results/code_cov/gem_exec_basic_basic.info
[3638.366790] Storing code coverage results...
Done.

core_auth_basic_auth.info:
  lines......: 11.7% (8217 of 70257 lines)
  functions..: 7.1% (776 of 10971 functions)
  branches...: 7.0% (3596 of 51041 branches)
Ignored......: non-drm headers and source files where none of its code ran.
Source files.: 23.27% (165 of 709 total), 29.57% (165 of 558 filtered)

debugfs_test_read_all_entries.info:
  lines......: 19.3% (20266 of 104802 lines)
  functions..: 17.5% (1922 of 10971 functions)
  branches...: 12.7% (9462 of 74555 branches)
Ignored......: non-drm headers and source files where none of its code ran.
Source files.: 34.70% (246 of 709 total), 44.09% (246 of 558 filtered)

gem_exec_basic_basic.info:
  lines......: 17.1% (14964 of 87503 lines)
  functions..: 13.0% (1422 of 10971 functions)
  branches...: 10.1% (6446 of 63758 branches)
Ignored......: non-drm headers and source files where none of its code ran.
Source files.: 30.89% (219 of 709 total), 39.25% (219 of 558 filtered)

TOTAL:
  lines......: 15.5% (25821 of 166849 lines)
  functions..: 22.1% (2429 of 10971 functions)
  branches...: 10.5% (11869 of 112665 branches)
Ignored......: non-drm headers.
Source files.: 78.70% (558 of 709 total)

Reporting detailed function coverage stored on *.info files

The code_cov_parse_info script can be used alone in order to provide a text file output containing code coverage data obtained from a *.info file. For example, listing code coverage usage for all functions whose name contains “edid_” can be done with:

$ echo edid_ >filter.txt
$ code_cov_parse_info --func-filters filter.txt results/results.info -p -u --stat
TEST: Code_coverage_tests
__drm_get_edid_firmware_path(): unused
__drm_set_edid_firmware_path(): unused
displayid_iter_edid_begin(): executed 10 times
drm_add_edid_modes(): executed 2 times
drm_add_override_edid_modes(): unused
drm_connector_attach_edid_property(): unused
drm_connector_update_edid_property(): executed 8 times
drm_dp_send_real_edid_checksum(): unused
drm_edid_are_equal(): executed 4 times
drm_edid_block_valid(): executed 8 times
drm_edid_duplicate(): unused
drm_edid_get_monitor_name(): unused
drm_edid_header_is_valid(): executed 4 times
drm_edid_is_valid(): executed 2 times
drm_edid_to_eld(): executed 2 times
drm_edid_to_sad(): unused
drm_edid_to_speaker_allocation(): unused
drm_find_edid_extension(): executed 22 times
drm_get_edid_switcheroo(): unused
drm_load_edid_firmware(): executed 2 times
edid_firmware_get(): unused
edid_firmware_set(): unused
edid_fixup_preferred(): unused
edid_get_quirks(): executed 6 times
edid_load(): unused
edid_open(): executed 4 times
edid_show() from linux/drivers/gpu/drm/drm_debugfs.c: executed 4 times
edid_show() from linux/drivers/gpu/drm/drm_sysfs.c: unused
edid_vendor(): executed 348 times
edid_write(): unused
intel_panel_edid_downclock_mode(): unused
intel_panel_edid_fixed_mode(): unused
is_edid_digital_input_dp(): unused
  lines......: 5.5% (5 of 91 lines)
  functions..: 42.4% (14 of 33 functions)
  branches...: 1.9% (1 of 52 branches)
Ignored......: unmatched functions m/(?^:edid_)/ and source files where none of its code ran.
Source files.: 0.90% (5 of 558 total), 55.56% (5 of 9 filtered)

When the function is unique, it will just display the function name and how many times the IGT test(s) executed it. When the same function name exists on multiple files (like the edid_show() on the above example), it will display multiple lines, one for each different function/file combination.

References

More information is available at Kernel gcov documentation: Using gcov with the Linux kernel.