TESTING

Copyright (c) 2012 The Chromium OS Authors. All rights reserved.
Use of this source code is governed by a BSD-style license that can be
found in the LICENSE file.

Intro
-----
We test shill using two sets of tests: unit tests, and integration
tests. The unit tests are built using Google Test [1] and Google Mock
[2]; the integration tests use autotest [3].

Running unit tests
------------------
Here ${BOARD} is a valid board name, like link or x86-generic.
- build the shill_unittest target
- run the resulting shill_unittest binary
- run the unit tests from your host machine under gdb
  (chroot)$ FEATURES=test emerge-${BOARD} platform2
  (chroot)$ gdb_x86_local --board ${BOARD} \
                /build/${BOARD}/var/cache/portage/chromeos-base/platform2/out/Default/shill_unittest
  (Of course if the unit tests segfaulted, you wouldn't need the emerge
  step since the build directory would have been retained in the course
  of the test failing.)
- The emerge workflow given above is incremental. It uses ninja to rebuild only
  relevant objects in the shill target.
- You can restrict the test runs to only shill unittests by using
  (chroot)$ P2_TEST_FILTER="shill::*" FEATURES=test emerge-${BOARD} platform2
  The filter can be made more specific to include googletest filters like
  "shill::CellularTest.StartGSMRegister"
- if you want to set a breakpoint in gdb, make sure to include the shill
  namespace. e.g., run
    (cros-gdb) b shill::EthernetService::GetStorageIdentifier
    Breakpoint 2 at 0x5555563cc270: file ethernet_service.cc, line 63.
  rather than
    (cros-gdb) b EthernetService::GetStorageIdentifier
    Function "EthernetService::GetStorageIdentifier" not defined.
    Make breakpoint pending on future shared library load? (y or [n]) n

- alternate build command:
  - Another way to build which uses the emerge command behind the scenes:
    (chroot)$ cros_workon_make --board ${BOARD} platform2
  - to see the actual compiler commands that are run:
    (chroot)$ CFLAGS="-print-cmdline" cros_workon_make --reconf \
                --board=${BOARD} platform2
  - to abort compilation on the first error
    (chroot)$ MAKEFLAGS="--stop" cros_workon_make --test --board=${BOARD} \
                --reconf platform2

Running integration tests
-------------------------
- build a test image, suitable for a VM:
  (chroot) src/scripts$ ./build_packages --board=${BOARD}
  (chroot) src/scripts$ ./build_image --board=${BOARD} \
                            --noenable_rootfs_verification test
  (chroot) src/scripts$ ./image_to_vm.sh --board=${BOARD} --test_image

- start the VM
  (host)$ sudo kvm -m 2048 -vga std -pidfile /tmp/kvm.pid \
              -net nic,model=virtio -net user,hostfwd=tcp::9222-:22 \
              -hda <path to chroot>/src/build/images/${BOARD}/latest/chromiumos_qemu_image.bin

- DO NOT log in on the console.
  (doing so will load a user profile onto shill's profile stack; this
  interferes with the test profile that we use in the autotests)

- if you've modified the source after building the image, update shill on
  the image, and then restart shill:
  (chroot) src/scripts$ ./start_devserver
  (chroot) src/scripts$ ssh-keygen -R '[127.0.0.1]:9222'
  (chroot) src/scripts$ emerge-${BOARD} platform2
  (chroot) src/scripts$ cros deploy 127.0.0.1:9222 platform2

  (chroot) src/scripts$ ssh -p 9222 root@127.0.0.1
  localhost / # restart shill

- run the tests
  (chroot) src/scripts$ test_that 127.0.0.1 WiFiManager
                                  --args="config_file=wifi_vm_config"
                                  --ssh_options="-p 9222"

  To run a specific test out of the test suite, use test_pat option to --args.
  # Example: To just run the 035CheckWEPKeySyntax test:
  (chroot) src/scripts$ test_that 127.0.0.1 WiFiManager
                                  --args="config_file=wifi_vm_config test_pat=035CheckWEPKeySyntax"
                                  --ssh_options="-p 9222"

- configuration note: if you use a different port
  (e.g. hostfwd=tcp::9223-:22), you'll need to change:
    - the ssh_port argument to test_that
    - the port numbers in
      <chroot>/third_party/autotest/files/client/config/wifi_vm_config

- debugging test failures
  - "grep shill /var/log/messages" for log messages
  - "grep wpa_supplicant /var/log/messages" for supplicant log messages
  - "wpa_debug debug" to increase supplicant log level
  - try resetting the test infrastructure
    - rmmod mac80211_hwsim mac80211 cfg80211
    - restart wpasupplicant
    - rm /tmp/hostapd-test.control/*
  - examine autotest log files
    - check how far the test got before it failed
      $ grep -a ': step ' <test output>/<suite name>/<suite name>.<test name>/debug/<suite name>.<test name>.INFO
      e.g.
      (chroot) $ grep -a ': step ' /tmp/test_that_latest/network_WiFiRoaming/network_WiFiRoaming.002Suspend/debug/network_WiFiRoaming.002Suspend.INFO
    - read the log file
      (chroot) $ LESSOPEN= less /tmp/test_that_latest/network_WiFiRoaming/network_WiFiRoaming.002Suspend/debug/network_WiFiRoaming.002Suspend.INFO

      (LESSOPEN= prevents less from misinterpreting the logs as binary files,
       and piping them through hexdump.)

- additional test suites: we have a number of other WiFi test suites
  (in addition to WiFiManager). these are: WiFiMatFunc, WiFiPerf,
  WiFiRoaming, WiFiSecMat. the WiFiPerf tests are probably not too
  relevant to shill (just yet), but the rest probably are.

[1] http://code.google.com/p/googletest/
[2] http://code.google.com/p/googlemock/
[3] http://autotest.kernel.org/,
    http://www.chromium.org/chromium-os/testing/testing-faq