This doc covers technical details for how we run unittests for non-native architectures. It is geared towards people who want a deeper understanding of the overall system. If you simply want to execute your tests, see the Chromium OS Unit Testing document instead.
Chromium OS has a number of interesting features:
All of these taken together lead to an interesting state: we have a large code base and want to test it, but how can we do it without resorting to real hardware? Unit tests can be compiled and run using the SDK and gives us confidence in the API, but we‘re still not testing the code that’ll run on the system (which has a number of implications). It also means we can't run fuller (integration) tests.
The answers to our needs is QEMU. It allows us to run Linux programs compiled for a different architecture as if it were native code. It does mean we rely on QEMU‘s emulation of an architecture matching real hardware, but that’s OK. If we find discrepancies, we can fix QEMU. In practice, the project is pretty stable.
There is also another semi-minor point that might occur to the more astute reader. What if the unittest wants to run another program itself? Say something as innocent as using the system() helper function? Won‘t that execute the host’s /bin/sh and perhaps try to run target programs too? Indeed it will, but our system below accounts for it transparently! We certainly want to avoid having to modify code under test in ways that only get executed when testing.
Note that we are not doing full system emulation here. In that mode, we start off by booting a full kernel, and then userspace, and then running programs. Here we are just executing the unittests directly, and QEMU takes care of converting system calls from the target to the host and vice versa.
The overall process boils down to:
We also have to keep in mind that our testsuite might be running in parallel with other testsuites and that a testsuite may have multiple unittests which also might be running in parallel. So we have to be careful to not mess with resources that are shared among the whole system otherwise things will be racy and randomly fail. Flaky tests are terrible.
Note: All the setup described below is handled by chromite's qemu python module now. The shell code below is more for reference.
As part of the Chromium OS build system, we always have a full sysroot available for a target board (created by build_packages). This is usually found in the /board/$BOARD/ path inside of the sdk. We need this anyways in order to compile packages for the target.
The sysroot is not generally directly bootable, but that doesn't matter to us. We just need to be able to chroot into it.
We ship a copy of QEMU inside the chroot already, and we make sure it is statically linked. This lets us easily copy it into a sysroot and run it inside of a chroot without having to copy any other files.
arch="arm" name="qemu-${arch}" qemu_src_path="/usr/bin/${name}" qemu_sysroot_path="/build/${BOARD}/build/bin/${name}" # Make sure the files are the same. This let's us easily update qemu in the # chroot without having to worry about keeping all the sysroots up-to-date. if [[ $(stat -c %i "${qemu_src_path}" 2>/dev/null) != $(stat -c %i "${qemu_sysroot_path}" 2>/dev/null) ]]; then # Do the update atomically. We'll hardlink to a unique path, and then use `mv` # a rename() for us. Since it's on the same device, we know it'll be atomic. ln -fT "${qemu_src_path}" "${qemu_sysroot_path}.$$" mv -fT "${qemu_sysroot_path}.$$" "${qemu_sysroot_path}" fi
The Linux kernel provides a nifty pseudo filesystems called binfmt_misc. This allows you to directly execute files that are normally not natively executable. For example, you could configure the system so that whenever you run a Windows exe program, it‘ll automatically invoke Wine for you. It simplifies the overall process. In our case, we’ll register handlers so that QEMU is automatically executed whenever we try to run one of these target ELFs.
First we mount the filesystem if it is not yet mounted:
$ mount -t binfmt_misc binfmt_misc /proc/sys/fs/binfmt_misc
If this errors out with EBUSY, we should check to see if it has been mounted already. This way parallel testsuites can safely execute this.
All handlers are registered by name (and the name is arbitrary). You can check by name by looking at /proc/sys/fs/binfmt_misc/<name>. If it already exists, we need to make sure it points to our interpreter. If it doesn‘t, we have to unregister & re-register it. This phase is a little racy, but should rarely (if ever) happen, so we’ll just ignore the issue here.
# This magic string registers the ARM ELF format and tells it to use qemu to run. arch="arm" name="qemu-${arch}" qemu_path="/build/bin/${name}" qemu_magic=":${arch}:M::\x7fELF\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x28\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:${qemu_path}:" binfmt_path="/proc/sys/fs/binfmt_misc/${name}" if [[ -e ${binfmt_path} ]]; then # Make sure the existing registration matches. if [[ $(awk '$1 == "interpreter" {print $NF}' "${binfmt_path}") != "${qemu_path}" ]]; then # Unregister it since there's a mismatch (maybe due to upgrade?). echo -1 | sudo tee "${binfmt_path}" fi fi if [[ -e ${binfmt_path} ]]; then # Register it since it doesn't yet exist. # We should ignore errors EBUSY at this point in case we run in parallel. echo "${qemu_magic}" | sudo tee "/proc/sys/fs/binfmt_misc/register" fi
Linux programs often want some basic filesystems available in order to run. We bind mount them here to simplify things.
Normally the process would be to do these in a setup script that'd tail into running the unittests. That way we can use the unshare utility to run the unittest in its own mount namespace which allows us to create as many mounts as we want without impacting other processes (like unittests running in parallel).
#!/bin/sh # Make sure they've told us where to set things up. if [ -z "${SYSROOT}" ]; then echo "You must set \$SYSROOT first" exit 1 fi # Re-exec ourselves as root in a new mount namespace. [ -z "${_UNSHARE}" ] && exec sudo unshare -m -- "$0" "$@" # Do all the bind mounts we need. mount -n --bind /proc "${SYSROOT}"/proc mount -n --bind /dev "${SYSROOT}"/dev mount -n --bind /dev/pts "${SYSROOT}"/dev/pts mount -n --bind /sys "${SYSROOT}"/sys # Run the unittest now. exec chroot "${SYSROOT}" -- "$@"
Since the Chromium OS build system builds all packages inside of the sysroot already (typically /build/$BOARD/tmp/portage/...), we already have the files available to us. We simply need to use paths relative to inside the sysroot.
Using the script from above, we can now run unittests inside the sysroot. It'll take care of the sysroots and stuff for us.
$ ./helper-script.sh /tmp/portage/.../unittest
