breadcrumbs: Chromium OS Build > page_name: cbuildbot-overview title: Anatomy of a cbuildbot build
Introduction
This page describes the basic steps involved in a cbuildbot build. It is intended for people who are interested in learning more about how cbuildbot works.
If you're just looking to use cbuildbot, please see our other documentation:
Simplified Build Flow Diagram
The following diagram is a simplified view of a typical build:
In the test/archive phase of cbuildbot, every stage we launch runs in parallel. In the above example, there are 5 stages running in parallel (HWTest, UnitTest, UploadPrebuilts, VMTest, Archive).
With the exception of the build stages, each step uploads its own artifacts.
Build Overview
The following steps are run in order.
InitSDK: Create or update the chroot
If the chroot does not exist, or we find the chroot is broken in any way (e.g. we can’t enter the chroot with cros_sdk), we delete the chroot and replace it. Otherwise, this step is basically a no-op.
Uprev: Tell Portage about new changes
In this stage, we iterate through all of the stable cros-workon ebuilds and check whether the git hash inside the ebuild is up to date. If not, we update it. Similarly, we also check whether the developer has made changes to the unstable ebuild.
SetupBoard: Create the board root
Besides running the setup_board script, this step also sets up a shared user password, if applicable.
BuildPackages: Build all packages
This stage just runs the build_packages script. If PGO data is needed, then this step will wait for the PGO data to be ready first.
BuildImage: Build all images
The build image phase is called “BuildImage”, but it does more than build an image -- it also builds and uploads autotest tarballs and payloads, as shown in the diagram below.
Test Overview
All of the below tests are optional and run in parallel. Every builder will run a subset of the tests below. (Some builders even run no tests.)
AUTest: Run autoupdate tests on real hardware
Before launching autoupdate tests, this stage builds and uploads a tarball of the necessary autoupdate artifacts. The actual launching of the test is implemented using the logic from the HWTest stage.
ChromeSDK: Verify that Chrome compiles using the Simple Chrome workflow
On canaries, we actually build Chrome twice -- once inside the chroot, for use in the build, and once outside the chroot, in the ChromeSDK stage, to help verify that the simple Chrome workflow continues to work.
HWTest: Run a test suite on real hardware
This stage launches tests with a timeout. If the timeout is exceeded, we report a warning or an error, depending on configuration.
SignerTest: Run various security checks on the image
This stage just runs the security check script on the image.
VMTest: Run smoke tests inside a virtual machine
Besides running smoke tests on the image, this stage also verifies that gmerge can merge packages into a VM instance. Once the test results are built, they are uploaded to Google Storage, along with symbolized crash reports.
UnitTest: Run unit tests locally
This stage runs unit tests in parallel. If quick_unit is False, this is accomplished by looking for ebuilds which have the “src_test” clause and merging them with FEATURES=test. If quick_unit is True, we only run tests on ebuilds that have been uprevved in the Uprev stage.
Archive Overview
Release Process
Buildbot files
An edited list of the files under the buildbot subdirectory:
- builderstage.py: Module containing the base class for the stages that a builder runs. All of the stages inherit from this base class, so logic that is common to all stages is placed here.
- cbuildbot_commands.py: Module containing the various individual commands a builder can run. These functions basically just wrap RunCommand -- they provide a friendlier interface so that callers don’t need to build up the command line themselves. These commands are mostly called by cbuildbot_stages.py.
- cbuildbot_config.py: Buildbot configuration for cbuildbot. Tells us what buildbots to run.
- cbuildbot.py: Placeholder file to invoke bin/cbuildbot. See scripts/cbuildbot.py for the source code of cbuildbot.
- cbuildbot_results.py: Classes for collecting results of build stages as they run. Each build stage will record its success/failure status in this singleton object by calling Results.Record(...). This data is later used to build a report, and to determine whether the build was successful.
- cbuildbot_stages.py: Module containing the various stages that a builder runs. All of the stages discussed above are stored in this file.
- constants.py: This module contains constants used by cbuildbot and related code.
- lkgm_manager.py: This module implements the LKGMManager, which is used in both the commit queue and preflight queues. It helps the client track what the agents are doing, and helps the agents communicate their status back to the client. The client sends its manifest to the agents by committing it to git, and the agents communicate their status back to the client by uploading it to Google Storage.
- manifest_version.py: This module implements the BuildSpecsManager, which is used by canaries. It is simpler than LKGMManager because there is no need for bidirectional client/agents communication -- the workers just need to know the manifest to use and they are good to go.
- portage_utilities.py: Routines and classes for working with Portage overlays and ebuilds. This library is used in several places, including during the Uprev stage.
- remote_try.py: Class for launching remote tryjobs.
- repository.py: Routines and classes for working with git repositories. This is useful, for example, for syncing the source code.
- trybot_patch_pool.py: Classes for managing patches passed into a remote trybot run, and filtering them as appropriate.
- validation_pool.py: This module implements the core logic of the commit queue.