Chrome infrastructure

Buildbucket

Buildbucket is a generic build queue. A build requester can schedule a build and wait for a result. A building system, such as Buildbot, can lease it, build it and report a result back.

Build

A build is an element of buildbucket. It can be scheduled, leased, started, and completed.

A Build JSON object may have the following keys:

  • id: a unique ID of build, generated by buildbucket. Ids are monotonically decreasing integers. Queries return builds sorted by id, thus new builds are first.
  • created_ts: timestamp when the build was created.
  • created_by: identity string (e.g. email) of who made a request to create a build.
  • bucket: a string that defines what project/subproject a build belongs to. Required.
  • tags: indexed key-value pairs, such as ["buildername:Release", "user_agent:cq"]. Builds can be searched by tags. Also see tag conventions.
  • parameters_json: an immutable JSON blob passed by a build requester, represents build parameters. The format is specific to a build-system. Not reqiured by buildbucket, but may be required by a particular build-system, such as Buildbot.
  • url: a URL to a build-system-specific human-viewable page that displays build status. Set by a build-system when the build is started.
  • lease_expiration_ts: when current lease will expire, if any.
  • lease_key: a key generated by buildbucket to a leasee. The leasee must pass it when it mutates the build.
  • status: current status of the build, can be one of:
    • "SCHEDULED": available for processing.
    • "STARTED": leased and is being processed.
    • "COMPLETED": completed and cannot be changed. See result field for more info.
  • status_changed_ts: timestamp when last time build status has changed.
  • completed_ts: timestamp when the build was completed.
  • result: final result of the build. Can be set only in COMPLETED state. Can be one of:
    • "SUCCESS": the build has succeeded.
    • "FAILURE": the build has failed. See failure_reason field for more info.
    • "CANCELED": the build has been canceled. See CancelationReason for more info.
  • result_details_json: an immutable JSON blob passed by a build-system on build completion. Represents the result of the build.
  • failure_reason: why the build failed. Can be one of:
    • "BUILD_FAILURE": the build has failed normally, not due to an infrastructure failure.
    • "INFRA_FAILURE": infrastructure failure happened.
    • "BUILDBUCKET_FAILURE": buildbucket failure happened.
    • "INVALID_BUILD_DEFINITION": a build system could not understand the definition of the build. See result_details_json for more info.
  • cancelation_reason: why the build was canceled. Can be one of:
    • "CANCELED_EXPLICITLY": the build canceled was requested externally.
    • "TIMEOUT": the build was canceled on timeout. Currently happens only for builds that are in SCHEDULED or STARTED state for more than 48 hours.
  • updated_ts: timestamp when the build was modified last time. A modification includes leasing.
  • utcnow_ts: current UTC time on the server.
  • canary_preference: whether the build should use canary of build infrastructure. Can take values PROD, CANARY and AUTO (default). If CANARY and canary infrastructure does not exist, build creation response must be BAD REQUEST.
  • canary: a boolean that indicates whether a canary of the build infrastructure is used to run this build.
  • experimental: a boolean that indicates whether this build is experimental. An experimental build does not show up in search results by default, does not affect monitoring and its execution should not affect production.

All timestamps are in microseconds since Unix Epoch.

API

API for build requesters:

API for build systems:

Also see go/buildbucket-design for more details.

Leasing a build

A build-system is supposed to lease a build before processing it. A build lease is exclusive and a build cannot be mutated without a lease. A leasee receives a lease_key that it uses for mutation. When a build is leased, it is not returned in peek API call. When build lease expires, buildbucket makes the build available again.

Conventions

Some conventions were established in order to build requesters and build systems work together.

buildset tag

The buildset tag is used to group related builds. Established formats:

  • Rietveld CL: patch/rietveld/<hostname>/<issue_id>/<patchset_id> e.g. “patch/rietveld/codereview.chromium.org/2979743003/1”.
  • Gerrit CL: patch/gerrit/<hostname>/<numeric_change_id>/<numeric_patch_number> e.g. “patch/gerrit/chromium-review.googlesource.com/677784/5”.
  • Gitiles commit: commit/gitiles/<hostname>/<project_name>/+/<commit hash>, e.g. “commit/gitiles/chromium.googlesource.com/infra/luci/luci-go/+/b7a757f457487cd5cfe2dae83f65c5bc10e288b7”. Typically builds with such a buildset also have “gitiles_ref” tag.

When a build triggers a new build, the buildset tag must be copied.

user_agent tag

Since there may be different build requester systems, you may want to distinguish builds by different build requesters. The user_agent tag is used for that. Known chrome-infra build requesters:

  • "cq": Chromium Commit Queue
  • "rietveld"
  • "git_cl_try"

Configuration

Buckets are registered in <buildbucket-app-id>.cfg files in infra/config branch of project repositories, for example in chromium/src repo. See BuildbucketCfg protobuf message for file format.

Access control

Each bucket has its own ACLs. By default a bucket is accessible only by admins. If a bucket does not exist, its ACL is considered default.

Buildbot integration

Buildbot is integrated with buildbucket, with two-way sync. Using buildbucket you can schedule, cancel, search and read build results (but no logs). Read more in buildbot-buildbucket README.md.

Swarming integration (Swarmbucket)

See go/swarmbucket