Chrome infrastructure

Contacting troopers

This page can be found at: g.co/bugatrooper

Have an issue with a piece of build infrastructure? Our troopers are here to help. Learn more about Troopering at: go/trooper

Oncall hours: we cover working hours in the Pacific timezone:

  • 1600 - 0000 UTC (900 - 1700 MTV)

There is no designated oncall coverage for EMEA hours and APAC coverage is limited to some services owned by the Task Distribution team. Volunteers in those regions may provide assistance on mailing lists for urgent issues, but there's no guarantee.

If you are contacting a trooper to see if there is an issue with a service, visit the ChOps Status Dashboard first. If the “Current Status” of the service shows red/yellow, that means there is a known disruption or outage, and the Trooper is already aware. No need to contact us further!

The primary way to contact a trooper is via crbug.com using the templates and priorities established below. If you need to find the current trooper, check build.chromium.org, or vi/chrome_infra (internal link). If crbug.com is down and you are unable to file a bug, please contact the team on infra-dev@chromium.org.

Small or non-urgent questions can also be posted in the #ops Chromium slack channel or the chops-hangout channel (internal).

If you know your issue is with the physical hardware, or otherwise should be handled by the Systems team, please follow their Rules of Engagement (internal).

Bug Templates

For fastest response, please use the provided templates:

Also make sure to include the machine name (e.g. build11-m1) as well as the builder name (Builder: win-archive-rel) when applicable.

Priority Levels

Priorities are set using the Pri=N label. Use the following as your guideline:

  • Pri-0: Immediate attention desired. The owner will stop everything they are doing and investigate.
    • These reserved for massive outages, release blocking or multi-developer blocking productivity issues.
    • Examples: CQ no longer committing changes.
  • Pri-1: Respond within 24 hours, resolution within 1 week
    • These are non-P0 blocking issues that need attention from a Trooper
    • Examples: disk full on device, device offline, pending time high issues.
  • Pri-2: Respond within 1 week, resolution is variable, depending on the issue
    • These are non-blocking issues or requests that need attention from a Trooper
    • Examples: Non-blocking bugs or feature improvement suggestions
  • Pri-3: Non-urgent. It is ok to wait or unassign.
    • These are non-urgent issues or nice to have changes.
    • Examples: Large change that will require major infrastructure changes or something that is a moonshot.

Life of a Request

Status will be tracked using the Status field, with the ‘owner’ field unset. The trooper queue relies on the ‘owner’ field being unset to track issues properly, with troopers setting the owners field for particularly long-running issues. Please do not assign issues to the trooper directly, doing so may actually increase the time taken to respond to an issue. Behind the scenes, our Troopers will review and triage the bug, as-needed, to corresponding Chrome Operations teams to be addressed.

All requests should contain the “Infra-Troopers” label such that we can properly triage the request and to indicate that this is user-filed.

  • Untriaged: Your issue will show up in the queue to the trooper as untriaged. Once they acknowledge the bug, the status will change.
  • Available: Trooper has ack'ed, if not Pri-0, this means they have not started working on it.
  • Assigned:
    • Trooper has triaged and determined there is a suitable owner and appropriately assigned.
    • If that owner is YOU this indicates that they need more information from you in order to proceed.  Please provide the information, and then unset ‘owner’ so the issue shows up in the queue again.
  • Started: Your issue is being handled, either by the Trooper or other owner.
  • Fixed: The trooper believes the issue is resolved and no further action is required on their part.

Service Hours

Troopers provide full time coverage during local business hours and the expected response times outlined above during the work day. Other times support is provided best-effort.

More Information

View the current trooper queue.

Other available team trooper queues within Chrome Operations.

Common Non-Trooper Requests: