Chrome infrastructure

Contacting Infra Troopers

This page can be found at: go/bugatrooper

Have an issue with a piece of build infrastructure? Our troopers are here to help. Learn more about troopering at: go/trooper

Oncall hours: we cover working hours in the Pacific timezone:

  • 1600 - 0000 UTC (900 - 1700 MTV)

There is no designated oncall coverage for EMEA hours and APAC coverage is limited to some services owned by the Task Distribution team. Volunteers in those regions may provide assistance on mailing lists for urgent issues, but there's no guarantee.

If you are contacting a trooper to see if there is an issue with a service, visit the ChOps Status Dashboard first. If the “Current Status” of the service shows red/yellow, that means there is a known disruption or outage, and the trooper is already aware. No need to contact us further!

The primary way to contact a trooper is via issues.chromium.org using the templates and priorities established below. If you need to find the current trooper for a specific service, check build.chromium.org, or vi/chrome_infra (internal link). If crbug.com is down and you are unable to file a bug, please contact the team on infra-dev@chromium.org.

Small or non-urgent questions can also be posted in the #ops Chromium slack channel or the chops-hangout channel (internal).

If you know your issue is with the physical hardware, or otherwise should be handled by the Systems team, please follow their Rules of Engagement (internal).

Bug Templates

For fastest response, please use the provided templates:

Also make sure to include the machine name (e.g. build11-m1) as well as the builder name (Builder: win-archive-rel) when applicable.

Priority Levels

Priorities are set using the Priority=N label. Use the following as your guideline:

  • P0: Immediate attention desired. The owner will stop everything they are doing and investigate.
    • These reserved for massive outages, release blocking or multi-developer blocking productivity issues.
    • Examples: CQ no longer committing changes.
  • P1: Respond within 24 hours, resolution within 1 week
    • These are non-P0 blocking issues that need attention from a trooper
    • Examples: disk full on device, device offline, pending time high issues.
  • P2: Respond within 1 week, resolution is variable, depending on the issue
    • These are non-blocking issues or requests that need attention from a trooper
    • Examples: Non-blocking bugs or feature improvement suggestions
  • P3: Non-urgent. It is ok to wait or unassign.
    • These are non-urgent issues or nice to have changes.
    • Examples: Large change that will require major infrastructure changes or something that is a moonshot.

Life of a Request

All requests should be in the “Infra-Troopers” hotlist such that we can properly triage the request and to indicate that this is user-filed. Please do not assign issues to the trooper directly, as doing so may actually increase the time taken to respond to an issue.

The statuses are used as follows:

  • New: Your issue will show up in the queue to the trooper as untriaged. This will automatically get assigned to someone to triage.
  • Assigned: Someone is looking at the ticket, although it may still need to be triaged to another team.
  • If the ticket is assigned to YOU this indicates that more information from you is needed in order to proceed.  Please provide the information, and then unset ‘owner’ so the issue shows up in the queue again.
  • Fixed: The trooper believes the issue is resolved and no further action is required on their part.

Service Hours

Troopers provide full time coverage during working hours in the Pacific timezone and the expected response times outlined above during the work day. Other times support is provided best-effort.

More Information

View the current trooper queue.

Common Non-Trooper Requests: