Google GitGoogle Git
Sign in
chromium/infra/infra/refs/heads/main/./go/src/infra/appengine/crosskylabadmin
tree: 20e90505e69cd012d5a9ed8be71396b912a673d2
  1. api/
  2. cloudrun/
  3. internal/
  4. site/
  5. .gitattributes
  6. adminclient
  7. crosadm
  8. Makefile
  9. OWNERS
  10. README.md
go/src/infra/appengine/crosskylabadmin/README.md

ChromeOS Skylab Admin

Overview

Chrome OS Skylab Admin is a Google App Engine app (in Go) that supports the following:

Keep Devices Healthy

This service is trying to keep devices in a healthy, schedulable state at all times. Device health can degrade for many reasons (bad test, provisioning failure, etc...) and this service supports recovering those devices with the following:

  • Scheduling automated repair jobs for any devices it detects as unhealthy:
    • DUTs: Automated admin tasks (repair, reset) are scheduled based on the device's specific dut_state (needs_repair, repair_failed, needs_manual_repair).
    • Labstations: Periodic repair checks and reboot tasks (push-repair-jobs-for-labstations) are scheduled hourly for idle labstations whose dut_state is ready, serving, needs_repair, repair_failed, or needs_manual_repair. Labstations in reserved state (or other ineligible states such as disabled and decommissioned) are intentionally excluded from automated periodic repair checks to prevent interrupting manual debugging or reserved usage.

Other functionality

  • Read recovery version from configs
  • Provides API to read recovery version from Datastore

Code/Development Setup

For initial setup, follow the Chrome Infra Go procedures

For environment setup, follow the Bootstrap procedures

Application Environments

Test

  • Unit testing - ‘cd go/src/infra/appengine/crosskylabadmin && make test’
  • Functional/integration testing - No automated testing

Release Procedures

Crosskylabadmin (CSA) deployments and rollbacks are managed primarily through automated LUCI Promoters (recipes/fleet/csa_promoter.py).

1. Staging Promotion (csa-dev-promoter)

  • Builder: csa-dev-promoter
  • Schedule: Triggered on demand via the LUCI Scheduler UI or bb add (bb add infra-internal/fleet-services/csa-dev-promoter).
  • Actions:
    1. Builds and uploads the App Engine version (default service) and configurations (cron.yaml, queue.yaml).
    2. Switches 100% traffic on skylab-staging-bot-fleet.appspot.com.

2. Production Promotion (csa-prod-promoter)

  • Builder: csa-prod-promoter
  • Schedule: Triggered on demand via the LUCI Scheduler UI or bb add (bb add infra-internal/fleet-services/csa-prod-promoter).
  • Strict Staging-First Promotion Rule (CRITICAL): csa-prod-promoter automatically queries the active App Engine version currently serving on skylab-staging-bot-fleet.appspot.com, verifies that it is valid, and checks out that exact Git commit before deploying to production. Staging (skylab-staging-bot-fleet) must always be pushed and verified first before triggering the production promoter.
  • Actions:
    1. Verifies and checks out the active Staging commit.
    2. Builds and uploads the App Engine version (default service) and configurations (cron.yaml, queue.yaml).
    3. Switches 100% traffic on chromeos-skylab-bot-fleet.appspot.com.

3. Emergency Rollbacks

If an issue occurs after a deployment, rollback builders are available to immediately restore service without recompiling:

  • Production Rollback: csa-prod-promoter-rollback (bb add infra-internal/fleet-services/csa-prod-promoter-rollback)
    • Automatically queries App Engine for the most recent non-active version and switches live production traffic back to it (chromeos-skylab-bot-fleet).
    • Optionally accepts an explicit target_version property (-p target_version="<version>") if rolling back to a specific earlier release.
  • Staging Rollback: csa-dev-promoter-rollback (bb add infra-internal/fleet-services/csa-dev-promoter-rollback)
    • Switches Staging (skylab-staging-bot-fleet) traffic back to the prior serving version.

4. Manual Fallback (gae.py)

For emergency or local manual workflows, App Engine commands can be run directly from go/src/infra/appengine/crosskylabadmin:

  • Staging: make up-staging && make switch-staging
  • Production: make up-prod && make switch-prod
  • Manual Rollback: gae.py switch -A chromeos-skylab-bot-fleet --app-dir ./internal/app/appengine <previous_version>