This document is intended to be a reference for any developers looking to modify or add functionality to UFS. It provides necessary details for most of the use cases that we expect. Please contact chrome-fleet-automation@google.com for any questions with regards to UFS
Run UFS locally on your workstation (provided you have permissions). Makefile has a few builds that help you with this.
To run UFS with a coneectioon to the dev UFS datastore, run the following command:
make dev
To connect to a local datastore emulator, run the following command:
make dev-local-db
This will start a local datastore emulator and a UFS instance that is configured to connect to it.
The local Datastore emulator requires a Java 8+ JRE to be installed and on your system PATH.
Mac OS:
brew install openjdk@11 export PATH="/opt/homebrew/opt/openjdk@11/bin:$PATH"
To populate the local datastore with data from the production environment, follow these steps:
Export the production Datastore data to a GCS bucket:
gcloud datastore export --project="unified-fleet-system" gs://unified-fleet-system-backup/$(date +%Y%m%d_%H%M%S)
Download the snapshot to your local machine:
mkdir /tmp/ufs_snapshot gsutil -m cp -r gs://unified-fleet-system-backup/<SNAPSHOT_NAME> /tmp/ufs_snapshot
Import the data into the local Datastore emulator:
curl -X POST localhost:8081/v1/projects/test-project:import \ -H 'Content-Type: application/json' \ -d '{"input_url":"/tmp/ufs_snapshot/<SNAPSHOT_NAME>/<SNAPSHOT_NAME>.overall_export_metadata"}'
You might need to run
luci-auth login -scopes "https://www.googleapis.com/auth/cloud-platform https://www.googleapis.com/auth/userinfo.email"
to setup local auth for the service.
You might need to run
gcloud auth application-default login
for PubSub permissions.
You can make calls to your local version of the UFS service using prpc command line tool.
prpc call -metadata "namespace:os" localhost:8800 unifiedfleet.api.v1.rpc.Fleet.GetMachineLSE <<EOF {"name": "machineLSEs/<dut_name>"} EOF
If you're using VSCode, you can attach to your locally running service through the debug menu. When prompted, find the ufs-service process and debug through VSCode as usual. If you get an error, try following the prompt to modify your Yama settings and try again.
The UFS dumper is a job that syncs the UFS DataStore data into BigQuery for use for analysis.
Run dumper locally on your workstation.
make dev-dumper
You might need to run
gcloud auth application-default login
for BigQuery permissions.
shivas can be used to trigger the cron jobs locally. The makefile in shivas source creates a local version dev-shivas. This can be used to trigger cron on the local instance of dumper.
dev-shivas admin cron <cron-job-name>
The UFS production push process has been migrated to the automated ufs-promoter builder in LUCI.
Adding a new field to a proto includes some subtleties and may involve changes outside this directory.
Proto entity updates are logged at the controller layer in change.go. Make sure any new fields get added there.
If the proto is used in BigQuery, the BigQuery table schemas should be updated once after the proto is merged to dev and once after a UFS push to prod. To do so, run the setup_bigquery.sh script.
In addition, any PLX scripts that require this field should get updated within google3.
If the field needs shivas support, update the appropriate files in the shivas repo. Notably, this may require UFS changes that aren't directly used by the service, such as app/util/input.go
ChromeOS Swarming fetches DUT info via a shivas command.
If the new field is not used to schedule tests, shivas should be updated to expose the relevant field.
If the new field is used to schedule tests, shivas does not need any updates. Instead, the skylab swarming library will need to be updated in two areas:
In addition, another UFS util file, app/util/osutil/exporting_adapter.go, will need to include the new field when constructing a SchedulableLabels struct for the corresponding entity.
Any skylab proto changes will require regeneration of code via go generate within both the skylab library and UFS directories.