This document is intended to be a reference for any developers looking to modify or add functionality to shivas. It provides necessary details for most of the use cases that will be encountered when modifying shivas. If you are looking for user-manual, the built-in -help would give the most updated and correct information. Please contact us at browser-fleet-automation@google.com if you find something missing in this document.
Shivas is a command line tool that is used to communicate with UFS. In addition shivas also supports communicating with various other services including inventory V2, and swarming. The tool is released on cipd and therefore requires depot_tools to be installed. Anyone can compile the tool and run it, however the services that tool connects to will require user authorization.
Shivas is shipped with a makefile that handles most of the compilation. Compile shivas using make and delete the compiled files using make clean. It is also possible to compile shivas using go build.
Note:
It is possible to compile shivas for testing locally. This will make shivas hit localhost for UFS endpoints. You need to run UFS locally to be able use shivas this way. Compile shivas for local use by running
make dev
UFS APIs are generally given as a set of four operations per entity. They are
Create:
Takes the complete proto message describing the entity for update and returns the updated proto message
Example:CreateAssetRPC takes the asset proto to be created
message CreateAssetRequest {
// The asset to register
models.asset asset = 1 [(google.api.field_behavior) = REQUIRED];
}
Read: (& List)
Read takes the primary key of the entity to be retrieved and returns the requested entity. This will always return a single entity.
Example:GetAsset RPC takes the asset name to be retrieved.
message GetAssetRequest {
// The name of the asset to retrieve.
string name = 1 [
(google.api.field_behavior) = REQUIRED,
(google.api.resource_reference) = { type: "unified-fleet-system.appspot.com/asset" }
];
}
List is generally used when the result of the query is more than one. List allows for filters, which are usually used in order to obtain the result. The results of the query might also be broken up into multiple messages. These are handled using a cursor to get rest of the data as required
Example:ListAsset RPC takes page_size, page_token(cursor), filter and keys_only flag and returns a list of assets along with an optional token for the next_page
message ListAssetsRequest {
// The maximum number of assets to return. The service may return fewer than
// this value.
// If unspecified, at most 100 assets will be returned.
// The maximum value is 1000; values above 1000 will be coerced to 1000.
int32 page_size = 1;
// A page token, received from a previous `ListAssets` call.
// Provide this to retrieve the subsequent page.
//
// When paginating, all other parameters provided to `ListAssets` must match
// the call that provided the page token.
string page_token = 2;
// filter takes the filtering condition
string filter = 3;
// if this is true, only keys will be returned else the entire object
// will be returned. By setting this to true, the list call be will faster.
bool keysOnly = 4;
}
message ListAssetsResponse {
// The assets from datastore.
repeated models.asset assets = 1;
// A token, which can be sent as `page_token` to retrieve the next page.
// If this field is omitted, there are no subsequent pages.
string next_page_token = 2;
}
Update:
This type of operation generally takes two inputs, entity and a list of strings called update_mask.
Example:
message UpdateAssetRequest {
// The asset to update.
models.asset asset = 1 [(google.api.field_behavior) = REQUIRED];
// The list of fields to be updated.
google.protobuf.FieldMask update_mask = 2;
}
There are two types of update operations that are available.
update_mask == nilmy-laptop {
"name": "my-laptop",
"type": "DUT",
"model": "bluebird",
"location": {
"aisle": "",
"row": "14",
"rack": "chromeos6-row14-rack23",
"rackNumber": "23",
"shelf": "",
"position": "16",
"barcodeName": "",
"zone": "ZONE_CHROMEOS6"
},
"info": {
"assetTag": "my-laptop",
"serialNumber": "laptop-serial-xoxo",
"model": "bluebird",
"buildTarget": "octopus",
"referenceBoard": "octopus",
"ethernetMacAddress": "ff:ee:dd:cc:bb:aa",
"sku": "2",
},
}
Updating it with {
"name": "my-laptop",
"type": "LABSTATION",
"model": "bluebird",
"location": {
"aisle": "",
"row": "1",
"rack": "chromeos6-row1-rack21",
"rackNumber": "21",
"shelf": "",
"position": "15",
"barcodeName": "",
"zone": "ZONE_CHROMEOS6"
},
}
Will replace entire row with the new configuration. This results in the info field being reset.update_mask. {
"asset" : {
"name": "my-laptop",
"type": "DUT",
"model": "eve",
"location": {
"aisle": "10",
"position": "15",
"zone": "ZONE_CHROMEOS6"
},
},
"update_mask": ["type", "model", "location.aisle"]
}
This will only update the type, model and location.aisle. The position and zone fields are ignored. This will update the given entry as {
"name": "my-laptop",
"type": "DUT",
"model": "eve",
"location": {
"aisle": "10",
"row": "1",
"rack": "chromeos6-row1-rack21",
"rackNumber": "21",
"shelf": "",
"position": "15",
"barcodeName": "",
"zone": "ZONE_CHROMEOS6"
},
}
Delete:
Only take the primary key for the entity and deletes the record
Example:
message DeleteAssetRequest {
// The name of the asset to delete
string name = 1 [
(google.api.field_behavior) = REQUIRED,
(google.api.resource_reference) = { type: "unified-fleet-system.appspot.com/Asset" }
];
}
Entities tracked by UFS are generally required to support these 4 operations. This is commonly referred to as CRUD operations
Note:
generic_rename_cmd.goupdate commands usually do a full update on JSON. This is because the expected input for the the command is not the Request proto (which contains the update_mask). But the entity proto itself. Check the help for the particular command to see what is the JSON file expectationsCommand line flags can be added to any given command. The flags are created using flags package. Strings are by far the most common ones, but bools, strings slices, ints ... etc,. are also supported. It is also possible to create a custom flags type. Make sure to validate the inputs for the new flag.
All shivas commands include three default flagsets. These are auth flags that handle the authorization options, env flags that handle processing the environment vars, common flags which include verbose option and output flags handle the output format for shivas. Include these in your command as required.
This is generally not recommended as it is likely to break something downstream. If you have to update the CSV fields, they are usually recorded as mcsvFields. CSV updates are treated as if command line updates were performed quickly in succession.
Note:
It is possible to run UFS locally and test your changes to the code. UFS is compiled as a binary which in theory can be run on any computer. This can be done by running make dev in the UFS source code. This will allow you to test your changes locally with a few catches.
-dev flag with shivas to avoid updating the prodgo build -tags='dev' to hit the local endpointsIf this is your first time you will also need to provide creds to Shivas.
shivas login
UFS differentiates between the browser or os namespace to perform updates to the relevant databases. This is done in shivas by setting SHIVAS_NAMESPACE to either browser or os. You can also include env flags in your command that will allow for -namespace flag that can be set wile running the command.
If you wish your command to run only one one of these namespaces, you need to explicitly set the namespace in your context as follows
ctx = utils.SetupContext(ctx, ufsUtil.OSNamespace)
where ufsUtil is the utils package in UFS.
Shivas supports using barcode reader for updates. This is done to speed up the work for certain workflows. Barcode reader can only be used in certain circumstances, this is due to the obvious limitations with amount of data that can be read through the barcode. By default barcode readers work like a keyboard input device. Whatever is scanned to barcode is dumped to stdin of the process terminated by a new line. This allows us to use barcode readers in certain processes to speed up the work. Shivas handles barcode reader based updates by calling the required RPC as soon as all the data is available to perform the operation.
Current use cases include registering a bunch of assets into a rack in OS lab. Deleting a list of assets. Updating location data for a bunch of assets and other such processes.
Check -scan functionality in add/update assets for some implementation details.
Shivas was intended to be used as an user tool. That is no longer true and various wrappers in multiple languages has been written for shivas. It is generally recommended to call the UFS API directly if possible. But as UFS RPCs are implemented using gRPC and it might not be supported in your situation, some wrappers were inevitable for shivas. Before implementing a new wrapper for shivas, please check if any of the following is available.
If no alternative is available, please announce your intention to shivas maintainers at ufs-announce@google.com. Check -version option in shivas and use it to avoid unexpected crashes when someone changes the output format. And finally add your implementation to the list below
Shivas code is generally not tested. This was because it was never intended to be used in it's current context and was initially conceived as a light weight tool for updating UFS. That no longer being the case, it is recommended that you add unit tests to any new command that gets added to shivas. Owners approval will be hard to get without any tests written for the tool. We will be really happy if you can add unit tests to existing code.