py/hwid/README.md

# ChromeOS HWID Database User Guide

[TOC]

This document is aimed at defining the standard way to update the HWID database.

## Introduction to HWID Database

The role of a hardware ID (HWID) configuration file is to verify that every
hardware component used in a Chrome device is qualified. The config file mainly
lists the phase of the device `image_id`, the pattern of the encoded string
`pattern`, the mapping of bit value to the component `encoded_field`, and the
probe result of each component `component`.
For convenience, we call the HWID config file **HWID database**.

## Update Rule

The HWID database has two tasks: (1) verifies the probe result and generates the
HWID string when building new devices at the factory, and (2) decodes the HWID
string generated by every previous version of the database. To ensure the new
database can successfully decode the old HWID string, there are some rules while
updating the database.

### 1. Common Rule: Only Append, Do Not Modify or Remove Items
The first common rule is: do not remove anything. Because we not only need to
decode current HWID, but also need to decode *EVERY* previous valid HWID. Only
these items can be modified:
- The probeable attribute of the component
- The status of the component item
- The rules items

### 2. Pattern: Do Not Modify the Set of encoded_field {#rule_encoded_field}
It's obvious that we are not allowed to remove the field in the pattern. Adding
a field whose component class is new seems not break the previous HWID, but it
actually causes ambiguity. After adding the component in the pattern, we would
decode old HWID as index = 0. But actually we have no information about this
component in old HWID. There are 3 possible situations:

1. The old device doesn't have the component, so the component is not in the
   HWID database.
2. The old device has the component, but it cannot be probed previously.
3. The old device has the component and it can be probed, but user forgot to
   add into HWID database.

Most of the case is 1, the old device doesn't have the component. But we don't
enforce index 0, which means there is no probed result (or no component)
currently. So it is weird to allow this happen. The reasonable way is: the
component set in each pattern cannot be modified. It can prevent this ambiguity
from happening.

However, we might lean to relax this rule in the early build. The case (2)
has a high chance of happening: every old device has the component, but it
cannot be probed. Creating a new pattern for adding an extra component seems
overkill. In this case we allow appending the pattern directly.

In summary, if we confirm that all the old device has the same kind of the
component, then it is allowed to append to the current pattern. Otherwise,
please create a new pattern when we need to modify the set of component class.

### 3. Encoded_fields: Do Not Modify Encoded Mapping
Because `encoded_field` is shared with all patterns, the existed items in the
`encoded_field` cannot be modified. Otherwise the old HWID cannot lookup the
correct mapping. Therefore, if we need to change the mapping of a component, we
can only add new field into `encoded_fields`.

### 4. Components: Use deprecated/unsupported to Remove Component
Because the components items should be encoded in `encoded_field` in normal
case, we cannot remove them. When we don't allow a component in the new unit,
use deprecated status. If you don't even allow to match it, use unsupported
status. Please refer [Status of a Component Item](#component_status) for more
details.

## Database Builder

Originally it was tedious to convert the probe result to HWID database. Most of
the time we copy and paste it from an existing project and modify the probe
value. In addition, there are many implicit rules and relations between each
section. For example, the number of bits in the field of the pattern should be
large enough to contain all items in the encoded field.

Now we have a tool called **database builder** to generate or update the HWID
database. This tool is aimed at covering most of the common and tedious HWID
database change, like adding or removing components. But the HWID database still
needs to be manually updated in the following scenarios:

### Scenarios that Cannot Use Database Builder

- **Multiple image_ids are active**

  In the database builder, we assume that only the latest (maximum) `image_id`
  is active. So we recommend not to use the tool if the assumption is not true.

- **Modify the rules**

  The rule section gives us much flexibility in assigning and verifying the
  value of the component. On the other hand, there is no universal diagram for
  the rule section. We still leave it to user update the rules manually.

- **Change the status of component items**

  Because changing the status is quite easy. Just change it :)

### Prerequisites

The command `hwid` must be run on a
[Factory Preflash Image](https://chromium.googlesource.com/chromiumos/platform/factory/+/HEAD/README.md#imaging-methods)
(test image in partition A and release image on partition B) because `hwid`
loads information from other partitions in the factory preflash image. You might
encounter unexpected error if running on the wrong partition.

### Command of Database Builder

```shell
# Views the usage of building a new HWID database.
$ hwid build-database --help

# Views the usage of updating an existing HWID database.
$ hwid update-database --help
```

### Post-Processing
Though the database builder handle most of the work, there is still some
manual post-processing.

- **Check the name of component items**

  The builder will decide a name by the probe result, but it might be
  meaningless. Please change the name in **encoded_fields** and **components**
  if you don't like it.

- **Confirm the status of the component items**

  All component items generated by the database builder are **unqualified**.
  After the SIE or TAM confirms the component is qualified at AVL, please change
  the status to **supported**.


## Use Cases

Below, we will list some common use cases of HWID database change, and provide
the recommendation way to do it.

### 1. Get an initial HWID Database

To update (e.g. add components, change status, etc) the HWID database, we
firstly need to have a HWID database file.  To get the initial HWID database
file, please request one from your device management page on ChromeOS Device
Lifecycle Management (DLM).  Once the request is accepted, a HWID bundle with
the initial HWID database will be created for you to download.

### 2. Add Probed Value, or Second Source
This is the most common update in HWID database. When we have the initial HWID
DB, we can start to add the probed values, introduce second source, or update
firmware version.  We need to add new component item into HWID database. Just
append the new item in the component field and check if the encoding bit is
enough or not.  If we update the firmware version, we should mark the old one
as "deprecated" to ensure all DUTs in the factory use the latest firmware.

There are two types of unprobeable components we can use while updating the HWID
DB.

1. (Optional) Check if there are missing components in the probe result. If so,
   add a default item with the argument "--add-default-component". Please refer
   [Add a Default Item](#add_default_item) for more details.
2. (Optional) Check if there are components that do not exist in some SKU. For
   example, one SKU has cellular and another SKU does not. If so, add a null
   item with the argument "--add-null-component". Please refer
   [Add a Null Item](#add_null_item) for more details.

The command for this scenario is:
```shell
# Get the probe result.
$ hwid collect-material --output-file /tmp/material.yaml

# Update the database at /usr/local/factory/hwid/GOOGLE
$ hwid update-database \
    --material-file /tmp/material.yaml \
    --project GOOGLE \
    --add-default-component battery \
    --add-null-component cellular
```

**Note**: Depends on project settings, some components might require
factory device data ready to be probed correctly.  Therefore, it's
highly suggested to run `hwid collect-material` on DUT with factory
toolkit ready when preparing the HWID database for on-going
manufacturing builds.

### 3. Add a Default Item {#add_default_item}
When we bring up a device, the probing code for some hardware components might
not be ready. For example, a device cannot probe cellular in EVT build, and can
probe it later. Before the probing code is ready, please add a **default item**.
After the probing code is ready, add the component item and deprecate the
default item.

|||---|||
#### Before updating the HWID config file
```yaml
image_id:
  0: EVT

pattern:
- image_ids: [0]
  encoding_scheme: base8192
  fields: []

encoded_fields: {}

components: {}
```

#### After updating the HWID config file
```yaml
image_id:
  0: EVT

pattern:
- image_ids: [0]
  encoding_scheme: base8192
  fields:
  - cellular_field: 3
  - other_probeable_component_field: 2

encoded_fields:
  cellular_field:
    0: {cellular: cellular_default}
  other_probeable_component_field:
    0: {other_probeable_component: comp_1}

components:
  cellular:
    items:
      cellular_default:
        value: NULL
  other_probeable_component:
    items:
      comp_1:
        value: {compact_str: aa}
```
|||---|||

The command for this scenario is:

```shell
# While updating the database by the probed result, add a default item with
# "--add-default-component" argument.
$ hwid update-database \
    --project GOOGLE \
    --add-default-component cellular \
    --material-file /tmp/material.yaml  # the probe result doesn't contain
                                        # cellulars.
```

### 4. Add a Null Item {#add_null_item}
When we have a special SKU that does not have a component, please add a **null
item**. For example, we have a SKU with cellular component and another SKU
without the cellular component. Then the HWID database should be like:

|||---|||
#### Before adding the null item
```yaml
image_id:
  0: EVT

pattern:
- image_ids: [0]
  encoding_scheme: base8192
  fields:
  - cellular_field: 3

encoded_fields:
  cellular_field:
    0: {cellular: aa}

components:
  cellular:
    items:
      aa:
        value: {compact_str: aa}
```

#### After adding the null item
```yaml
image_id:
  0: EVT

pattern:
- image_ids: [0]
  encoding_scheme: base8192
  fields:
  - cellular_field: 3

encoded_fields:
  cellular_field:
    0: {cellular: aa}
    1: {cellular: NULL}

components:
  cellular:
    items:
      aa:
        value: {compact_str: aa}
```
|||---|||

The command for this scenario is:
```shell
# Add a default item with "--add-null-component" argument.
$ hwid update-database \
    --project GOOGLE \
    --add-null-component cellular
```

### 5. Deprecate a Component Item
Deprecating the component item is quite easy. Please manually add
"status: deprecated" in the target component item. Then the new unit may not use
this component item.

The example below is to deprecate the cellular component item `aa`.

|||---|||
#### Before deprecating the item
```yaml
image_id:
  0: EVT

pattern:
- image_ids: [0]
  encoding_scheme: base8192
  fields:
  - cellular_field: 3

encoded_fields:
  cellular_field:
    0: {cellular: aa}
    1: {cellular: bb}

components:
  cellular:
    items:
      aa:
        value: {compact_str: aa}
      bb:
        value: {compact_str: bb}
```
#### After deprecating the item
```yaml
image_id:
  0: EVT

pattern:
- image_ids: [0]
  encoding_scheme: base8192
  fields:
  - cellular_field: 3

encoded_fields:
  cellular_field:
    0: {cellular: aa}
    1: {cellular: bb}

components:
  cellular:
    items:
      aa:
        status: deprecated
        value: {compact_str: aa}
      bb:
        value: {compact_str: bb}
```
|||---|||

### 6. Add New Component in Device
When we add new component types during the build, we need to create a new
`pattern` and new `image_id`. Please refer [Update Rule 2](#rule_encoded_field)
for more details.

For example, we don't have cellular in EVT build, but add cellular in DVT build.
To complicate the situation even more, in DVT build we might have one SKU that
has cellular and another SKU that doesn't. In this case, we need to distinguish:
1. No cellular in EVT build
2. No cellular in DVT build
3. Has cellular in DVT build

The difference between 1 and 2 is: in EVT build we don't check the probed result
of the cellular component. The actual meaning should be "we don't have
information about cellular" instead of "this device doesn't have cellular". To
distinguish the case 1 and 2, we need to create a new pattern that adds the new
component, while retaining the old pattern which lacks the component.

The HWID database for this example:
|||---|||
#### Before adding the cellular component
```yaml
image_id:
  0: EVT

pattern:
- image_ids: [0]
  encoding_scheme: base8192
  fields:
  - cpu_field: 3

encoded_fields:
  cpu_field:
    0: {cpu: cpu_aa}

components:
  cpu:
    items:
      cpu_aa:
        value: {compact_str: cpu_aa}
```
#### After adding the cellular component
```yaml
image_id:
  0: EVT
  1: DVT

pattern:
- image_ids: [0]
  encoding_scheme: base8192
  fields:
  - cpu_field: 3

- image_ids: [1]
  encoding_scheme: base8192
  fields:
  - cpu_field: 3
  - cellular_field: 3

encoded_fields:
  cpu_field:
    0: {cpu: cpu_aa}
  cellular_field:
    0: {cellular: NULL}
    1: {cellular: aa}

components:
  cpu:
    items:
      cpu_aa:
        value: {compact_str: cpu_aa}
  cellular:
    items:
      aa:
        value: {compact_str: aa}
```
|||---|||

The command for this scenario is:
```shell
# Update the database at EVT build
$ hwid update-database \
    --project GOOGLE \
    --image-id EVT \
    --material-file /tmp/material.yaml  # the probe result contains doesn't
                                        # contain cellulars.


# Update the database at DVT build.
$ hwid update-database \
    --project GOOGLE \
    --image-id DVT \
    --add-null-component cellular \
    --material-file /tmp/material.yaml  # the probe result contains the
                                        # cellular "aa".
```

*** note
**Note:** If all the old devices have the same kind of the component, then it is
allowed to append the component in the current pattern.
***

|||---|||
#### Before adding the cellular component
```yaml
image_id:
  0: EVT

pattern:
- image_ids: [0]
  encoding_scheme: base8192
  fields:
  - cpu_field: 3

encoded_fields:
  cpu_field:
    0: {cpu: cpu_aa}

components:
  cpu:
    items:
      cpu_aa:
        value: {compact_str: cpu_aa}
```
#### After adding the cellular component
```yaml
image_id:
  0: EVT

pattern:
- image_ids: [0]
  encoding_scheme: base8192
  fields:
  - cpu_field: 3
  - cellular_field: 3

encoded_fields:
  cpu_field:
    0: {cpu: cpu_aa}
  cellular_field:
    0: {cellular: aa}

components:
  cpu:
    items:
      cpu_aa:
        value: {compact_str: cpu_aa}
  cellular:
    items:
      aa:
        value: {compact_str: aa}
```
|||---|||

The command for this scenario is:
```shell
# Update the database without assigning the image_id.
$ hwid update-database \
    --project GOOGLE \
    --material-file /tmp/material.yaml  # the probe result contains the
                                        # cellular "aa".

WARNING: Extra fields [cellular_field] without assigning a new image_id.
If the fields are added into the current pattern, the index of these fields will
be encoded to index 0 for all old HWID string. Enter "y" if you are sure all old
devices with old HWID string have the component with index 0. [y/N] y
```

Please choose **"Y"** for the question, then the database builder will append
the component into the current pattern.

### 7. Append Full Component Combinations to the Encoded Field

The argument `--fill-combination <ENCODED_FIELD>:<NUMBER_OF_COMPONENTS>`
helps ensure the encoded field contains all possible component combinations.
When a device is equipped with multiple components of the same type (2 cameras,
one for user-facing and one for world-facing, for example), the corresponding
encoded field allocates each unuqie combination a dedicate index number.
Which means that the user will need to register all possible combinations into
the HWID DB.  Approaches include running
`hwid updat-database --material-file ...` on all SKUs, or manually appending
all possible combinations based on the build plan.  Both of the approaches
involve many manual works, and hence `--fill-combination` is here to simplify
the process.

For example, assuming the HWID DB already includes some existing cameras.
And now we want to add a new camera into HWID DB.  The following command

```shell
hwid update-database \
    --material-file <COLLECTED_MATERIAL_FILE_FROM_A_DEVICE_WITH_NEW_CAMERA> \
    --fill-combination camera_field:2  # Specify to append full combinations of
                                       # 2 components to camera_field.
```

will not only add the new camera into the HWID DB but also make sure
`camera_field` includes all *pairs* of cameras.  So if the HWID DB contains
4 cameras in total after update, `camera_fields` will contain at least 10
combinations.

Another usage is that if all the candidate cameras already exist and the user
only want to fulfill the encoded field, they can run the following command,
which will not update the HWID DB based on collected materials and only touch
the specified encoded field.

```shell
hwid update-database \
    --skip-update-by-collected-material \
    --fill-combination camera_field:2  # Specify to append full combinations of
                                       # 2 components to camera_field.
```

**Note**: Only `unqualified` or `supported` components are used to generate
the combinations.

## Appendix

### Unprobeable Component (Deprecated)

**Deprecation note**: Unprobeable components are no longer allowed in new
HWID configuration files.

When a component with the attribute `probeable: False`, it means one of these:

- The component class is removed at the latest pattern.
- The value is determined at the evaluation time. Set at rule.
- Exception: If there is only one item in this component class, set the index
  to 0. It is only for backward compatibility.
  **Please use "default" argument instead of using this.**

### Default Component Item

If a component item has a `null` probe values (i.e. `values: null`), or has
the attribute `default: True`, it is a default component item. The feature of
the default component item is:

- It is always matched when generating HWID string, no matter the probe result
  exists or not.
- If another component is also matched, then the default component item will be
  ignored.

### Status of a Component Item {#component_status}

- **supported**

  Default status. The component is currently used to build new units.

- **unqualified**

  The component is now allowed to be used after PVT phase. The components added
  by the auto-generator should be unqualified. Then SIE or TAM should remove it
  after the component is confirmed.

- **deprecated**

  The component is no longer being used to build new units, but is supported in
  RMA process.

- **unsupported**

  The component is not allowed to be used to build new units, and is not
  supported in RMA process. The component cannot be matched. It is used for the
  wrong component or the equivalent component but with different probed result.

- **duplicate**

  The component is a subset of another component, so it should not be matched
  while encoding.  For example:

```
  comp_1:
    values:
      key: !re 'PART_NUMBER_1-.*'
  comp_2:
    status: duplicate  # this is necessary, otherwise 'PART_NUMBER_1-2' will
                       # match comp_1 and comp_2, and cause an error.
    values:
      key: PART_NUMBER_1-2
```

### Internal Only Database Builder Use Cases

The following use cases are **only for Googlers' test purpose**.  They are not
applicable for a regular ChromeOS device development journey.

#### 1. Manually Register a Minimal HWID Database for a New Project

**Deprecation note**: This use case is deprecated, please find the minimal
HWID database from the HWID bundle downloaded from ChromeOS Device Lifecycle
Management (DLM) page.  For HWID repository owners, please follow project
tracker instructions to create the minimal HWID database on DLM directly.

The steps of generating a minimal HWID database are:

1. Create a minimal DB with the project name and corresponding phase.
```shell
# Create a minimal DB
$ hwid build-database --minimal --project <proj_name> --image-id <initial_phase>
```
an example HWID DB will look like:
```yaml
checksum: <hash>

project: GOOGLE

encoding_patterns:
  0: default

image_id:
  0: EVT

pattern:
- image_ids:
  - 0
  encoding_scheme: base8192
  fields: []

encoded_fields:
  region_field: !region_field []

components:
  region: !region_component

rules: []
```

2. Add the project info in `projects.yaml` manifest file.
```yaml
GOOGLE:
    board: GOOGLE
    branch: main
    version: 3
    path: v3/GOOGLE
```
3. Submit the change of the two files.

#### 2. Convert Legacy Region Style to List Style
We use magic tag `"!region_field"` to generate the region encoded field. There
are two styles: legacy and list style. In legacy style, each region has its own
index mapping. For example, "us" maps to 29 and "gb" maps to 16. However, in
list style, only the listed regions are encoded. In the example below,
"us" maps to 1 and "gb" maps to 2.

```yaml
encoded_fields:
  # Legacy style
  old_region_field: !region_field
  # List style
  new_region_field: !region_field ['us', 'gb']
```

Because the update rule, we cannot just replace the legacy style because the
encoding mapping is different. If we want to convert legacy style to list
style, we should create another `encoded_field` and remove the old one from the
`pattern`. For example:

|||---|||
#### Before converting the style
```yaml
image_id:
  0: EVT

pattern:
- image_ids: [0]
  encoding_scheme: base8192
  fields:
  - region_field: 8

encoded_fields:
  region_field: !region_field

components:
  region: !region_component
```
#### After converting the style
```yaml
image_id:
  0: EVT
  1: DVT

pattern:
- image_ids: [0]
  encoding_scheme: base8192
  fields:
  - region_field: 8
- image_ids: [1]
  encoding_scheme: base8192
  fields:
  - new_region_field: 5

encoded_fields:
  region_field: !region_field
  new_region_field: !region_field ['us']

components:
  region: !region_component
```
|||---|||

The database builder will automatically convert the style if the old database
uses the legacy style.
*** note
**Note:** a new `image_id` is required when converting.
***