This is a Firmware test manual targeted for any tester needing to work with AP, EC, or GSC firmware.
servod and test_that commands.generate_logs and the script will print the location at the end in the form of /tmp/log-*.tar.gz. You will need to attach this log to the bug.Template drop down select [FW Test] with Component Chrome OS > Firmware.Run generate_logs on the device, scp the generated .tar.gz file to your desktop and attach it to the bug.
First, you must locate the target PTY (pseudo-TTY) to use for console communication.
dut-control usbpd_uart_pty.dut-control ec_uart_pty to retrieve the pty.dut-control cpu_uart_pty.Once the desired PTY is known, multiple different tools can be used to connect to that interface.
sudo emerge minicom.minicom -D /dev/pts/{PTS} to open console.Ctrl+A and q to exit.cu -s 115200 -l /dev/pts/{PTS}.Return ~~ and return to exit.screen /dev/pts/{PTS} 115200.Ctrl+A and quit to exitUnless otherwise specified, device (sometime refer to as DUT - Device Under Test) are assumed to be setup with Chrome OS test image and updated with the target test firmware.
If your DUT is running a production cr50 image, see the Closed Case Debugging documentation.
chrome://system from browser tab.bios_info section and click on the Expand... button next to it.mainfw_type should be set to normal or developercrossystem mainfw_type at the prompt, the value should be normal, developer or recovery.crossystem mainfw_type.developerOS verification is OFFPress SPACE to re-enableOS verification is OFFPress ENTER to confirm you wish to turn OS verification on.Your system will reboot and local data will be cleared.To go back press ESC.OS verification is ONYour system will reboot and all data will be cleared.Press the following key sequence to put the device into recovery mode
Esc + F3 + Power (F3 is reload). On device like pixel, you will need to hold the keys for more than 200 ms. On some device you will need to press the Power button last and release first.Chrome OS is missing or damaged. Please insert a recovery USB stick or SD card.OS verification is OFF, the device is already in developer mode. On older chromebox, flip the switch on the back to go to the developer mode.normalEsc + F3/Refresh + Power to enter recovery mode. The screen should display:Chrome OS is missing or damaged.Please insert a recovery USB stick or SD card.To turn OS verification OFF, press ENTER.Your system will reboot and local data will be cleared.To go back, press ESC.OS verification is OFFPress SPACE to re-enable.Esc+F3+Power at the login screen.Ctrl+D at Chrome OS is missing or damaged..To turn OS verification OFF, press ENTER..OS verification is OFF.Your system is transitioning to Developer Mode..Preparing system for Developer Mode. after 30 seconds.Firmware Shellball is a shell archive or shar. Located on device as /usr/sbin/chromeos-firmwareupdate.
Refer to the Firmware Updater guide for chromeos-firmwareupdate mode options.
Refer to the Firmware Updater Package guide for more information.
/tmp, example: mkdir /tmp/fw_binariescp /usr/sbin/chromeos-firmwareupdate /tmp/fwupdaterFollow the below instructions to repack the shellball
DIR using --unpack <DIR> command, example: chromeos-firmwareupdate --unpack /tmp/fw_binaries/<DIR>/images, example cp bios.bin ec.bin /tmp/fw_binaries/images<DIR>/models/<MODEL>/setvars.sh), example: IMAGE_MAIN="images/bios.bin"<DIR>/images and <DIR>/models directories, so just replace the bios.bin, ec.bin, pd.bin in top folder (<DIR>/), example: cp bios.bin ec.bin /tmp/fw_binaries/--repack <DIR> command, example: chromeos-firmwareupdate --repack /tmp/fw_binaries//usr/share/vboot/bin/make_dev_ssd.sh --remove_rootfs_verification --partitions 2 to make rootFS writable.reboot.crossystem ro_fwidcrossystem fwidectool versionectool --name=cros_pd version OR ectool --dev 1 versiongsctool --any --fwverchromeos-firmwareupdate -Vvpd -l (if empty, there’s nothing to save).flashrom -p host -r -i RO_VPD:/tmp/ro_vpd.bin.bios.bin and ec.bin files to /tmp on DUT. sudo flashrom -p host -w /tmp/bios.bin.flashrom -p host -w -i RO_VPD:/tmp/ro_vpd.bin --fast-verify.sudo flashrom -p ec -w /tmp/ec.bin.sudo flashrom -p ec:type=pd -w /tmp/pd.bin.The normal test firmwares are signed with “RO-NORMAL” firmware preamble flag by default. In other words, they won't boot from RW section unless you turn off the flag. To turn off, go to chroot:
cd /mnt/host/source/src/platform/vboot_reference/scripts/image_signing
./resign_firmwarefd.sh {PATH_TO_yourimage.bin} {PATH_TO_output.bin} \
../../tests/devkeys/firmware_data_key.vbprivk \
../../tests/devkeys/firmware.keyblock \
../../tests/devkeys/dev_firmware_data_key.vbprivk \
../../tests/devkeys/dev_firmware.keyblock \
../../tests/devkeys/kernel_subkey.vbpubk \
1 0
The PATH_TO_output.bin will contain the resigned firmware image. Note the final “0” in parameter list is the one that cleared preamble flag. You should see a message “Using firmware preamble flag: --flag 0” during execution of resign_firmwarefd.sh.
cros_sdk.sudo dd if=/dev/sdd of=/tmp/chromiumos_image.bin bs=1M./mnt/host/source/src/scripts/mount_gpt_image.sh -f /tmp -i chromiumos_image.bin./tmp/m/usr/sbin/chromeos-firmwareupdate -V for firmware version.cat /tmp/m/etc/lsb-release for Chrome OS version./mnt/host/source/src/scripts/mount_gpt_image.sh -u to unmount.grep ‘^ME:.*version’ /sys/firmware/log.For details about write protect, and how to enable and disable the feature, please read Write Protection.
This only apply to device with PD. (Example glados, samus, oak platforms). To enable PD write protection:
flashrom -p ec:type=pd --wp-enableectool --name=cros_pd reboot_ec RO at-shutdownflashrom -p ec:type=pd --wp-status and note that the protect range len is not 0ectool --name=cros_pd flashprotect and note that wp_gpio_asserted is on.To create a USB stick with a Chrome OS image, follow the instructions in the developer guide at Put your image on a USB disk.
If using a prebuilt image instead of one that is locally built, do the following:
tar xvf *.tar.xz.$ cd ~/chromiumos cros flash usb:// chromiumos_test_image.binTo boot from a USB disk, follow the instructions at Boot from your USB disk in the Chromium OS Developer guide.
Test cases will refer to various SCREEN, here is a list and it's corresponding string.
Chrome OS is missing or damaged. Please insert a recovery USB stick or SD card.Ctrl+D)OS verification is OFF Press SPACE to re-enable.Ctrl-U)The device you inserted does not contain Chrome OS.Please remove all external devices to begin recovery.To turn OS verification OFF, press ENTER. Your system will reboot and local data will be cleared. To go back, press ESC.Press ENTER to confirm you wish to turn OS verification on. Your system will reboot and local data will be cleared. To go back, press ESC.OS verification is on. Your system will reboot and local data will be cleared.Your system is applying a critical update. Please do not turn off.New revised recovery screen for most device created post 2016.
Chrome OS is missing or damaged. Please remove all connected devices and start recoveryPlease insert a recovery USB stick or SD card.There are 4 types of terminals.
Ctrl + Alt + F1 (top row back arrow key).Ctrl + Alt + F2 (top row forward arrow key).Ctrl + Alt + F3 mostly for console log output.Ctrl + Shift + T.GBB flags are located in a read-only region of firmware, and control various firmware features (developer mode, EC software sync, etc). They can be manipulated for ease of developer and testing experience:
/usr/share/vboot/bin/set_gbb_flags.sh 0 followed by reboot. If set_gbb_flags.sh returns error, you will need to make the BIOS writeable and retry.flashrom -p host -r /tmp/bios.bin gbb_utility --get --flags /tmp/bios.bintpm_fwver and tpm_kernver.F3/Refresh + Power to reset EC. Note that this is not applicable to chromebox and wilco devices.F3/Refresh + Power will reset EC and also put the device in recovery mode.TPM value mismatch will prevent you from booting Chrome OS. Run crossystem tpm_fwver tpm_kernver to check TPM versions. If the values mismatch, follow the below procedure to reset the values.
ESC + Refresh/F3 + Power and insert a test image to boot.chromeos-tpm-recovery.reboot the device upon success.crossystem tpm_fwver tpm_kernver.Run power_supply_info from VT2 to report power related stat like details on battery state, battery charge percentage, if AC power is plugged in.
ip addr.A factory error has been detected, this indicates there is a hardware ID mismatch and mostly due to installation of test image. Either skip for now OR use gbb_utility -s --hwid="DESIRED_HWID" to update the HWID in GBB flags and reflash firmware.Refer to the “LED behavior” section of the latest requirements for the relevant form factor. See the Latest Chromebook requirements for clamshell devices; that page has links to requirements for other form factors.
fdisk -l to locate the USB device name./dev/sdb or /dev/sda1.mount device_name /mntto mount USB stick as /mnt, for example mount/dev/sda1 /mnt.flashrom -r -i SECTION:PATH where PATH is the location of the file to save and SECTION is one of the name listed in data_fmap_expect_p.txt.flashrom --help for more details.Before putting away the device for long term storage, open VT2 and run ectool batterycutoff.
In order for the firmware to help, we will need to know the following details:
On recent firmware (both x86 and arm), a new feature is added to help diagnosing boot (recovery screen) failure. When you see the BIOS screen (whether a blue/pink developer screen or recovery screen), try to press the TAB key. Some debug message should appear on left-top corner of the screen, including HWID string and a code for the reason why your device enters recovery mode.
Find the message “Recovery Reason: 0x??”, where ?? is a hex number; then find the corresponding entry in this document (or the source header).
Here is the list of well-known recovery reasons and possible solutions.
chromeos-tpm-recovery.chromeos-tpm-recovery.chromeos-firmwareupdate --mode=recovery --forceTo boot into system, the verified firmware must complete following:
The verified boot firmware has changed the recovery code several times. For old devices, failure in step 1~4 may result in error code 0x48, 0x5a, 0x42, or other values. For most devices today the error codes for each steps listed above are:
0x5a VB2_RECOVERY_RW_NO_DISK0x5b VB2_RECOVERY_RW_NO_KERNEL0x5b VB2_RECOVERY_RW_NO_KERNEL0x43 VB2_RECOVERY_RW_INVALID_OSThe difference between 0x5b “No bootable kernel found on disk” and 0x43 “OS kernel or rootfs failed signature check” is if the firmware can find at least one partition table entry with “Chrome OS kernel” type and proper boot priority attributes. However, the two errors can be particularly frustrating to debug, as there are many potential causes:
When on the error screen, press [tab] to get debug information and collect.
Since there are three potential reasons to the failure, to debug we need to read back the firmware and disk contents of the device. Connect servo (or enable CCD) and then:
image.bin.# Copy the image file readback from device to a new one. cp image.bin image_0x38.bin # inside chroot futility gbb -s --flags 0x38 image_0x38.bin
image_0x38.bin back to the device using Servo (or CCD).# Find the right internal storage on your device. That can be: # eMMC = /dev/mmcblk0, mmcblk0p2, mmcblk0p4 # SATA/SSD = /dev/sda, sda2, sda4 # NVMe = /dev/nvme0n1, nvme0n1p2, nvmp0n1p4 # The example below is assuming you have eMMC: cgpt show /dev/mmcblk0 >/tmp/gpt.txt dd if=/dev/mmcblk0p2 bs=1M | gzip -c >/tmp/kern.2.gz dd if=/dev/mmcblk0p4 bs=1M | gzip -c >/tmp/kern.4.gz
image.bin, gpt.txt, kern.2.gz, kern.4.gz) and upload them to report the issue for analysis.If the device goes to the recovery page with “OS missing” errors immediately after running factory finalization, that may be caused by “unexpected reboot or failure during finalization”, since in the process we will de-active test image partitions (2) and enable release partition (4). Check the factory logs for this.
As a reference, we have seen few finalization issues that failed after GBB was cleared (and before the new release partition was activated):
If the DUT goes to 0x43 during run-in tests, it's usually caused by DMVERROR or other storage driver issues. The following sections will help you to get more details.
Read gpt.txt and look for partition 2 and 4, which should look like
69 65536 2 Label: "KERN-A" Type: ChromeOS kernel UUID: 8AF3A85D-AC80-A44F-AA0A-CA8C73E66811 Attr: priority=1 tries=0 successful=1 ... 65605 65536 4 Label: "KERN-B" Type: ChromeOS kernel UUID: 0BC160CA-E154-1749-9F79-D5838FAAD064 Attr: priority=0 tries=15 successful=0
There must be at least one partition with priority > 0 and (tries > 0 or successful = 1). Use that partition for the following tests.
Note for devices failed immediately after factory finalization, they should have active = 4. For devices failed recovery, active = 2. For failure after OOBE or Auto Update, both partitions may be active, so choose the one with higher priority.
The examples below will assume your active one is 4 (kern.4.gz).
Run the command in chroot to check:
gunzip kern.4.gz futility vbutil_kernel --verify kern.4
If the kernel looks good you should see some information like:
Signature: ignored Size: 0x4b8 Flags: 7 !DEV DEV !REC Data key algorithm: 4 RSA2048 SHA256 Data key version: 1 Data key sha1sum: d6170aa480136f1f29cf339a5ab1b960585fa444 Preamble: Size: 0xfb48 Header version: 2.2 Kernel version: 1 Body load address: 0x100000 Body size: 0x7d7000 Bootloader address: 0x8d6000 Bootloader size: 0x1000 Flags : 0 Body verification succeeded. Config: console= loglevel=7 init=/sbin/init cros_secure root=/dev/dm-0 ......
Otherwise, the kernel is either corrupted with unknown reason or by DMVERROR.
Print the first 8 bytes of kernel blob to check.
dd if=kern.4 bs=1 count=8 2>/dev/null
A real Chrome OS kernel should have CHROMEOS.
For devices failed due to DM verity error, that will be DMVERROR. You will want to run a memory stress test (stressapptest) to see if this is caused by memory issues. Otherwise, it may be failure by storage driver, or even hardware failure inside storage.
If you see anything else (or nothing), it is probably fully corrupted or an empty partition.
In normal mode, firmware and storage partitions must be signed with matching keys for boot to succeed. If the keys mismatch, you may encounter this error.
The debug screen contains key information in the following fields:
gbb.rootkeygbb.recovery_key These keys correspond to entries in the platform/chromeos-hwid repository. For newer boards, the corresponding file is v3/${BOARD}. See entries under the firmware_keys section for valid values for a given device. Also, if you see b11d74edd286c144e1135b49e7f0bc20cf041f10 as root key then your firmware is using DEV (also known as “unsigned”) keys.To change the keys on a device, disable hardware write-protect and flash a recovery image with the desired key. See the Firmware Write Protection document for details.
If you can't access chromeos-hwid or are not sure which keys were installed on the DUT and just want to know if the boot failure is caused by keys mismatch, follow the steps:
futility gbb --rootkey=rk.bin image.bin futility dump_fmap image.bin -x VBLOCK_A:vb.a VBLOCK_B:vb.b FW_MAIN_A:fw.a FW_MAIN_B:fw.b futility vbutil_firmware --verify vb.a --signpubkey rk.bin --fv fw.a --kernelkey kk.a futility vbutil_firmware --verify vb.b --signpubkey rk.bin --fv fw.b --kernelkey kk.b futility vbutil_kernel --verify kern.4 --signpubkey kk.a futility vbutil_kernel --verify kern.4 --signpubkey kk.b
At least one of kk.a or kk.b should decode kernel partition properly with something like:
Keyblock: Signature: valid Size: 0x4b8 Flags: 7 !DEV DEV !REC Data key algorithm: 4 RSA2048 SHA256 Data key version: 1 Data key sha1sum: d6170aa480136f1f29cf339a5ab1b960585fa444 Preamble: Size: 0xfb48 Header version: 2.2 Kernel version: 1 Body load address: 0x100000 Body size: 0x7d7000 Bootloader address: 0x8d6000 Bootloader size: 0x1000 Flags : 0 Body verification succeeded. Config: console= loglevel=7 init=/sbin/init cros_secure root=/dev/dm-0 ...
If you can‘t verify the active kernel by either firmware (in A or B), then it’s probably caused by firmware key mismatch.
The simplest cause of this error is that there is no valid image on the disk. To confirm if this is the case, you should follow the following steps:
dd if=/dev/mmcblk0p2 of=/usr/local/dump_2.bin bs=8 count=1 and hexdump -n 8 /usr/local/dump_2.bin -c dd if=/dev/mmcblk0p4 of=/usr/local/dump_4.bin bs=8 count=1 and hexdump -n 8 /usr/local/dump_4.bin -cBoth should return with the string “CHROMEOS”. If not, the partition is corrupted, and that is what is preventing boot.
Consult the Chrome OS Disk format document for more details on expected image format.
Data corruption from storage can cause signature checking to fail. This is a good case to check for if using a new storage part or the hardware design is early in development. This is especially suspicious if the system boots reliably over USB.
To diagnose this potential issue, boot an image from a USB stick and test storage via using the test used for hardware qualification: test_that --iterations 100 $IP hardware_StorageFio.hwqual
Bad hardware strap information for H1 can disrupt communication between cr50 and the AP (Application processor). The cr50 console will log strapping information at boot, with a line like this: [0.006304 Valid strap: 0x2 properties: 0x21]
If the strapping configuration is invalid, instead, a line like this will be emitted: [0.006244 Invalid strap pins! Default properties = 0x46]
cr50 sends pulses on the interrupt line that are ~2.6us in length. SoCs should be careful to manage clock gating registers to ensure that pulses that short may be detected, otherwise TPM communication may fail.
This document describes the process to restore your device when the device no lo nger boots or is stuck at the recovery screen. The general problem is usually a key mismatch between OS and firmware. There are three main types of keys: MP, pre-MP and Dev (test). Special care is needed with key changes. Here is a table on when you need special care and when it is a simple upgrade.
Your device must be in working order in order to switch OS with a different key. Once you are stuck at the recovery screen you will need to perform extra steps to restore the device to working order.
The fool-proof method is to install a dev sign OS and firmware before switching to the new OS. Here are the high level steps:
VT2, run chromeos-firmwareupdate --mode=factory to update dev signed firmware on deviceesc + refresh + powerStarts from “DUT able to boot to CROS” and ends at “Reboot and done”
cat /etc/lsb-release and check the value of CHROMEOS_RELEASE_BOARD to determine your image key. For example, on swanky, CHROMEOS_RELEASE_BOARD is set to swankey-signed-mpkeys which indicates that the device has MP signed image on the system. You will need this information in order to determine which path to take below. If you don’t know there is a way to find out in Step 2.
There are 3 types of keys: MP, PreMP and Unsigned (aka test image). If your source image key and target image key is the same, you will have no issue. If the source and target is different you will need special care. It is rare that you will need to go from MP key to PreMP key, if you must, switch to dev image first.
Fully working means you can boot to Chrome OS successfully. If your DUT can boot to Chrome OS, skip to section “Migrate your device” and follow the directions.
You will need to restore your device to a working order before continue. You will need to reinstall the device with an OS signed with the same key type. Once you have done that, you can go to “Migrate your device” and follow the directions.
There are two ways to determine the original image key of your DUT if you can get to the OS:
gbb.rootkey, locate key in factory firmware and get key typeTo get gbb.rootkey, press tab at the recovery screen.
To find the HWID file that describes the keys for a given device, go to the chromeos-hwid repository and find the desired file for the given device name.
Below are list of high level steps.
You will need test firmware, and a USB stick with the target OS image flashed onto it.
crossystem wpsw_cur from VT2 should return 0 to indicate you have successfully disabled write protect.flashrom -p [ host | ec ] --wp-status should return disabledflashrom -p ec:type=pd --wp-statuscrossystem dev_boot_usb=1chromeos-firmwareupdate --mode=factorychromeos-installtpm_fwver or tpm_kernver != 0x10001Same instructions as the section above.
You will need an USB stick with the unsigned Chrome OS image.
flashrom -p [ host | ec ] --wp-disablechromeos-firmwareupdate --mode=factorychromeos-installshutdown -P 0 to shutdowntpm_fwver or tpm_kernver > 0x10001Same as Reset DUT with unsigned image except you will need to boot your device to Chrome OS with:
Device in dev mode
crossystem dev_boot_usb=1