Chromium OS Recovery Kernels

This is used to recovery Chrome OS devices. It's responsible going through the entire flow and making sure the security of the process is not compromised.

Code flow overview

The init script is responsible for setting up and verifying the initial environment. It's not meant to do much more than that. It needs to make sure the TPM and firmware are in a good state (verified/locked) first. Then it hands off execution to recovery_init.sh.

The recovery_init.sh script is where the main execution logic all lives. It verifies the rootfs first, then sets up the mounts so it can execute the actual recovery logic. Then it hands off execution to the chromeos-recovery script which runs off and inside of the verified rootfs.

This means any new recovery logic should be added to the installer project. The only time the initramfs logic itself should be updated is if code needs to run before the rootfs verification step. This allows the logic to be written in a proper language (e.g. C++) rather than untestable shell, and avoids bloating the initramfs (which has severe size constraints).

All about image resources in recovery images

The code in messages.sh depends on a collection of PNG files stored in /etc/screens in the initramfs file system. The message images are generated by the make_images script. Other images are generated by the make_images script, or supplied by the chromeos-assets package.

Simple instructions if you just want to create images for review:

1. Run this command from inside the chroot: ./make_images
2. Point your browser at the ‘screens’ subdirectory in your sources (this should have been created in step 1.)
3. Review the content.

Note that the work must be done inside the chroot, in order to guarantee correct font selection.

Contents of /etc/screens, and what it's for

• screens/constants.sh
  This is a small shell fragment sourced from messages.sh. It contains parameter definitions relating to the image files.

  The shell code is generated automatically by the make_images script.

• screens/<locale>/<msgname>.png
  These are the localized individual message files displayed during recovery. is a standard locale name (e.g. “en-US”, “fr”, “zh-CN”, etc.). is the name used in “messages.sh”.

  The message source text for for each “.png” file can be found in localized_text/<locale>/<msgname>.txt.

• screens/progress_box.png
screens/progress_increment.png
  Images used for the progress bar in progress_bar. The progress_box image shows a progress bar at 0% completion. The progress_increment image is a column of pixels corresponding to a 1% increment.

  This file is generated by the make_images script.

• screens/boot_message.png
  This is a background image designed to frame all messages. The image shows a Chrome or Chromium logo at the top, with horizontal lines above and below the area for the messages.

  This file comes from the chromeos-assets package; see the ebuild for details.

• screens/spinner/*.png
  While we're installing the recovery image, we present a spinner animation. These files represent the individual frames of that animation.

  This file comes from the chromeos-assets package; see the ebuild for details.

Adding a new locale

All available locales are currently stored under localized_text; however, not all locales are included in the final output. Each Chrome computer is manufactured to be used in a specific locale; the ‘screens’ locale content is only generated for the locales to which units could actually be shipped.

The dictionary of supported locales and their associated fonts is stored in a global setting at the top of make_images. A new locale can be added by creating a mapping for the locale to an appropriate font. The list of font families assigned to a locale here should typically match the families used in Chrome's UI; see the IDS_UI_FONT_FAMILY_CROS string in Chrome.

Note that you should not accept on faith that simply updating the list of supported locales is enough to produce a good set of images. A number of things may go wrong:

• The text layout may not wrap lines properly, causing text to be truncated or otherwise mis-formatted.
• Text may be truncated for lack of space.
• Writing systems that display right-to-left are untested, and may not render properly. Most notable in this class are ‘ar’ (Arabic) and ‘iw’ (Hebrew). Give these special attention when/if they're added.
• Fonts may be missing, incomplete, or just plain illegible.
• The default font may or may not render the text correctly for the locale.
• The translators didn't have access to a Chromebook to observe the messages in context, and may have mis-translated some messages.
• Translators are human, and sometimes make mistakes.

When adding a new locale, you should at minimum visually inspect the images to confirm that it looks like the original text. If you can, it would be wise to find someone familiar with the target language to confirm there are no glaring errors.

How to generate content

All necessary graphic image content is generated automatically in the ebuild; if you intend to test simply by booting a device, you don‘t need to do anything. However, it’s a good idea to generate images manually and then use a browser to display the images. Below are the commands for the most likely use cases.

NOTE: To get proper font selection, the commands must be run inside the chroot.

To regenerate graphic images and messages for all supported locales:

./make_images

To regenerate graphic images and messages for specific locales, such as for locales not in the current supported list:

./make_images localized_text screens <locale> [ <locale> ... ]

This command will use the default font for unknown locales; this may or may not be right.

How to override user interface

The user interface for recovery program was made primarily for systems with display. For headless (without display) devices, you may need to override the console output and images (frecon) to serial console and other peripherals like LED or speaker.

To do this, you have to:

• Override the virtual/chromeos-bsp-initramfs with your own version
• Install additional files to /var/lib/initramfs/recovery
• Override the variables and functions defined in defaults.sh by /var/lib/initramfs/recovery/lib/board_recovery.sh

Note we only allow overriding variables and functions in defaults.sh. All other variables and functions are reserved internally and may change in future.

Debugging a Recovery image

These notes are intended as a bridge until all of this is made less difficult to work with. Tracking bug to make this less painful: https://crbug.com/715162.

Related reading:

• Creating a recovery image that has been modified for test
• Chromium OS [serial] Console Debugging HOWTO
• Controlling Enabled Consoles
• Servo
• Case Closed Debugging with Suzy-Q (Google-internal and partner only)
• Various board-specific “Care and Feeding” documents (Google-internal and partner only)

Creating a recovery image that has been modified for test is the starter document for this. However, serial console debugging is not part of those instructions yet. Further, the flags that are needed cannot (yet) be passed on to the parts of the build system that need changes. Here are the changes that are needed to make serial console work inside of Recovery:

1. Build a serial-console-enabled test image. Doc: built a test image with serial console debugging enabled. That page has been updated with a lot of clarifications that help with confusion.

2. The mod_image_for_recovery.sh script takes the test image as input and then invokes parts of the build process to convert this image to a Recovery image. We need to modify this script to enable serial console. Open this script in a text editor (no need to cros_workon).

   1. Look for a section like the following and add the --enable_serial line below:

      ${SCRIPTS_DIR}/build_kernel_image.sh
        --board="${FLAGS_board}"
        --arch="${ARCH}"
        --to="$RECOVERY_KERNEL_IMAGE"
        --vmlinuz="$vmlinuz"
        --working_dir="${IMAGE_DIR}"
        --boot_args="noinitrd panic=60 cros_recovery…
        --enable_serial=ttyS2
        --keep_work
        --keys_dir="${FLAGS_keys_dir}"
        ${enable_rootfs_verification_flag}
        --public="recovery_key.vbpubk"
        --private="recovery_kernel_data_key.vbprivk"
        --keyblock="recovery_kernel.keyblock" 1>&2 …
      

      The specific ttyS# will vary by test board (see “Care and Feeding” doc for the board you are debugging on).

   2. Look for a line that looks like the following and add the pcserial or samsung_serial flag like this:

      RECOVERY_KERNEL_FLAGS="... i2cdev vfat pcserial"
      

      Note that pcserial is only for x86 platforms.

3. Run cros_workon --board=${BOARD} chromeos-base/chromeos-initramfs.

4. Edit the Recovery init script to send debug output to the console (and, in turn, the serial console). Open src/platform/initramfs/recovery/init in a text editor. In the main() function, comment out a line that looks like:

   #exec >"${LOG_FILE}" 2>&1
   

5. Finally, build the recovery image from within the chroot:

   sudo ./mod_image_for_recovery.sh --board=${BOARD} --nominimize_image \
     --image ~/trunk/src/build/images/${BOARD}/latest/chromiumos_test_image.bin \
     --to ~/trunk/src/build/images/${BOARD}/latest/recovery_test_image.bin
   

   You don’t need to build_packages or build_image: initramfs is rebuilt during mod_image_for_recovery.sh. (Note: nominimize_image allows for the preservation of the contents of the stateful partition during install.)

Finally, you’re ready to boot your image:

1. cros_flash the ~/trunk/src/build/images/${BOARD}/latest/recovery_test_image.bin file to USB normally. Note that you cannot network install a recovery image for testing.

2. On your board, CTRL-U to USB boot. Don’t use the recovery boot mode: we haven’t signed this image. USB boot is sufficient for testing (so long as verified boot is not part of your test).

Back on your Servo or SuzyQ-connected host:

1. Make sure that servod is running per the Servo instructions. This also works for SuzyQ.

2. After it’s running, run dut-control cpu_uart_pty to obtain the PTY that has the console output on it.

3. Run cu -l /dev/pts/# to attach.

4. You should see the console output from the init script. The output is “sh -x”-style shell execution tracing.

Unpacking an initramfs for inspection

1. In some directory in chroot: mkdir ./tmp-initramfs

2. cd tmp-initramfs

3. xzcat /build/${BOARD}/var/lib/initramfs/recovery_ramfs.cpio.xz | cpio -i