This document describes the layout of data on the disk drive for a ChromiumOS device and the process by which the OS is booted.
Goals for the drive partitioning scheme are as follows:
Goals for the boot process are as follows:
ChromiumOS is essentially a specially-tailored GNU/Linux distribution. We want to make as few modifications to the upstream kernel as possible, ideally none. But as with any other GNU/Linux system, the pre-kernel boot process is unavoidably dependent on the hardware, BIOS, and bootloader.
Both ARM and (recent) x86 devices use U-Boot as their bootloader. On x86 we use Coreboot to set up RAM and load U-Boot. You can find an overview of the verified boot process in the U-Boot Porting Guide. U-Boot still uses the EFI partition table described below.
Legacy boot for x86 Linux has three steps:
Legacy BIOSes will continue to boot ChromiumOS from the MBR. The ChromiumOS build process places GPT-aware boot sector code from syslinux in the MBR. That code can specify one GPT partition to boot, indentified by a matching UniquePartitionGUID field in the Partition Entry Array. We use partition 12 for this purpose. The second-stage syslinux bootloader is installed on that partition, along with its corresponding config file (/syslinux/syslinux.cfg). We have a tool and scripts that can change the boot partition GUID in the MBR when we need to select an alternate boot path.
Virtualized systems (vmware, qemu, etc.) typically have their own legacy BIOS implementations and will use this method to boot ChromiumOS images.
The Extensible Firmware Interface is a BIOS replacement originally developed by Intel® for its Itanium® systems and later expanded to include x86 and other architectures. While not enthusiastically embraced by the Linux kernel developers, it offers some advantages over legacy BIOS and is becoming more widely used, especially for 64-bit x86 systems.
EFI BIOS boots like this:
The ChromiumOS build process creates an EFI System Partition (partition 12) and installs a 64-bit version of grub2 as the bootloader (/efi/boot/bootx64.efi), along with its config file (/efi/boot/grub.cfg). 64-bit EFI BIOSes will use this bootloader. It is possible to also install a 32-bit bootloader in the same partition, but we currently do not do that. To change the boot partition, we just need to edit the grub.cfg file. Note that different EFI BIOSes may have different requirements for the pathname of the bootloader.
Most EFI BIOSes contain a “Compatibility Support Module” component which makes them act like legacy BIOSes, so they may boot either way.
Google ChromeOS devices (x86/x86_64/arm) have custom BIOSes that use yet another boot method to ensure that the user is running only the bits that are intended. Instead of a separate bootloader and kernel, there is one binary blob contained in its own GPT partition. That blob is cryptographically signed and the signature is verified before booting. Under normal conditions, the process is:
The ChromiumOS build process creates signed kernel images needed by the ChromeOS BIOS and installs them in their own partitions. They are signed with test keys that are found in the source tree. Official releases will of course be signed with private Google keys.
For any booting (x86) configuration, there are at least three separate kernels (along with their command lines) on the disk image. Legacy BIOS will use syslinux, which uses its own copy of the chosen kernel that‘s kept in partition 12. EFI BIOSes will use /boot/vmlinuz from the target rootfs. ChromeOS BIOS uses the signed kernel embedded in its own partition. Our build and update process is carefully crafted to try to keep all three of these kernels in sync. However, if you’re fiddling with the kernel and commandline, you may find that your changes are being ignored. This is usually an indication that you‘re modifying the wrong one. In /proc/cmdline, you should see one of the strings “cros_legacy”, “cros_efi”, or “cros_secure”. These identify which method the kernel used to boot (and that’s all they do - we don't use them for any run-time decisions AFAIK).
Bootable ChromiumOS drives (removable or not) share a common drive format. In the discussion that follows, “sector” refers to a 512-byte disk sector, addressed by its Logical Block Address (LBA). Although the UEFI specs allow for disk sectors of other sizes, in practice 512 bytes is the norm. We do not use the old Cylinder-Head-Sector addresses at all.
The master boot record is the first sector on the hard drive (LBA 0). As mentioned above, legacy BIOSes will boot from this sector.
To protect the GUID partitions on the drive from legacy OSes, the MBR partition table normally contains a single partition entry of type 0xEE, filling the entire drive.
The second sector (LBA 1) contains the primary GPT header, followed immediately by 16K (32 sectors) of the primary GUID Partition Entry Array. In conformance with the EFI spec, another copy of these data should be located at the end of the disk as well, with the secondary GPT header in the last accessible sector and the secondary GUID Partition Entry Array immediately preceding it.
GPT allows a large number of partitions on a drive. In an attempt to reduce the effect that later partitioning changes might have on deployed systems, we are trying to enumerate the known partitions first, while leaving room for future growth. Here’s the current layout:
|1||user state, aka “stateful partition”||User's browsing history, downloads, cache, etc. Encrypted per-user.|
|2||kernel A||Initially installed kernel.|
|3||rootfs A||Initially installed rootfs.|
|4||kernel B||Alternate kernel, for use by automatic upgrades.|
|5||rootfs B||Alternate rootfs, for use by automatic upgrades.|
|6||kernel C||Minimal-size partition for future third kernel. There are rare cases where a third partition could help us avoid recovery mode (AU in progress + random corruption on boot partition + system crash). We decided it's not worth the space in V1, but that may change.|
|7||rootfs C||Minimal-size partition for future third rootfs. Same reasons as above.|
|8||OEM customization||Web pages, links, themes, etc. from OEM.|
|9||minios A||MiniOS recovery partition A (Only supported on disk layout v3 and newer).|
|10||minios B||MiniOS recovery partition B (Only supported on disk layout v3 and newer).|
|11||Powerwash data||Powerwash data, including enterprise rollback data (Only supported on disk layout v3 and newer).|
|12||EFI System Partition||Contains 64-bit grub2 bootloader for EFI BIOSes, and second-stage syslinux bootloader for legacy BIOSes.|
Note that the reserved partitions will actually be present on the image, so that the partition numbering remains constant from now on. Each minimal-size partition (including the C kernel and C rootfs) is only 512 bytes, and is shoved into some space lost to filesystem alignment (between the primary partition table and the stateful partition). 64M of empty space is set aside for use by those reserved partitions if they ever need it.
Bootable USB keys have the same layout, except that kernel B and rootfs B are minimal-size, and partition 1 is limited to 720M. The total USB image size is around 1.5G. When the USB image is installed on a fixed drive, the B image is duplicated from the A image, and partition 1 is made as large as possible so that the entire disk is in use.
The exact sizes and layouts are managed by a json file. See the Disk Layout Format page for more information.
Each GPT Partition Entry contains a PartitionTypeGUID to identify the purpose of the partition, a UniquePartitionGUID which is specific to an individual partition on an individual drive, a PartitionName (not the same as the filesystem's label, and apparently unused by the Linux kernel or userspace), and some Attributes bits that the ChromeOS BIOS will use to select the bootable image.
There are several standard PartitionTypeGUIDs. We use two of them, and we’ve created three new ones to identify the ChromeOS kernel and rootfs partitions and to reserve partitions for future use.
|Linux data (standard)||ebd0a0a2-b9e5-4433-87c0-68b6b72699c7|
|EFI System Partition (standard)||c12a7328-f81f-11d2-ba4b-00a0c93ec93b|
|ChromeOS future use||2e0a753d-9e48-43b0-8337-b15192cb1b5e|
At various times, Linux has used a number of means to refer to disk partitions. For the kernel command line, it may be by means of parameters like this:
root=/dev/sda3 root=LABEL=C-ROOT root=UUID=86f0f84d-e2rd0-41e7-ad44-df4faad61e73
For userspace mount points, those may correspond to paths like this:
/dev/sda3 /dev/disk/by-label/C-ROOT /dev/disk/by-uuid/86f0f84d-e2rd0-41e7-ad44-df4faad61e73
In those examples, when the kernel refers to a partition by its UUID, that UUID doesn’t come from the GPT. Each filesystem has its own UUID (and label), and that’s what the kernel looks at. Typically using the UUID notation requires starting udev in an initramfs, which takes extra time.
For legacy or standard EFI BIOSes, the /dev/fooN format is used, to keep boot times to a minimum. This must be specified in the bootloader config file. The ChromeOS BIOS and bootstub passes an additional argument on the kernel command line:
This allows the kernel to identify the GPT partition from which it was loaded. The root partition is the next higher partition.
The filesystem and kernel partitions are all 2MB aligned and sized. However, in the future we may move down to 1MB to be in sync with what other OSes are doing.
The physical layout of the partitions does not have to match their order in the partition table. In fact, there are reasons why it might be advantageous that it doesn't. For example it may be necessary to resize some partitions, which is made much easier with certain physical layouts. Refer to the Partition Resizing document for details.
Here’s the current fixed-disk layout:
Only ChromeOS BIOS will implement secure boot from first power-on. Portions of the firmware are read-only, forming the basis of trust to validate the read/write portions of the firmware. Once the firmware has been validated, we will continue the boot process by reading the kernel from the disk.
It is not possible to sign the GPT using public key encryption. The contents of the GPT (in particular, the partition-dependent attributes fields for the kernel GPT entries) will change as autoupdate applies updates and devices reboot and attempt to use newly updated partitions. Since the GPT is not signed and thus cannot be trusted, all firmware or software that accesses the GPT must pass security review.
Firmware needs to sanity-check all GPT values before using them. Most forms of corrupted or damaged partition tables will just cause the firmware to read a portion of the drive that doesn't contain a valid kernel signature header, in which case the firmware initiates recovery mode. But we must also protect against malicious GPT entries that might open security holes, so if the GPT is suspicious or corrupted in ways that can’t be repaired, we can’t boot this device.
There are at least two kernel partitions, to support autoupdate and accidental corruption. Each kernel partition is paired with a rootfs partition; kernel A should only boot rootfs A, kernel B should only boot rootfs B, etc. The kernel partition is separate from the rootfs partition so that:
The GPT Partition Entry contains a 64-bit Attributes field. Bits 48-63 are available for use by a partition of any given type.
ChromeOS Kernel partitions use the following attribute flags:
|56||Successful Boot Flag||Set to 1 the first time the system has successfully booted from this partition (see the File System/Autoupdate design document for the definition of success).|
|55-52||Tries Remaining||Number of times to attempt booting this partition. Used only when the Successful Boot Flag is 0.|
|51-48||Priority||4-bit number: 15 = highest, 1 = lowest, 0 = not bootable.|
|47-0||Reserved by EFI Spec|
Kernel partitions can be in the following states:
|State||Priority||Tries Remaining||Successful Boot Flag||Description|
|Active||A, where A>0||0||1||Kernel that has booted successfully at least once.|
|Backup||B, where A>B>0||0||1||Another kernel that has booted successfully but has lower priority than the active kernel.|
|Updated||C, where C>A>0||T>0||0||Newly updated kernel, which has not booted successfully yet. Since it has higher priority than the active kernel, it will be attempted next boot.|
|Not bootable||0||0||0||Kernel partition that is not currently bootable: in the process of being autoupdated, or ran out of boot tries before booting successfully, or failed its signature check.|
Using a Google-supplied library (in src/platform/vboot_reference/firmware), the BIOS searches the GPT to find the ChromeOS kernel with the highest Priority value and then runs the following checks on it:
If no valid kernel is found, we can’t boot this device.
After the OS finishes booting successfully, it will modify its partition table entry, ensuring that Successful Boot Flag == 1 and Tries Remaining == 0. We can edit the other attribute fields manually if we need to change the primary boot partition.
Here’s the flow in graphical form:
The same library that sanity-checks the GPT and selects the kernel partition also checks the kernel’s cryptographic signature. The kernel partition consists of the following structure:
The first 64K bytes are the cryptographic signature header blob, which contains the keys and signatures needed to verify the rest of the kernel blob (plus a few pointers and version numbers). The kernel blob consists of the 32-bit part of the Linux kernel, a config file (just the kernel command line string at the moment), a mostly-complete zeropage table, and our bootloader stub to complete the transition from BIOS to kernel.
As it’s verified, the kernel blob is copied into RAM starting at the 32-bit kernel entry location of 0x100000 on x86 (for ARM the address varies by sub-architecture). Once the verification is complete, the bootloader stub is invoked, which finishes initializing the params table and jumps to the kernel.
Developers may want to do a rapid turnaround of the kernel only. This suggested procedure may help:
rootdev -s. This shows where the rootfs is mounted.
emerge-x86-generic kernelor similar. You‘ll need the
vmlinuz) file to create the signed kernel partition image. It’s usually left in
config.txtfile, which will specify the kernel command line. You can make your own, or just reuse the one that’s left in
src/build/images/<board>/latest/by the last
vbutil_kernel --pack new_kern.bin \ --keyblock /usr/share/vboot/devkeys/kernel.keyblock \ --signprivate /usr/share/vboot/devkeys/kernel_data_key.vbprivk \ --version 1 \ --config config.txt \ --bootloader /lib64/bootstub/bootstub.efi \ --vmlinuz /build/x86-generic/boot/vmlinuz
new_kern.bininto partition 4 on the target (from console):
scp USER@SOMEWHERE:new_kern.bin /tmp sudo dd if=/tmp/new_kern.bin /dev/sda4
Sometimes all one needs is to change the kernel command line, for instance to enable or disable the verified rootfs. This can be done as follows (moving the kernel blob between the target and host is required in case the keys are not available on the target):
Move the kernel which needs modifying into a file (using the appropriate source device,
<src_part> below is most likely to be
sudo dd if=/dev/<src_part> of=/tmp/kernel.old
Save the old kernel command line to a file:
vbutil_kernel --verify /tmp/kernel.old --verbose | \ tail -1 > /tmp/cmd.line.old
Modify the command line as required and save it in a file (say
/tmp/cmd.line.new). Repack the kernel blob using the new command line:
vbutil_kernel --repack /tmp/kernel.new \ --config /tmp/cmd.line.new \ --signprivate <private_key> \ --oldblob /tmp/kern.old
For the recovery kernel on a removeable device,
<private_key> above is
recovery_kernel_data_key.vbprivk and for the main kernel on the hard drive, the
kernel_data_key.vbprivk. The full path to the key file is required, of course.
Then verify things look OK:
vbutil_kernel --verify /tmp/kernel.new --verbose
Finally get your kernel back to the device it came from:
sudo dd if=/tmp/kernel.new of=/dev/<src_part>