Running Custom Containers Under Chrome OS

Welcome to the containers project where we support running arbitrary code inside of VMs in Chrome OS. This is a heavily-technical document, but more user-friendly information will be coming in the future.

We won't get into technical details for specific projects as each one already has relevant documentation. We instead will link to them for further reading.

Overview

There are many codenames and technologies involved in this project, so hopefully we can demystify things here.

Crostini is the umbrella term for making Linux application support easy to use and integrating well with Chrome OS. It largely focuses on getting you a Terminal with a container with easy access to installing whatever developer-focused tools you might want. It's the default first-party experience.

The Terminal app is the first entry point to that environment. It‘s basically just crosh. It takes care of kicking off everything else in the system that you’ll interact with.

crosvm is a custom virtual machine monitor that takes care of managing KVM, the guest VM, and facilitating the low-level (virtio-based) communication.

Termina is a VM image with a stripped-down Chrome OS linux kernel and userland tools. Its only goal is to boot up as quickly as possible and start running containers. Many of the programs/tools are custom here. In hindsight, we might not have named it one letter off from “Terminal”, but so it goes.

Maitred is our init and service/container manager inside of the VM, and is responsible for communicating with concierge (which runs outside of the VM). Concierge sends it requests and Maitred is responsible for carrying those out.

Garcon runs inside the container and provides integration with Concierge/Chrome for more convenient/natural behavior. For example, if the container wants to open a URL, Garcon takes care of plumbing that request back out.

Sommelier is a Wayland proxy compositor that runs inside the container. Sommelier provides seamless forwarding of contents, input events, clipboard data, etc... between applications inside the container and Chrome. Chrome does not run an X server or otherwise support the X protocol; it only supports Wayland clients. So Sommelier is also responsible for translating the X protocol inside the container into the Wayland protocol that Chrome can understand.

You can launch crosh and use the vmc command to create new VMs manually. It will only run Termina at this point in time. You can use [vsh] to connect to a VM instance and use LXC to run containers.

Quickstart

Here's a quick run down of how to get started.

  • Buy a Google Pixelbook. It is our objectively biased opinion that it's a nice piece of hardware.
  • Switch to the dev channel.
  • Enable support.
    • Go to Chrome OS settings (chrome://settings).
    • Scroll down to “Linux (Beta)”.
    • Turn it on!
  • Open the app switcher (press the Search/Launcher key) and type “Terminal”.
  • Launch the Terminal app.
  • Profit!

If you're interested in Android Studio, check out their documentation.

Runtime Features

OK, so you've got your container going, but what exactly can you expect to work?

  • Outbound network connections (IPv4).
  • Unaccelerated Graphics.
  • Wayland programs (preferred).
  • X programs (compatibility via Sommelier).

Missing Features

There‘s a lot of low-hanging fruit we’re working on fleshing out.

  • Audio support.
  • Peripheral access (USB/Bluetooth/etc…).
  • Accelerated graphics.
  • Video hardware decoding.
  • IMEs.

There are more things we‘re thinking about, but we’re being very careful/cautious in rolling out features as we want to make sure we aren't compromising overall system security in the process. The (large) FAQ below should hopefully hit a lot of those topics.

Security

While running arbitrary code is normally a security risk, we believe we've come up with a runtime model that addresses this. The VM is our security boundary, so everything inside of the VM is considered untrusted. Our current VM guest image is also running our hardened kernel to further improve the security of the containers, but we consider this a nice feature rather than relying on it for overall system security.

In this model, the rest of the Chrome OS system should remain protected from arbitrary code (malicious or accidental) that runs inside of the containers inside of the VM.

The only contact with the outside world is via crosvm, and each channel talks to individual processes (each of which are heavily sandboxed).

User Data In The Container

With the shift to cloud services, current security thinking highlights the fact that getting account credentials (e.g. your Google/Facebook passwords) is way more interesting than attacking your desktop/laptop. They are not wrong. The current VM/container Chrome OS solution does not currently improve on this. Put plainly, anything entered into the container is the responsibility of the user currently. So if you run an insecure/compromised container, and then type your passwords into the container, they can be stolen even while the rest of the Chrome OS system remains secure.

Persistence

Linux apps do not survive logout (since they live in the user‘s encrypted storage). They also do not automatically start at login (to avoid persistent attacks), nor can they automatically run at boot (without a login session) since they wouldn’t be accessible (they‘re in the user’s encrypted storage).

Lifecycles

Once you've got the Terminal installed (which takes care of installing all the other necessary components like Termina), the system is ready to use.

By virtue of having things installed, nothing starts running right away. In that regard, when you log out, everything is shutdown and killed, and when you login, nothing is automatically restarted.

When you run the Terminal, the Termina VM will be started automatically, and the default Crostini container will be started in that. You can now connect to the container via SSH or SFTP (via the Files app).

Similarly, if you run a Linux application diretly (e.g. pinned to your shelf or via the launcher), the Termina VM will be started automatically, and the container that application belongs to will be launched. There's no need to run Terminal manually in these situations.

When you close all visible appliations, the VM/containers are not shutdown. If you want to manually stop them, you an do so via crosh and the vmc command.

Similarly, if you want to spawn independent VMs, or more containers, you can do so via crosh and the vmc and vsh commands.

Device Support

While we would like to be able to bring this work to all Chromebooks, the kernel and hardware features required limit where we can deploy this. A lot of features we use had to be backported, and the further back we go, the more difficult & risky it is to do so. We don't want to compromise system stability and security here.

Supported Now

The initial platform is the Google Pixelbook (eve) running an Intel processor (x86_64) with Linux 4.4.

Hardware Requirements

We are not planning on requiring a minimum amount of RAM, storage, or CPU speed, but certainly the more you have of each of these, the better off things will perform.

You will need a CPU that has hardware virtualization support.

Glossary

  • ARC (App Runtime for Chrome): The old/deprecated method of running Android apps in a Chrome NaCl (Native Client) sandbox. Had random compatibility issues.
  • ARC++ (Android Runtime for Chrome [plus plus]): The current method for booting Android in a container under Chrome OS.
  • Concierge: Chrome OS daemon that manages VM/container life cycles.
  • Container: A package (tarball/filesystem image/etc...) full of programs ready to be executed with some levels of isolation.
  • crosh (Chrome OS shell): A restricted developer shell for running a handful of commands.
  • Crostini: An umbrella name for providing a polished UI experience to run Linux apps.
  • crosvm: The Chrome OS Virtual Machine Monitor (akin to QEMU).
  • Garcon: Daemon in the container for passing requests between the container and Chrome via concierge.
  • KVM (Kernel Virtual Machine): The Linux interface for managing virtual machines.
  • kvmtool: A simple/fast virtualization tool.
  • LXC/lxd: Linux container solution.
  • Maitred: Agent that runs inside the VM and manages containers.
  • QEMU: A large/complete virtual machine emulator.
  • Sommelier: Wayland proxy compositor in the container that provides seamless forwarding of contents, input events, clipboard data, etc... between Linux apps and Chrome.
  • Termina: Codename for the custom VM that we boot.
  • Terminal: Public name for getting a full Linux command line environment and running Crostini.
  • VM (Virtual Machine): A way to boot a different operating system in a strongly isolated environment.
  • vsh: Launch a shell inside the VM (not inside of the container).
  • Wayland: The new graphics stack in the Linux world.
  • XWayland: An X server that outputs to Wayland.

FAQ

Where can I chat with developers?

All Chromium OS development discussions happen in our chromium-os-dev Google Group. Feel free to ask anything!

Where can I file feature requests?

As a nascent project, we‘ve got a lot on our plate and planning on releasing, so it’d be nice to hold off for now and check back in after a few Chrome OS releases.

Feel free to chat/ask on the mailing list above in the meantime.

Once we are in a more stable place, you can use our issue tracker. See the next question for details.

Where can I file bugs?

Please first make sure you're using the latest dev channel. A lot of work is still ongoing.

Next, please make sure the issue isn't already known or fixed. You can check the existing bug list.

If you still want to send feedback, you can file a feedback report and include #crostini in the description. Feedback about any part of Chrome OS can be filed with “Alt-Shift-i”.

If you still want to file a bug with the developers, use this link to route to the right people.

Can I boot another OS like Windows, macOS, Linux, *BSD, etc...?

Currently, no, you can only boot our custom Linux VM named Termina. See also the next few questions.

Can I run my own VM/kernel?

Currently, no, you can only boot Termina which uses our custom Linux kernel and configs. Stay tuned!

Can I run a different Linux distro?

Of course! The full LXD command line is available, and the included images remote has lots of other distros to choose from. However, we don't test with anything other than the default container that we ship, so things may be broken when running another distro.

I'm running , how do I get {gui apps, launcher icons, etc...}?

Sommelier and Garcon binaries are bind-mounted into every container, so no need to install or cross-compile. The systemd units and config files from cros-container-guest-tools will start these daemons in a systemd user session. It's also a good idea to run loginctl enable-linger <user> to allow these to remain running in the background.

Am I running Crostini?

If you're using the Terminal app, or programs in the default container we provide that includes our programs to ease integration (e.g. Sommelier), then yes.

If you're running your own container or VM, then no.

How do I share files between Chrome OS & the container?

Using Secure Shell, you can set up a SFTP mount to the remote container and then browse via the Files app. Work is on going to automate this step by default.

Can I access files when the container isn't running?

Currently, the container must be running in order to access its content.

Can I install custom kernel modules?

Currently, no, Termina does not include module support. That means trying to use software that requires building or loading custom kernel modules (e.g. VirtualBox) will not work. See the next question too.

Can I mount filesystems?

Currently, no (*). The containers are implemented using Linux user namespaces and those are quite restricted (by design). We're looking into supporting FUSE though.

(*): Technically you can mount a few limited pseudo filesystems (like memory-backed tmpfs), but most people aren't interested in those.

Can I run a VM inside the VM?

Currently, no, nested KVM is not supported. You could run qemu-system to emulate the hardware and boot whatever OS you want inside of that. Unfortunately, it‘ll be quite slow as QEMU won’t be able to utilize KVM for hardware acceleration.

Can I run a container inside the container?

Yes! You'll probably need to install the relevant packages first for whatever container format you want to run.

What container formats are supported?

Termina currently only supports LXC directly. We're aware of Kubernetes/Docker/OCI/rkt/etc... and hope to make them all easy to use.

See the previous question for a workaround in the mean time.

What architecture works on my system?

Since everything is all native code execution, it depends on the device you have.

If you don't know what device you have, you can find this out in two different ways:

  • Open chrome://settings/help/details and look at the Platform, then match the board name with our public device list. Look at the “User ABI” field to see what kind of CPU you have.
  • Open up crosh and run uname -m. This will print the architecture of your current device.

If you see x86_64, you'll be able to run code compiled for Intel/AMD (32-bit/64-bit/x32 should all work).

If you see arm (or something similar like armv7l) or aarch64, you'll be able to run code compiled for ARM/ARM64.

Can I run other architectures?

There is currently no integrated support for running e.g. ARM code on an Intel system, or vice-versa. You could handle this yourself (e.g. by using qemu-user), but if you're familiar with qemu-user, then you already knew that :).

How many VMs can I run?

You can spawn as many as your system can handle (RAM/CPU-wise). They are all independent of each other.

How many containers can I run?

You can spawn as many as your system can handle (RAM/CPU-wise). Each VM instance can host multiple containers.

Can I run programs that keep running after logout?

Nope! All VMs (and their containers) are tied to your login session. As soon as you log out, all programs are shutdown/killed by design.

Since all your data lives in your encrypted home, we wouldn't want that to possibly leak when you logout.

For more details, see the Security section in this doc.

Can I autorun programs when I login?

Nope! All VMs (and their containers) need to be manually relaunched. This helps prevent persistent exploits.

For more details, see the Security section in this doc.

Can I autorun programs when I boot?

Nope! See the previous questions, and the Security section.

Are my VMs/containers/data synced/backed up?

Currently, no, nothing is synced or backed up. You're responsible for any data going into the containers.

We hope to improve this situation greatly.

Can I use IPv6?

Unfortunately, only IPv4 is currently supported. Yes, we‘re fully aware that everything should be IPv6-compatible in 2018. We’re working on it.

Can I access layer 2 networking?

Currently, no, networking access is only at layer 3 (i.e. IP). So you won't be able to do any bridging or lower level fun stuff.

It‘s not clear if/when this will change. Bridging with the outside world is difficult with WiFi, and not many devices have Ethernet connections. We could support layer 2 between containers, but it’s not clear how many people want this in order to justify the effort involved.

Can I access hardware (e.g. USB/Bluetooth/serial)?

Currently, no, but we are working on it. Stay tuned!

Can I run graphical applications?

Yes, but currently things are unaccelerated. So if you‘re looking to play the latest Quake game, it’s not going to work well. See the next few questions.

Can I run Wayland programs?

Yes, and in fact, these are preferred! Chrome itself deals with Wayland clients heavily, and so you're much more likely to have things “just work” if you upgrade.

Can I run X programs?

Yes, via our Sommelier helper. We're still working out some compatibility kinks, and it probably will never be as perfect as running an X server, but with the community moving to Wayland, it should be good enough.

Why are windows sometimes tiny/fuzzy?

While Chrome supports high DPI displays, many Linux applications don‘t. When a program doesn’t properly support DPI scaling, poor results follow.

Currently we expose the native resolution and DPI directly to applications. If they show up tiny or fuzzy, it‘s because they don’t support scaling properly. You should report these issues to the respective upstream projects so that, hopefully someday, it'll “just work”.

In the mean time, Sommelier exposes some runtime settings so you can set the scale factor on a per-program basis to workaround the misbehavior. Check out Sommelier's documentation for more details.

If you're applying a system wide zoom or otherwise changing the default display resolution, we attempt to scale the application output to match. This can lead to blurry results. You can adjust the resolution of your display, or tweak things via Sommelier (see above for more details).

Can I run Windows programs?

Sure, give WINE a try. Compatibility will largely depend on WINE though, so please don't ask us for support.

Can I run Steam?

Sure, give Steam a shot. Just remember that without accelerated graphics or sound, it's probably not going to be too much fun.

Can I run macOS programs?

Probably not. You could try various existing Linux solutions, but chances are good that they are even rougher around the edges.

Can I develop Android apps (for ARC++)?

Check out the Android Studio site for more details on this.

Why implement crosvm from scratch (instead of using QEMU/kvmtool/etc...)?

We have nothing against any of these other projects. In fact, they're all pretty great, and their designs influenced ours. Most significantly, they did more than we needed and did not have as good a security model as we were able to attain by writing our own. While crosvm cannot do everything those other projects can, it does only what we need it to.

For more details, check out the crosvm project.

Why run VMs? Aren't containers secure?

While containers often isolate themselves (via Linux namespaces), they do not isolate the kernel or similar system resources. That means it only takes a single bug in the kernel to fully exploit the system and steal your data.

That isn't good enough for Chrome OS, hence we put everything inside a VM. Now you have to exploit crosvm via its limited interactions with the guest, and crosvm itself is heavily sandboxed.

For more details, see the Security section in this doc.

Don't Android apps (ARC++) run in a container and not a VM?

Unfortunately, yes, Android apps currently run only in a container.

We try to isolate them quite a bit (using namespaces, seccomp, alt syscall, SELinux, etc...), but at the end of the day, they have direct access to many syscalls and kernel interfaces, so a bug in there is reachable via code compiled with Android's NDK.

If Android apps are in a container, why can't users run code too?

We don't usually accept a low security bar in one place as a valid reason to lower the security bar everywhere. Instead, we want to constantly raise the security bar for all code.

Are Android apps (ARC++) going away?

There are no plans to merge the two projects. We share/re-use a lot of the Chrome bridge code though, so it‘s not like we’re doing everything from scratch.

Don't VMs slow everything down?

It is certainly true that VMs add overhead when compared to running in only a container or directly in the system. However, in our tests, the overhead is negligble to the user experience, and well worth the strong gains in system security.

For more details, see the Security section in this doc.

Why run containers inside the VM? Why not run programs directly in the VM?

In order to keep VM startup times low, we need Termina to be as slim as possible. That means cutting out programs/files we don't need or are about.

We use SquashFS to make the image smaller and faster to load, but it means the image/root filesystem is always read-only.

Further, the versions of programs/libraries we ship are frequently newer than other distros (since we build off of Gentoo), and are compiled with extra security flags.

It would also make it more difficult to have a stateless image that always worked and would be immune from user mistakes.

Altogether, it's difficult to support running arbitrary programs, and ends up being undesirable. Forcing everything into a container produces a more robust solution, and allows users to freely experiment without worry.

Also, we love turtles.

Can I disable these features?

Administrators can control access to containers/VMs via the management console, so enterprise/education organizations that want to limit this can.

Initially there is a “Linux (Beta)” option under the standard Chrome OS settings, but the long-term plan is to remove this knob so things work on-demand. At which point, there will be no knob for unmanaged devices.