| commit | dd8f8dbc3030d5a2dc5225ea068e76a70f4eb1c0 | [log] [tgz] |
|---|---|---|
| author | Ken Rockot <rockot@google.com> | Fri May 20 21:37:05 2022 |
| committer | Takuto Ikuta <tikuta@google.com> | Fri Dec 02 01:45:28 2022 |
| tree | cf2f725ffa1ddca86fed4a5d7c73fec3735c9006 | |
| parent | 6fea57cc04d5b2f381f2380b8bf14a4efb97a82d [diff] |
Reland "ipcz: OS primitives for reference drivers" This is a reland of commit a0ce43fb810d0f706b921663ca7b8793c1bb4e1e Deltas from the original CL: - Fixes macOS debug builds where assertions were stupidly doing infinite recursion. Not originally caught by CQ because ABSL_ASSERT() is not enabled when DCHECK is on. Yay. - Forces ipc_tests to target Ubuntu 18 for some MSan bots, because for some reason this change elicits a dependency on glibc 2.27 and Ubuntu 16 is at 2.25. Original change's description: > ipcz: OS primitives for reference drivers > > In preparation for landing an async multiprocess reference driver, this > lands some foundational primitives first. These are essentially cheesier > versions of what we find in Chromium for scoped handles and shared > memory. > > As with all reference driver code, these are not intended for production > use, but for documentation and use in tests. > > Bug: 1299283 > Change-Id: I18357f3401073b2c4d4f9711737ab6a67ec64e43 > Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/3648395 > Commit-Queue: Ken Rockot <rockot@google.com> > Reviewed-by: Daniel Cheng <dcheng@chromium.org> > Cr-Commit-Position: refs/heads/main@{#1005512} Bug: 1299283 Change-Id: I459b642ae264dbfe145227f4b5f9a95768f2757f Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/3656572 Auto-Submit: Ken Rockot <rockot@google.com> Reviewed-by: Daniel Cheng <dcheng@chromium.org> Reviewed-by: Ben Pastene <bpastene@chromium.org> Commit-Queue: Daniel Cheng <dcheng@chromium.org> Cr-Commit-Position: refs/heads/main@{#1005968} NOKEYCHECK=True GitOrigin-RevId: 83999b6b8da2d6bdf07d583b3a41091b1c45ef95
ipcz is a fully cross-platform C library for interprocess communication (IPC) intended to address two generic problems: routing and data transfer.
With ipcz, applications create pairs of entangled portals to facilitate bidirectional communication. These are grouped into collections called nodes, which typically correspond 1:1 to OS processes.
Nodes may be explicitly connected by an application to establish portal pairs which span the node boundary.
┌───────┐
Connect │ │ Connect
┌──────────>O A O<───────────┐
│ │ │ │
│ └───────┘ │
│ │
v v
┌───O───┐ ┌───O───┐
│ │ │ │
│ B │ │ C │
│ │ │ │
└───────┘ └───────┘
Here nodes A and B are explicitly connected by the application, as are nodes A and C. B can put stuff into its portal, and that stuff will come out of the linked portal on A.
But portals may also be sent over other portals. For example, B may create a new pair of portals...
┌───────┐
Connect │ │ Connect
┌──────────>O A O<───────────┐
│ │ │ │
│ └───────┘ │
│ │
v v
┌───O───┐ ┌───O───┐
│ │ │ │
│ B │ │ C │
│ O───O │ │ │
└───────┘ └───────┘
...and send one over B's existing portal to A:
┌───────┐
Connect │ │ Connect
┌──────────>O A O<───────────┐
│ ┌────────>O │ │
│ │ └───────┘ │
│ │ │
v v v
┌───O─O─┐ ┌───O───┐
│ │ │ │
│ B │ │ C │
│ │ │ │
└───────┘ └───────┘
Node A may then forward this new portal along its own existing portal to node C:
┌───────┐
Connect │ │ Connect
┌──────────>O A O<───────────┐
│ │ │ │
│ └───────┘ │
│ │
v v
┌───O───┐ ┌───O───┐
│ │ │ │
│ B O────────────────────────O C │
│ │ │ │
└───────┘ └───────┘
As a result, the application ends up with a portal on node B linked directly to a portal on node C, despite no explicit effort by the application to connect these two nodes directly.
ipcz enables this seamless creation and transferrence of routes with minimal end-to-end latency and amortized overhead.
ipcz supports an arbitrarily large number of interconnected portals across the system, with potentially hundreds of thousands of individual portal pairs spanning any two nodes. Apart from managing how these portal pairs route their communications (i.e. which nodes they must traverse from end-to-end), ipcz is also concerned with how each communication is physically conveyed from one node to another.
In a traditional IPC system, each transmission typically corresponds to a system I/O call of some kind (e.g. POSIX writev(), or WriteFile() on Windows). These calls may require extra copies of transmitted data, incur additional context switches, and potentially elicit other forms of overhead (e.g. redundant idle CPU wakes) under various conditions.
ipcz tries to avoid such I/O operations in favor of pure userspace memory transactions, falling back onto system I/O only for signaling and less frequent edge cases. To facilitate this behavior, every pair of interconnected nodes has a private shared memory pool managed by ipcz.
To set up a new local repository, first install depot_tools and make sure it's in your PATH.
Then from within the repository root:
cp .gclient-default .gclient gclient sync
When updating a local copy of the repository, it's a good idea to rerun gclient sync to ensure that all external dependencies are up-to-date.
ipcz uses GN for builds. This is provided by the depot_tools installation.
To create a new build configuration, first create a directory for it. For example on Linux or macOS:
mkdir -p out/Debug
Then run gn args to create and edit the build configuration:
gn args out/Debug
For a typical debug build the contents may be as simple as:
is_debug = true
Now targets can be built:
ninja -C out/Debug ipcz_tests # Hope they all pass! ./ipcz_tests
ipcz may be statically linked into a project, or it may be consumed as a shared library. A shared library can be built with the ipcz_shared target.
The library is meant to be consumed exclusively through the C ABI defined in include/ipcz/ipcz.h. Applications populate an IpczAPI structure by calling IpczGetAPI(), the library's only exported symbol. From there they can create and connect nodes and establish portals for higher-level communication.
Applications must provide each node with an implementation of the IpczDriver function table to perform a variety of straightforward, platform- and environment-specific tasks such as establishing a basic I/O transport, generating random numbers, and allocating shared memory regions. See reference drivers for examples.
This directory in the Chromium tree is the source of truth for ipcz. It is not a mirror of an external repository, so there is no separate maintenance of local modifications or other versioning considerations.
The decision to place ipcz sources in //third_party/ipcz was made in light of some unique characteristics:
No dependencies on //base or other Chromium directories are allowed, with the exception of a very small number of carefully chosen APIs allowed when integrating with Chromium builds.
The library is structured and maintained to be useful as a standalone dependency, without needing any other contents of the Chromium tree or its large set of dependencies.
Certain style and dependency violations are made in service of the above two points; for example, ipcz depends on parts of Abseil disallowed in the rest of upstream Chromium, and ipcz internally uses relative include paths rather than paths rooted in Chromium's top-level directory.
Some extensive coverage of ipcz design details can be found here.