This document aims to explain the latest implementation details of OffscreenCanvas.commit(). Some parts (e.g, compositor frame submission mechanism, image resources) may remain unchanged in the future; some (3-way mojo communication) might be dropped and picked up in a different format later; some (e.g., frame dispatching rate) will become irrelevant in a future date, as a refactoring is done by fserb@ due to specification change.
The General Mojo IPC Flow
The following diagram illustrates the essential mojo calls for executing OffscreenCanvas.commit().
OffscreenCanvas.commit() involves communication between three parties--the main thread and the worker thread in the renderer process, plus the UI thread in the browser process. There are some more complex inter-process/thread communication in the implementation of display compositor in Viz, beyond what’s being drawn in the browser side in this diagram.
Placeholder Canvas Initiating the Mojo Call
In the beginning, we have an HTMLCanvasElement in the main thread (called “placeholder canvas”). We also have an EmbeddedFrameSinkProviderImpl in the browser (one instance per renderer process), which is the implementation of the mojo interface EmbeddedFrameSinkProvider’s receiving end; it is set up during the initialization of the render process, as one of the many initial mojo interfaces (see RenderProcessHostImpl::RegisterMojoInterfaces()).
When users perform transferControlToOffscreen() on the canvas (see HTMLCanvasElementModule::transferControlToOffscreen()), this HTMLCanvasElement will create a reference to a SurfaceLayerBridge instance. The constructor of SurfaceLayerBridge generates a FrameSinkId that is unique in the current renderer process (see Platform::GenerateFrameSinkId()), extracts the FrameSinkId of the layer tree view under which the placeholder canvas is attached, and then sends these IDs in a mojo call RegisterEmbeddedFrameSink() to the EmbeddedFrameSinkProviderImpl.
Upon receiving the mojo call RegisterEmbeddedFrameSink(), the global instance of EmbeddedFrameSinkProviderImpl will instantiate an EmbeddedFrameSinkImpl, and saves this instance in the frame_sink_map_, identified by the FrameSinkId. In the meantime, the EmbeddedFrameSinkImpl gains access to its client--the SurfaceLayerBridge instance--and can send mojo calls to it in the future.
On the blink side, a SolidColorLayer with transparent background is created inside the SurfaceLayerBridge object as a placeholder layer; this layer is registered to the graphics layer set, just like any other content layer. In a separate process of layer compositing, the graphics layer updater will identify a canvas with SurfaceLayerBridge to be a special case that requires a separate platform layer (see CompositedLayerCompositedLayerMapping:: UpdateGraphicsLayerConfiguration()).
Eventually, an OffscreenCanvas object is created, which contains the FrameSinkId, the dimension and DOMNodeId of the placeholder canvas. From users’ point of view, this is the time when they see an OffscreenCanvas JS object getting created.
Placeholder Canvas Transferring the OffscreenCanvas to Worker
Users can then perform the following script to post an OffscreenCanvas from main thread to worker:
worker.postMessage(offscreenCanvas, [offscreenCanvas])
Then they can extract this OffscreenCanvas in the worker script:
self.onmessage = function(e) { var transferredOffscreenCanvas = e.data; }
This message sending and receiving to transfer a JavaScript object is implemented by the V8ScriptValueSerializer and V8ScriptValueDeserializer. To post an OffscreenCanvas, the serializer will write an OffscreenCanvasTransferTag, followed by 5 different uint32_t values--the width, the height, the DOMNodeId of the placeholder canvas, the ClientId and the SinkId. Note that the ClientId and the SinkId are two components of a FrameSinkId. To receive an OffscreenCanvas, the deserializer will identify the transferred DOMObject as an OffscreenCanvas based on the OffscreenCanvasTransferTag and read the 5 uint32_t values in the same order of writing, before constructing a new OffscreenCanvas object on the worker thread.
As a result, we will have an OffscreenCanvas object on worker, with the information of placeholder canvas’s dimension, DOMNodeId as well as the FrameSinkId.
OffscreenCanvas calling commit() for the first time
The OffscreenCanvas on worker can create either a “2d” or an “WebGL” rendering context just like a usual canvas. Either of this offscreen rendering context has an API commit(). In the current specification and implementation, it returns a Promise when an image frame has been successfully dispatched; this Promise-based design is going to be changed soon in the future.
When users call commit() on the rendering context, a snapshot of the OffscreenCanvas will be taken. The snapshot is represented by a StaticBitmapImage object, which will be centrally managed in the OffscreenCanvasFrameDispatcher. In the current design, the image snapshot will not be immediately dispatched. Instead, it will be saved in the current_frame_ instance member of the OffscreenCanvas object and wait for an appropriate moment to be dispatched.
Meanwhile, OffscreenCanvas will instantiate an OffscreenCanvasFrameDispatcher. During the construction of an OffscreenCanvasFrameDispatcher, a current LocalSurfaceId will be generated (Note: a LocalSurfaceId and a FrameSinkId together form a unique SurfaceId, which identifies a Surface in browser). Then it sends a mojo call CreateCompositorFrameSink() to the always running EmbeddedFrameSinkProviderImpl in the browser; this mojo call carries the FrameSinkId that OffscreenCanvas got from the placeholder canvas during the control transfer, as well as a CompositorFrameSink pointer sink_ that’s waiting to be set by the display compositor in the future. On the receiving end, EmbeddedFrameSinkProviderImpl will use the FrameSinkId to find from its frame_sink_map_ the corresponding EmbeddedFrameSinkImpl which links to this OffscreenCanvas’s placeholder canvas, and then pass along the request of creating CompositorFrameSink to the EmbeddedFrameSinkImpl. EmbeddedFrameSinkImpl will forward this request to the HostFrameSinkManager, which has access to the privileged interface FrameSinkManager that has the authority to control the creation and destruction of CompositorFrameSinks in the display compositor. In addition, it will also register the parent FrameSinkId and FrameSinkId as a parent-child hierarchy relationship to the HostFrameSinkManager. A lot of call forwarding has been carried out in the display compositor side; but for OffscreenCanvas alone, we can treat the behind-scene implementation as a black box for easier understanding. As a result of this mojo call CreateCompositorFrameSink(), the created CompositorFrameSink will now have a two-way direct communication channel with the OffscreenCanvasFrameDispatcher.
Upon having the pointer sink_ to directly communicate with the CompositorFrameSink, OffscreenCanvasFrameDispatcher will send another mojo call SetNeedsBeginFrame(true) to it, indicating its interest to receive BeginFrame signals from the display compositor. BeginFrame signals are similar to the vSync signals and could be used as a way to sync the frame dispatching rhythm to an appropriate rate.
When it is the appropriate time to dispatch the saved image, OffscreenCanvasFrameDispatcher will prepare a CompositorFrame according to the image type and acceleration mode (see OffscreenCanvasFrameDispatcher:: DispatchFrame()). Then it sends the CompositorFrame as well as the current LocalSurfaceId to the CompositorFrameSink via a mojo call SubmitCompositorFrame(). The receiving end, the CompositorFrameSinkImpl, will handle this CompositorFrame and queues it to the corresponding Surface.
Because this is the first time commit() is called, the display compositor does not have the Surface for the given SurfaceId (a combination of FrameSinkId of the CompositorFrameSink and the LocalSurfaceId) yet. The display compositor will then create a new Surface (see CompositorFrameSinkSupport::MaybeSubmitCompositorFrame()). Once the Surface is created, there is going to be a few more chain calls in the display compositor, to inform different observers about this new creation. Afterwards, the CompositorFrame that we sends in this case will be queued to the newly created Surface (see Surface::QueueFrame()), which will result in a few more chain calls to inform different parties/observers. Ultimately, EmbeddedFrameSinkImpl::OnFirstSurfaceActivation() will be invoked, sending another mojo call OnFirstSurfaceActivation() to the SurfaceLayerBridge. Then the SurfaceLayerBridge will replace the SolidColorLayer with a real SurfaceLayer with the appropriate SurfaceId information; it will then trigger a compositing update in the placeholder canvas.
OffscreenCanvas Resizing
OffscreenCanvas resizing uses the same mojo call mechanism as the first OffscreenCanvas.commit(). Users initiate the resizing by setting OffscreenCanvas.width (or OffscreenCanvas.height) to a value different from its original one. OffscreenCanvasFrameDispatcher will save these values and set change_size_for_next_commit_ to be “true”; at this point of time, no new mojo calls are performed.
If a new image frame is dispatched, OffscreenCanvasFrameDispatcher will detect that change_size_for_next_commit_ is true; in this case, it will generate a new LocalSurfaceId. (Reminder: SurfaceId = FrameSinkId + LocalSurfaceId). This is the intended design of Surfaces--a new Surface is required for a rectangle with a new dimension but we want to maintain the attachment with the same CompositorFrameSink (which is identified by the FrameSinkId); therefore during a resizing, we create a new SurfaceId by using the same FrameSinkId and a new LocalSurfaceId. This new LocalSurfaceId will be sent in the next SubmitCompositorFrame() mojo call, together with the newly prepared CompositorFrame. The receiving CompositorFrameSink will detect a change of LocalSurfaceId and re-create a new Surface.
In addition, the intrinsic size of the placeholder canvas will be also updated via OffscreenCanvasFrameDispatcher::PostImageToPlaceholderIfNotBlocked(), which is cross-thread propagation that sends latest image of OffscreenCanvas to its placeholder canvas.
The Image Resources in OffscreenCanvas.Commit()
Extracting the Image Resources from Rendering Contexts
The image resources of OffscreenCanvas’s 2d rendering context are centrally managed by a CanvasResourceProvider instance. CanvasResourceProvider is an abstract class that encapsulates a drawable graphics resources and its implementations and could be managing different specific resource types, ranging from Gpu textures, Gpu Memory Buffers, to shared bitmap in RAM. When commit() is invoked by the user, an image snapshot would be taken from the SkSurface managed by the CanvasResourceProvider; the SkSurface can be on Gpu context (see SkSurface::MakeRenderTarget()) or on RAM memory (see SkSurface::MakeRaster()). The image snapshot would be saved as a StaticBitmapImage instance.
The image resources of OffscreenCanvas’s WebGL rendering context are centrally managed by its DrawingBuffer instance. There are two ways of extracting the current image from WebGL rendering context and this is determined by the value of the WebGL attribute preserveDrawingBuffer. If preserveDrawingBuffer is true, the texture on the WebGL back buffer is copied into the backing TextureId of a temporary CanvasResourceProvider instance before a snapshot is taken from there (see WebGLRenderingContextBase::GetImage()). If preserveDrawingBuffer is false, the texture on the back_color_buffer_ is transferred and returned, while the original back_color_buffer_ member is reset, i.e., put back in the recycled color buffer queue (see DrawingBuffer::TransferToStaticBitmapImage()), thus making any second attempt to retrieve the WebGL image impossible (hence the meaning of “not preserving”).
In the case of a false preserveDrawingBuffer, the returned ColorBuffer contains the actual GPU memory buffer. Its resource release is carefully handled by a release callback. This release callback is saved in the DrawingBuffer instance and is swapped only when a new commit() is coming or the DrawingBuffer is destructed; when the swapping happens, the previously saved callback will be run, releasing the ColorBuffer that holds up the GPU memory resource. This release mechanism is not consistent with the rest of canvas resource; ideally, everything should be kept under CanvasResourceProvider; this might be a future to-do item.
Preparing the CompositorFrame
The image extracted from the previous step is now encapsulated as a StaticBitmapImage instance, which is saved in the OffscreenCanvas. When it is the right time to dispatch the image, OffscreenCanvasFrameDispatcher will prepare a CompositorFrame based on this StaticBitmapImage instance and sends it off to the CompositorFrameSink (see OffscreenCanvasFrameDispatcher::DispatchFrame()).
OffscreenCanvasFrameDispatcher has delegated all its resource management tasks to its OffscreenCanvasResourceProvider instance (which will be merged to CanvasResourceProvider soon by junov@). This resource provider has two different methods in converting a StaticBitmapImage into a viz::TransferableResource, which is an essential component of a CompositorFrame. There are four scenarios here but there are only two ways of handling.
Syncing the OffscreenCanvas Animation
Placeholder Canvas Accessing the Image at all times
What happens when the users perform canvas.toBlob() on the placeholder canvas? What happens when the users want to right click the placeholder canvas and copy the image? Because we directly “commit” the CompositorFrame from worker thread to the display compositor, the placeholder canvas on the main thread actually does not have access to the image resource. The solution we adopt here is to post the image from worker thread to the main thread, whenever there is a new image committed, subject to the message receiving capacity of the main thread.
OffscreenCanvasFrameDispatcher::PostImageToPlaceholderIfNotBlocked() is the function to send an image from OffscreenCanvas to placeholder canvas, using a cross-thread task. The cross-thread task sends the following:
The image;
The placeholder canvas’s DOMNodeId, useful to identify the HTMLCanvasElement instance;
The pointers of OffscreenCanvasFrameDispatcher and the current worker thread’s task runner, both useful in posting a “return call” from the placeholder canvas to the OffscreenCanvas instance.
We have a mechanism to prevent over-sending messages from OffscreenCanvas to the main thread, by keeping track the number of unreclaimed frames on the OffscreenCanvas. When this number exceeds a threshold (which is currently set to 3), OffscreenCanvasFrameDispatcher will save the unposted image, until the first “reclaim resource” call from placeholder canvas is received to release the previous image resource. The following sequence diagram illustrates a few different scenarios that may possibly arise in this main-worker communication.
The actual release of an image resource is only executed when the ReclaimResource() is invoked in both of these places:
A mojo call from the display compositor DidReceiveCompositorFrameAck(), indicating that the particular CompositorFrame using this image resource has been processed;
A message sent from the placeholder canvas indicating that it no longer needs this image resource (either because it now has another new image or the canvas is destructed).
The second invocation is illustrated as the “Release ImageA” message in the above sequence diagram.
Dispatching Frame on Appropriate Time
TODO(fserb@): Edit this section of documentation when commit() is finished.
The current implementation assumes that there are two time points when we can do the actual dispatching:
At the end of the current JavaScript Task;
On the beginning of a BeginFrame signal.
The latest plans, carried out by fserb@, would be to revamp the commit() design as follows:
Keep the
commit(), but drop its returnedPromise. Make thiscommit()a blocking API, instead of an async API.Implement
requestAnimationFrame()inWorkerGlobalScope.