As part of supporting a Chromium runtime for chromecast (as defined below), the CmaBackendProxy class and related code contained in the chromecast/media/cma/backend/proxy/ directory was added. This infrastructure exists to proxy audio data across the CastRuntimeAudioChannel gRPC Service without impacting local playback of video or audio media, and requires significant complexity in order to do so. This document provides a high-level overview of the CmaBackendProxy implementation and the pipeline into which it calls. It is intended to be used as a reference for understanding this pipeline, to simplify both the code review and the onboarding process.
The chromecast pipeline, as exists today in Chromium's chromecast/ directory, is rather complex, so only the high-level will be discussed here. When playing out media, an implementation of the MediaPipelineBackend API will be created, and this will be wrapped by a platform-specific (e.g. Android, Fuchsia, Desktop, etc…) CmaBackend as exists today in the Chromium repo. The CmaBackend will create an AudioDecoder and VideoDecoder instance as needed, which is responsible for operations such as playback control, queueing up media playout, and decrypting DRM (if needed), then passing these commands to the MediaPipelineBackend.
gRPC is a modern open source high performance RPC framework that can run in any environment. It can efficiently connect services in and across data centers with pluggable support for load balancing, tracing, health checking and authentication. gRPC was chosen as the RPC framework to use for the Chromium runtime to simplify the upgrade story for when both sides of the RPC channel can upgrade independently.
A runtime to be used by hardware devices to play out audio and video data as received from a Cast session. It is expected to run on an embedded device.
At a high level, the CmaBackendProxy (“proxy backend”) sits on top of another CmaBackend, which for the purposes of this document will be referred to as the delegated backend. The proxy backend acts to send all audio and video data to the delegated backend on which it sits, while also sending the same audio to the CastRuntimeAudioChannel gRPC Channel used to proxy data, along with synchronization data to keep playback between the two in-line. The architecture used for this pipeline can be seen as follows:
The pipeline can be roughly divided into two parts:
CmaBackendProxy gets audio and video data it receives to the platform-specific MediaPipelineBackend, resulting in playout of the local device's speakers .Both can be summarized in this diagram, with further details to follow:
The CmaBackendProxy, by virtue of being a BackendProxy, can create an AudioDecoder and a VideoDecoder. VideoDecoder creation and all video processing is delegated to the underlying delegated backend with no changes, so it will not be discussed further. The AudioDecoder is more interesting - there are 3 CmaBackend::AudioDecoder types relevant in this architecture through which audio data will pass - two proxy specific and the delegated backend's decoder. In processing order, they are:
MultizoneAudioDecoderProxy: This class is the starting-point for sending data over the CastRuntimeAudioChannel gRPC, using the pipeline described in the following section.MediaPipelineBufferExtension: This class performs no processing itself, but instead just acts to store the data from PushBuffer calls locally, so that an extra few seconds of data may be queued up in addition to what the local decoder stores.CmaBackend::AudioDecoder for this specific platform, as described above. This is owned by the Delegated backend itself.All audio data passes through these three layers so that it may be played out locally. For local playback as described here, this functionality is for the most part uninteresting - by design, the end user should notice no difference in audio playout. At an architecture level, the only noticeable difference for an embedder is that extra data is queued locally in the MediaPipelineBufferExtension.
As an abstraction on top of the above, the AudioDecoderPipelineNode was introduced as a parent for MultizoneAudioDecoderProxy and MediaPipelineBufferExtension, though its existence can for the most part be ignored. All functions in this base class act as pass-through methods, directly calling into the lower layer's method of the same name. Meaning that this class functions purely for software-engineering reasons, as a way to reduce code complexity of the child classes.
The MultizoneAudioDecoderProxy, as mentioned above, functions to proxy data across the CastRuntimeAudioChannel gRPC channel. The pipeline used to do uses the following classes, in order:
MultizoneAudioDecoderProxyImpl: As mentioned above, all CmaBackend::AudioDecoder methods call into here first, as well as methods corresponding to the public calls on CmaBackend. This class mostly acts as a pass-through, immediately calling into the below for all above method calls, although the BufferIdManager owned by this class adds a bit of complexity, in that it assigns Buffer Ids to PushBuffer calls, to be used for audio timing synchronization between both sides of the CastRuntimeAudioChannel gRPC channel.ProxyCallTranslator: As the name suggests, this class exists to convert between the media-pipeline understood types, as called into the above layer, with those supported by CastRuntimeAudioChannel gRPC.To this end, PushBufferQueue helps to turn the “Push” model used by the CmaBackend::AudioDecoder‘s PushBuffer() method into a “Pull” model that better aligns with the gRPC async model. The PendingPushBufferHelper works with the above, to support returning a kBufferPending response as required by the PushBuffer() method’s contract. In addition, this class is also responsible for handling threading assumptions made by the above and below classes, such as how data returned by the below layer must be processed on the correct thread in the above layer.CastRuntimeAudioChannelBroker: Implementations of this abstract class take the CastRuntimeAudioChannel gRPC types supplied by the above, then use them to make calls against the gRPC Service.In order to support varied user scenarios, a number of build flags have been introduced to customize the functionality of this pipeline:
This flag enables or disables support for sending data across the CastRuntimeAudioChannel. When enabled, the CmaBackendFactoryImpl class will wrap the implementation-specific CmaBackend with a CmaBackendProxy instance, as described above. This flag defaults to False.