| {{+bindTo:partials.standard_nacl_article}} |
| |
| <b><font color="#cc0000"> |
| NOTE: |
| Deprecation of the technologies described here has been announced |
| for platforms other than ChromeOS.<br/> |
| Please visit our |
| <a href="/native-client/migration">migration guide</a> |
| for details. |
| </font></b> |
| <hr/><section id="audio"> |
| <h1 id="audio">Audio</h1> |
| <div class="contents local" id="contents" style="display: none"> |
| <ul class="small-gap"> |
| <li><a class="reference internal" href="#reference-information" id="id1">Reference information</a></li> |
| <li><a class="reference internal" href="#about-the-pepper-audio-api" id="id2">About the Pepper audio API</a></li> |
| <li><a class="reference internal" href="#digital-audio-concepts" id="id3">Digital audio concepts</a></li> |
| <li><a class="reference internal" href="#setting-up-the-module" id="id4">Setting up the module</a></li> |
| <li><p class="first"><a class="reference internal" href="#creating-an-audio-configuration-resource" id="id5">Creating an audio configuration resource</a></p> |
| <ul class="small-gap"> |
| <li><a class="reference internal" href="#resources" id="id6">Resources</a></li> |
| <li><a class="reference internal" href="#sample-frame-count" id="id7">Sample frame count</a></li> |
| <li><a class="reference internal" href="#supported-audio-configurations" id="id8">Supported audio configurations</a></li> |
| </ul> |
| </li> |
| <li><a class="reference internal" href="#creating-an-audio-resource" id="id9">Creating an audio resource</a></li> |
| <li><p class="first"><a class="reference internal" href="#implementing-a-callback-function" id="id10">Implementing a callback function</a></p> |
| <ul class="small-gap"> |
| <li><a class="reference internal" href="#application-threads-and-real-time-requirements" id="id11">Application threads and real-time requirements</a></li> |
| </ul> |
| </li> |
| <li><a class="reference internal" href="#starting-and-stopping-playback" id="id12">Starting and stopping playback</a></li> |
| </ul> |
| |
| </div><p>This section describes how to use the Pepper audio API to play an audio |
| stream. The Pepper audio API provides a low-level means of playing a stream of |
| audio samples generated by a Native Client module. The API generally works as |
| follows: A Native Client module creates an audio resource that represents an |
| audio stream, and tells the browser to start or stop playing the audio |
| resource. The browser calls a function in the Native Client module to fill a |
| buffer with audio samples every time it needs data to play from the audio |
| stream.</p> |
| <p>The code examples in this section describe a simple Native Client module that |
| generates audio samples using a sine wave with a frequency of 440 Hz. The module |
| starts playing the audio samples as soon as it is loaded into the browser. For a |
| slightly more sophisticated example, see the <code>audio</code> example (source code in |
| the SDK directory <code>examples/api/audio</code>), which lets users specify a frequency |
| for the sine wave and click buttons to start and stop audio playback.</p> |
| <h2 id="reference-information">Reference information</h2> |
| <p>For reference information related to the Pepper audio API, see the following |
| documentation:</p> |
| <ul class="small-gap"> |
| <li><a class="reference external" href="/native-client/pepper_stable/cpp/classpp_1_1_audio_config">pp::AudioConfig class</a></li> |
| <li><a class="reference external" href="/native-client/pepper_stable/cpp/classpp_1_1_audio">pp::Audio class</a></li> |
| <li><a class="reference external" href="/native-client/pepper_cpp/audio__config_8h">audio_config.h</a></li> |
| <li><a class="reference external" href="/native-client/pepper_stable/cpp/audio_8h">audio.h</a></li> |
| <li><a class="reference external" href="/native-client/pepper_stable/c/group___enums#gaee750c350655f2fb0fe04c04029e0ff8">PP_AudioSampleRate</a></li> |
| </ul> |
| <h2 id="about-the-pepper-audio-api">About the Pepper audio API</h2> |
| <p>The Pepper audio API lets Native Client modules play audio streams in a |
| browser. To play an audio stream, a module generates audio samples and writes |
| them into a buffer. The browser reads the audio samples from the buffer and |
| plays them using an audio device on the client computer.</p> |
| <img alt="/native-client/images/pepper-audio-buffer.png" src="/native-client/images/pepper-audio-buffer.png" /> |
| <p>This mechanism is simple but low-level. If you want to play plain sound files in |
| a web application, you may want to consider higher-level alternatives such as |
| using the HTML <code><audio></code> tag, JavaScript, or the new <a class="reference external" href="http://chromium.googlecode.com/svn/trunk/samples/audio/index.html">Web Audio API</a>.</p> |
| <p>The Pepper audio API is a good option for playing audio data if you want to do |
| audio processing in your web application. You might use the audio API, for |
| example, if you want to apply audio effects to sounds, synthesize your own |
| sounds, or do any other type of CPU-intensive processing of audio |
| samples. Another likely use case is gaming applications: you might use a gaming |
| library to process audio data, and then simply use the audio API to output the |
| processed data.</p> |
| <p>The Pepper audio API is straightforward to use:</p> |
| <ol class="arabic simple"> |
| <li>Your module creates an audio configuration resource and an audio resource.</li> |
| <li>Your module implements a callback function that fills an audio buffer with |
| data.</li> |
| <li>Your module invokes the StartPlayback and StopPlayback methods of the audio |
| resource (e.g., when certain events occur).</li> |
| <li>The browser invokes your callback function whenever it needs audio data to |
| play. Your callback function can generate the audio data in a number of |
| ways—e.g., it can generate new data, or it can copy pre-mixed data into the |
| audio buffer.</li> |
| </ol> |
| <p>This basic interaction is illustrated below, and described in detail in the |
| sections that follow.</p> |
| <img alt="/native-client/images/pepper-audio-api.png" src="/native-client/images/pepper-audio-api.png" /> |
| <h2 id="digital-audio-concepts">Digital audio concepts</h2> |
| <p>Before you use the Pepper audio API, it’s helpful to understand a few concepts |
| that are fundamental to how digital audio is recorded and played back:</p> |
| <dl class="docutils"> |
| <dt>sample rate</dt> |
| <dd>the number of times an input sound source is sampled per second; |
| correspondingly, the number of samples that are played back per second</dd> |
| <dt>bit depth</dt> |
| <dd>the number of bits used to represent a sample</dd> |
| <dt>channels</dt> |
| <dd>the number of input sources recorded in each sampling interval; |
| correspondingly, the number of outputs that are played back simultaneously |
| (typically using different speakers)</dd> |
| </dl> |
| <p>The higher the sample rate and bit depth used to record a sound wave, the more |
| accurately the sound wave can be reproduced, since it will have been sampled |
| more frequently and stored using a higher level of quantization. Common sampling |
| rates include 44,100 Hz (44,100 samples/second, the sample rate used on CDs), |
| and 48,000 Hz (the sample rate used on DVDs and Digital Audio Tapes). A common |
| bit depth is 16 bits per sample, and a common number of channels is 2 (left and |
| right channels for stereo sound).</p> |
| <p id="pepper-audio-configurations">The Pepper audio API currently lets Native Client modules play audio streams |
| with the following configurations:</p> |
| <ul class="small-gap"> |
| <li><strong>sample rate</strong>: 44,100 Hz or 48,000 Hz</li> |
| <li><strong>bit depth</strong>: 16</li> |
| <li><strong>channels</strong>: 2 (stereo)</li> |
| </ul> |
| <h2 id="setting-up-the-module">Setting up the module</h2> |
| <p>The code examples below describe a simple Native Client module that generates |
| audio samples using a sine wave with a frequency of 440 Hz. The module starts |
| playing the audio samples as soon as it is loaded into the browser.</p> |
| <p>The Native Client module is set up by implementing subclasses of the |
| <code>pp::Module</code> and <code>pp::Instance</code> classes, as normal.</p> |
| <pre class="prettyprint"> |
| class SineSynthInstance : public pp::Instance { |
| public: |
| explicit SineSynthInstance(PP_Instance instance); |
| virtual ~SineSynthInstance() {} |
| |
| // Called by the browser once the NaCl module is loaded and ready to |
| // initialize. Creates a Pepper audio context and initializes it. Returns |
| // true on success. Returning false causes the NaCl module to be deleted |
| // and no other functions to be called. |
| virtual bool Init(uint32_t argc, const char* argn[], const char* argv[]); |
| |
| private: |
| // Function called by the browser when it needs more audio samples. |
| static void SineWaveCallback(void* samples, |
| uint32_t buffer_size, |
| void* data); |
| |
| // Audio resource. |
| pp::Audio audio_; |
| |
| ... |
| |
| }; |
| |
| class SineSynthModule : public pp::Module { |
| public: |
| SineSynthModule() : pp::Module() {} |
| ~SineSynthModule() {} |
| |
| // Create and return a SineSynthInstance object. |
| virtual pp::Instance* CreateInstance(PP_Instance instance) { |
| return new SineSynthInstance(instance); |
| } |
| }; |
| </pre> |
| <h2 id="creating-an-audio-configuration-resource">Creating an audio configuration resource</h2> |
| <h3 id="resources">Resources</h3> |
| <p>Before the module can play an audio stream, it must create two resources: an |
| audio configuration resource and an audio resource. Resources are handles to |
| objects that the browser provides to module instances. An audio resource is an |
| object that represents the state of an audio stream, including whether the |
| stream is paused or being played back, and which callback function to invoke |
| when the samples in the stream’s buffer run out. An audio configuration resource |
| is an object that stores configuration data for an audio resource, including the |
| sampling frequency of the audio samples, and the number of samples that the |
| callback function must provide when the browser invokes it.</p> |
| <h3 id="sample-frame-count">Sample frame count</h3> |
| <p>Prior to creating an audio configuration resource, the module should call |
| <code>RecommendSampleFrameCount</code> to obtain a <em>sample frame count</em> from the |
| browser. The sample frame count is the number of samples that the callback |
| function must provide per channel each time the browser invokes the callback |
| function. For example, if the sample frame count is 4096 for a stereo audio |
| stream, the callback function must provide a 8192 samples (4096 for the left |
| channel and 4096 for the right channel).</p> |
| <p>The module can request a specific sample frame count, but the browser may return |
| a different sample frame count depending on the capabilities of the client |
| device. At present, <code>RecommendSampleFrameCount</code> simply bound-checks the |
| requested sample frame count (see <code>include/ppapi/c/ppb_audio_config.h</code> for the |
| minimum and maximum sample frame counts, currently 64 and 32768). In the future, |
| <code>RecommendSampleFrameCount</code> may perform a more sophisticated calculation, |
| particularly if there is an intrinsic buffer size for the client device.</p> |
| <p>Selecting a sample frame count for an audio stream involves a tradeoff between |
| latency and CPU usage. If you want your module to have short audio latency so |
| that it can rapidly change what’s playing in the audio stream, you should |
| request a small sample frame count. That could be useful in gaming applications, |
| for example, where sounds have to change frequently in response to game |
| action. However, a small sample frame count results in higher CPU usage, since |
| the browser must invoke the callback function frequently to refill the audio |
| buffer. Conversely, a large sample frame count results in higher latency but |
| lower CPU usage. You should request a large sample frame count if your module |
| will play long, uninterrupted audio segments.</p> |
| <h3 id="supported-audio-configurations">Supported audio configurations</h3> |
| <p>After the module obtains a sample frame count, it can create an audio |
| configuration resource. Currently the Pepper audio API supports audio streams |
| with the configuration settings shown <a class="reference internal" href="#pepper-audio-configurations"><em>above</em></a>. |
| C++ modules can create a configuration resource by instantiating a |
| <code>pp::AudioConfig</code> object. Check <code>audio_config.h</code> for the latest |
| configurations that are supported.</p> |
| <pre class="prettyprint"> |
| bool SineSynthInstance::Init(uint32_t argc, |
| const char* argn[], |
| const char* argv[]) { |
| |
| // Ask the browser/device for an appropriate sample frame count size. |
| sample_frame_count_ = |
| pp::AudioConfig::RecommendSampleFrameCount(PP_AUDIOSAMPLERATE_44100, |
| kSampleFrameCount); |
| |
| // Create an audio configuration resource. |
| pp::AudioConfig audio_config = pp::AudioConfig(this, |
| PP_AUDIOSAMPLERATE_44100, |
| sample_frame_count_); |
| |
| // Create an audio resource. |
| audio_ = pp::Audio(this, |
| audio_config, |
| SineWaveCallback, |
| this); |
| |
| // Start playback when the module instance is initialized. |
| return audio_.StartPlayback(); |
| } |
| </pre> |
| <h2 id="creating-an-audio-resource">Creating an audio resource</h2> |
| <p>Once the module has created an audio configuration resource, it can create an |
| audio resource. To do so, it instantiates a <code>pp::Audio</code> object, passing in a |
| pointer to the module instance, the audio configuration resource, a callback |
| function, and a pointer to user data (data that is used in the callback |
| function). See the example above.</p> |
| <h2 id="implementing-a-callback-function">Implementing a callback function</h2> |
| <p>The browser calls the callback function associated with an audio resource every |
| time it needs more samples to play. The callback function can generate new |
| samples (e.g., by applying sound effects), or copy pre-mixed samples into the |
| audio buffer. The example below generates new samples by computing values of a |
| sine wave.</p> |
| <p>The last parameter passed to the callback function is generic user data that the |
| function can use in processing samples. In the example below, the user data is a |
| pointer to the module instance, which includes member variables |
| <code>sample_frame_count_</code> (the sample frame count obtained from the browser) and |
| <code>theta_</code> (the last angle that was used to compute a sine value in the previous |
| callback; this lets the function generate a smooth sine wave by starting at that |
| angle plus a small delta).</p> |
| <pre class="prettyprint"> |
| class SineSynthInstance : public pp::Instance { |
| public: |
| ... |
| |
| private: |
| static void SineWaveCallback(void* samples, |
| uint32_t buffer_size, |
| void* data) { |
| |
| // The user data in this example is a pointer to the module instance. |
| SineSynthInstance* sine_synth_instance = |
| reinterpret_cast<SineSynthInstance*>(data); |
| |
| // Delta by which to increase theta_ for each sample. |
| const double delta = kTwoPi * kFrequency / PP_AUDIOSAMPLERATE_44100; |
| // Amount by which to scale up the computed sine value. |
| const int16_t max_int16 = std::numeric_limits<int16_t>::max(); |
| |
| int16_t* buff = reinterpret_cast<int16_t*>(samples); |
| |
| // Make sure we can't write outside the buffer. |
| assert(buffer_size >= (sizeof(*buff) * kChannels * |
| sine_synth_instance->sample_frame_count_)); |
| |
| for (size_t sample_i = 0; |
| sample_i < sine_synth_instance->sample_frame_count_; |
| ++sample_i, sine_synth_instance->theta_ += delta) { |
| |
| // Keep theta_ from going beyond 2*Pi. |
| if (sine_synth_instance->theta_ > kTwoPi) { |
| sine_synth_instance->theta_ -= kTwoPi; |
| } |
| |
| // Compute the sine value for the current theta_, scale it up, |
| // and write it into the buffer once for each channel. |
| double sin_value(std::sin(sine_synth_instance->theta_)); |
| int16_t scaled_value = static_cast<int16_t>(sin_value * max_int16); |
| for (size_t channel = 0; channel < kChannels; ++channel) { |
| *buff++ = scaled_value; |
| } |
| } |
| } |
| |
| ... |
| }; |
| </pre> |
| <h3 id="application-threads-and-real-time-requirements">Application threads and real-time requirements</h3> |
| <p>The callback function runs in a background application thread. This allows audio |
| processing to continue even when the application is busy doing something |
| else. If the main application thread and the callback thread access the same |
| data, you may be tempted to use a lock to control access to that data. You |
| should avoid the use of locks in the callback thread, however, as attempting to |
| acquire a lock may cause the thread to get swapped out, resulting in audio |
| dropouts.</p> |
| <p>In general, you must program the callback thread carefully, as the Pepper audio |
| API is a very low level API that needs to meet hard real-time requirements. If |
| the callback thread spends too much time processing, it can easily miss the |
| real-time deadline, resulting in audio dropouts. One way the callback thread can |
| miss the deadline is by taking too much time doing computation. Another way the |
| callback thread can miss the deadline is by executing a function call that swaps |
| out the callback thread. Unfortunately, such function calls include just about |
| all C Run-Time (CRT) library calls and Pepper API calls. The callback thread |
| should therefore avoid calls to malloc, gettimeofday, mutex, condvars, critical |
| sections, and so forth; any such calls could attempt to take a lock and swap out |
| the callback thread, which would be disastrous for audio playback. Similarly, |
| the callback thread should avoid Pepper API calls. Audio dropouts due to thread |
| swapping can be very rare and very hard to track down and debug—it’s best to |
| avoid making system/Pepper calls in the first place. In short, the audio |
| (callback) thread should use “lock-free” techniques and avoid making CRT library |
| calls.</p> |
| <p>One other issue to be aware of is that the <code>StartPlayback</code> function (discussed |
| below) is an asynchronous RPC; i.e., it does not block. That means that the |
| callback function may not be called immediately after the call to |
| <code>StartPlayback</code>. If it’s important to synchronize the callback thread with |
| another thread so that the audio stream starts playing simultaneously with |
| another action in your application, you must handle such synchronization |
| manually.</p> |
| <h2 id="starting-and-stopping-playback">Starting and stopping playback</h2> |
| <p>To start and stop audio playback, the module simply reacts to JavaScript |
| messages.</p> |
| <pre class="prettyprint"> |
| const char* const kPlaySoundId = "playSound"; |
| const char* const kStopSoundId = "stopSound"; |
| |
| void SineSynthInstance::HandleMessage(const pp::Var& var_message) { |
| if (!var_message.is_string()) { |
| return; |
| } |
| std::string message = var_message.AsString(); |
| if (message == kPlaySoundId) { |
| audio_.StartPlayback(); |
| } else if (message == kStopSoundId) { |
| audio_.StopPlayback(); |
| } else if (...) { |
| ... |
| } |
| } |
| </pre> |
| </section> |
| |
| {{/partials.standard_nacl_article}} |