native_client_sdk/src/doc/devguide/coding/3D-graphics.rst

.. _devguide-coding-3D-graphics:

.. include:: /migration/deprecation.inc

###########
3D Graphics
###########

Native Client applications use the `OpenGL ES 2.0
<http://en.wikipedia.org/wiki/OpenGL_ES>`_ API for 3D rendering. This document
describes how to call the OpenGL ES 2.0 interface in a Native Client module and
how to build an efficient rendering loop. It also explains how to validate GPU
drivers and test for specific GPU capabilities, and provides tips to help ensure
your rendering code runs efficiently.

.. Note::
  :class: note

  **Note**: 3D drawing and OpenGL are complex topics. This document deals only
  with issues directly related to programming in the Native Client
  environment. To learn more about OpenGL ES 2.0 itself, see the `OpenGL ES 2.0
  Programming Guide <http://opengles-book.com/>`_.

Validating the client graphics platform
=======================================

Native Client is a software technology that lets you code an application once
and run it on multiple platforms without worrying about the implementation
details on every possible target platform. It's difficult to provide the same
support at the hardware level. Graphics hardware comes from many different
manufacturers and is controlled by drivers of varying quality. A particular GPU
driver may not support every OpenGL ES 2.0 feature, and some drivers are known
to have vulnerabilities that can be exploited.

Even if the GPU driver is safe to use, your program should perform a validation
check before you launch your application to ensure that the driver supports all
the features you need.

Vetting the driver in JavaScript
--------------------------------

At startup, the application should perform a few additional tests that can be
implemented in JavaScript on its hosting web page. The script that performs
these tests should be included before the module's ``embed`` tag, and ideally
the ``embed`` tag should appear on the hosting page only if these tests succeed.

The first thing to check is whether you can create a graphics context. If you
can, use the context to confirm the existence of any required OpenGL ES 2.0
extensions.  You may want to refer to the `extension registry
<http://www.khronos.org/registry/webgl/extensions/>`_ and include `vendor
prefixes <https://developer.mozilla.org/en-US/docs/WebGL/Using_Extensions>`_
when checking for extensions.

Vetting the driver in Native Client
-----------------------------------

Create a context
^^^^^^^^^^^^^^^^

Once you've passed the JavaScript validation tests, it's safe to add a Native
Client embed tag to the hosting web page and load the module. As part of the
module initialization code, you must create a graphics context for the app by
either creating a C++ ``Graphics3D`` object or calling ``PPB_Graphics3D`` API
function ``Create``. Don't assume this will always succeed; you still might have
problems creating the context. If you are in development mode and can't create
the context, try creating a simpler version to see if you're asking for an
unsupported feature or exceeding a driver resource limit. Your production code
should always check that the context was created and fail gracefully if that's
not the case.

Check for extensions and capabilities
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Not every GPU supports every extension or has the same amount of texture units,
vertex attributes, etc. On startup, call ``glGetString(GL_EXTENSIONS)`` and
check for the extensions and the features you need. For example:

* If you are using non power-of-2 texture with mipmaps, make sure
  ``GL_OES_texture_npot`` exists.

* If you are using floating point textures, make sure ``GL_OES_texture_float``
  exists.

* If you are using DXT1, DXT3, or DXT5 textures, make sure the corresponding
  extensions ``GL_ANGLE_texture_compression_dxt1``,
  ``GL_ANGLE_texture_compression_dxt3``, and
  ``GL_ANGLE_texture_compression_dxt5`` exist.

* If you are using the functions ``glDrawArraysInstancedANGLE``,
  ``glDrawElementsInstancedANGLE``, ``glVertexAttribDivisorANGLE``, or the PPAPI
  interface ``PPB_OpenGLES2InstancedArrays``, make sure the corresponding
  extension ``GL_ANGLE_instanced_arrays`` exists.

* If you are using the function ``glRenderbufferStorageMultisampleEXT``, or the
  PPAPI interface ``PPB_OpenGLES2FramebufferMultisample``, make sure the
  corresponding extension ``GL_CHROMIUM_framebuffer_multisample`` exists.

* If you are using the functions ``glGenQueriesEXT``, ``glDeleteQueriesEXT``,
  ``glIsQueryEXT``, ``glBeginQueryEXT``, ``glEndQueryEXT``, ``glGetQueryivEXT``,
  ``glGetQueryObjectuivEXT``, or the PPAPI interface ``PPB_OpenGLES2Query``,
  make sure the corresponding extension ``GL_EXT_occlusion_query_boolean``
  exists.

* If you are using the functions ``glMapBufferSubDataCHROMIUM``,
  ``glUnmapBufferSubDataCHROMIUM``, ``glMapTexSubImage2DCHROMIUM``,
  ``glUnmapTexSubImage2DCHROMIUM``, or the PPAPI interface
  ``PPB_OpenGLES2ChromiumMapSub``, make sure the corresponding extension
  ``GL_CHROMIUM_map_sub`` exists.

Check for system capabilites with ``glGetIntegerv`` and adjust shader programs
as well as texture and vertex data accordingly:

* If you are using textures in vertex shaders, make sure
  ``glGetIntegerv(GL_MAX_VERTEX_TEXTURE_IMAGE_UNITS, ...)`` and
  ``glGetIntegerv(GL_MAX_TEXTURE_SIZE, ...)`` return values greater than 0.

* If you are using more than 8 textures in a single shader, make sure
  ``glGetIntegerv(GL_MAX_TEXTURE_IMAGE_UNITS, ...)`` returns a value greater
  than or equal to the number of simultaneous textures you need.

Vetting the driver in the Chrome Web Store
------------------------------------------

If you choose to place your application in the `Chrome Web Store </webstore>`_,
its Web Store `manifest file </extensions/manifest>`_ can include the ``webgl``
feature in the requirements parameter. It looks like this:

.. naclcode::

  "requirements": {
    "3D": {
      "features": ["webgl"]
    }
  }

While WebGL is technically a JavaScript API, specifying the ``webgl`` feature
also works for OpenGL ES 2.0 because both interfaces use the same driver.

This manifest item is not required, but if you include it, the Chrome Web Store
will prevent a user from installing the application if the browser is running on
a machine that does not support OpenGL ES 2.0 or that is using a known
blacklisted GPU driver that could invite an attack.

If the Web Store determines that the user's driver is deficient, the app won't
appear on the store's tile display. However, it will appear in store search
results or if the user links to it directly, in which case the user could still
download it. But the manifest requirements will be checked when the user reaches
the install page, and if there is a problem, the browser will display the
message "This application is not supported on this computer. Installation has
been disabled."

The manifest-based check applies only to downloads directly from the Chrome Web
Store. It is not performed when an application is loaded via `inline
installation </webstore/inline_installation>`_.

What to do when there are problems
----------------------------------

Using the vetting procedure described above, you should be able to detect the
most common problems before your application runs. If there are problems, your
code should describe the issue as clearly as possible. That's easy if there is a
missing feature. Failure to create a graphics context is tougher to diagnose. At
the very least, you can suggest that the user try to update the driver.  You
might want to linke to the Chrome page that describes `how to do updates
<https://support.google.com/chrome/answer/1202946>`_.

If a user can't update the driver, or their problem persists, be sure to gather
information about their graphics environment. Ask for the contents of the Chrome
``about:gpu`` page.

Document unreliable drivers
---------------------------

It can be helpful to include information about known dubious drivers in your
user documentation. This might help identify if a rogue driver is the cause of a
problem. There are many sources of GPU driver blacklists. Two such lists can be
found at the `Chromium project
<http://src.chromium.org/viewvc/chrome/trunk/deps/gpu/software_rendering_list/software_rendering_list.json>`_
and `Khronos <http://www.khronos.org/webgl/wiki/BlacklistsAndWhitelists>`_. You
can use these lists to include information in your documentation that warns
users about dangerous drivers.

Test your defenses
------------------

You can test your driver validation code by running Chrome with the following
flags (all at once) and watching how your application responds:

* ``--disable-webgl``
* ``--disable-pepper-3d``
* ``--disable_multisampling``
* ``--disable-accelerated-compositing``
* ``--disable-accelerated-2d-canvas``

Calling OpenGL ES 2.0 commands
==============================

There are three ways to write OpenGL ES 2.0 calls in Native Client.

Use "pure" OpenGL ES 2.0 function calls
---------------------------------------

You can make OpenGL ES 2.0 calls through a Pepper extension library.  The SDK
example ``examples/api/graphics_3d`` works this way.  In the file
``graphics_3d.cc``, the key initialization steps are as follows:

* Add these includes at the top of the file:

  .. naclcode::

    #include <GLES2/gl2.h>
    #include "ppapi/lib/gl/gles2/gl2ext_ppapi.h"

* Define the function ``InitGL``. The exact specification of ``attrib_list``
  will be application specific.

  .. naclcode::

    bool InitGL(int32_t new_width, int32_t new_height) {
      if (!glInitializePPAPI(pp::Module::Get()->get_browser_interface())) {
        fprintf(stderr, "Unable to initialize GL PPAPI!\n");
        return false;
      }

      const int32_t attrib_list[] = {
        PP_GRAPHICS3DATTRIB_ALPHA_SIZE, 8,
        PP_GRAPHICS3DATTRIB_DEPTH_SIZE, 24,
        PP_GRAPHICS3DATTRIB_WIDTH, new_width,
        PP_GRAPHICS3DATTRIB_HEIGHT, new_height,
        PP_GRAPHICS3DATTRIB_NONE
      };

      context_ = pp::Graphics3D(this, attrib_list);
      if (!BindGraphics(context_)) {
        fprintf(stderr, "Unable to bind 3d context!\n");
        context_ = pp::Graphics3D();
        glSetCurrentContextPPAPI(0);
        return false;
      }

      glSetCurrentContextPPAPI(context_.pp_resource());
      return true;
    }

* Include logic in ``Instance::DidChangeView`` to call ``InitGL`` whenever
  necessary: upon application launch (when the graphics context is NULL) and
  whenever the module's View changes size.

Use Regal
---------

If you are porting an OpenGL ES 2.0 application, or are comfortable writing in
OpenGL ES 2.0, you should stick with the Pepper APIs or pure OpenGL ES 2.0 calls
described above. If you are porting an application that uses features not in
OpenGL ES 2.0, consider using Regal. Regal is an open source library that
supports many versions of OpenGL. Regal recently added support for Native
Client. Regal forwards most OpenGL calls directly to the underlying graphics
library, but it can also emulate other calls that are not included (when
hardware support exists). See `libregal
<http://www.altdevblogaday.com/2012/09/04/bringing-regal-opengl-to-native-client/>`_
for more info.

Use the Pepper API
------------------

Your code can call the Pepper PPB_OpenGLES2 API directly, as with any Pepper
interface. When you write in this way, each invocation of an OpenGL ES 2.0
function must begin with a reference to the Pepper interface, and the first
argument is the graphics context. To invoke the function ``glCompileShader``,
your code might look like:

.. naclcode::

  ppb_g3d_interface->CompileShader(graphicsContext, shader);

This approach specifically targets the Pepper APIs. Each call corresponds to a
OpenGL ES 2.0 function, but the syntax is unique to Native Client, so the source
file is not portable.

Implementing a rendering loop
=============================

Graphics applications require a continuous frame render-and-redraw cycle that
runs at a high frequency. To achieve the best frame rate, is important to
understand how the OpenGL ES 2.0 code in a Native Client module interacts with
Chrome.

The Chrome and Native Client processes
--------------------------------------

Chrome is a multi-process browser. Each Chrome tab is a separate process that is
running an application with its own main thread (we'll call it the Chrome main
thread). When an application launches a Native Client module, the module runs in
a new, separate sandboxed process. The module's process has its own main thread
(the Native Client thread). The Chrome and Native Client processes communicate
with each other using Pepper API calls on their main threads.

When the Chrome main thread calls the Native Client thread (keyboard and mouse
callbacks, for example), the Chrome main thread will block. This means that
lengthy operations on the Native Client thread can steal cycles from Chrome, and
performing blocking operations on the Native Client thread can bring your app to
a standstill.

Native Client uses callback functions to synchronize the main threads of the
two processes. Only certain Pepper functions use callbacks; `SwapBuffers
</native-client/pepper_stable/c/struct_p_p_b___graphics3_d__1__0#a293c6941c0da084267ffba3954793497>`_
is one.

``SwapBuffers`` and its callback function
-----------------------------------------

``SwapBuffers`` is non-blocking; it is called from the Native Client thread and
returns immediately. When ``SwapBuffers`` is called, it runs asynchronously on
the Chrome main thread. It switches the graphics data buffers, handles any
needed compositing operations, and redraws the screen. When the screen update is
complete, the callback function that was included as one of ``SwapBuffer``'s
arguments will be called from the Chrome thread and executed on the Native
Client thread.

To create a rendering loop, your Native Client module should include a function
that does the rendering work and then executes ``SwapBuffers``, passing itself
as the ``SwapBuffer`` callback. If your rendering code is efficient and runs
quickly, this scheme will achieve the highest frame rate possible. The
documentation for ``SwapBuffers`` explains why this is optimal: because the
callback is executed only when the plugin's current state is actually on the
screen, this function provides a way to rate-limit animations. By waiting until
the image is on the screen before painting the next frame, you can ensure you're
not generating updates faster than the screen can be updated.

The following diagram illustrates the interaction between the Chrome and Native
Client processes. The application-specific rendering code runs in the function
called ``Draw`` on the Native Client thread. Blue down-arrows are blocking calls
from the main thread to Native Client, green up-arrows are non-blocking
``SwapBuffers`` calls from Native Client to the main thread. All OpenGL ES 2.0
calls are made from ``Draw`` in the Native Client thread.

.. image:: /images/3d-graphics-render-loop.png

SDK example ``graphics_3d``
---------------------------

The SDK example ``graphics_3d`` uses the function ``MainLoop`` (in
``hello_world.cc``) to create a rendering loop as described above. ``MainLoop``
calls ``Render`` to do the rendering work, and then invokes ``SwapBuffers``,
passing itself as the callback.

.. naclcode::

  void MainLoop(void* foo, int bar) {
    if (g_LoadCnt == 3) {
      InitProgram();
      g_LoadCnt++;
    }
    if (g_LoadCnt > 3) {
      Render();
      PP_CompletionCallback cc = PP_MakeCompletionCallback(MainLoop, 0);
      ppb_g3d_interface->SwapBuffers(g_context, cc);
    } else {
      PP_CompletionCallback cc = PP_MakeCompletionCallback(MainLoop, 0);
      ppb_core_interface->CallOnMainThread(0, cc, 0);
    }
  }

Managing the OpenGL ES 2.0 pipeline
===================================

OpenGL ES 2.0 commands do not run in the Chrome or Native Client processes. They
are passed into a FIFO queue in shared memory which is best understood as a `GPU
command buffer
<http://www.chromium.org/developers/design-documents/gpu-command-buffer>`_. The
command buffer is shared by a dedicated GPU process. By using a separate GPU
process, Chrome implements another layer of runtime security, vetting all OpenGL
ES 2.0 commands and their arguments before they are sent on to the
GPU. Buffering commands through the FIFO also speeds up your code, since each
OpenGL ES 2.0 call in your Native Client thread returns immediately, while the
processing may be delayed as the GPU works down the commands queued up in the
FIFO.

Before the screen is updated, all the intervening OpenGL ES 2.0 commands must be
processed by the GPU. Programmers often try to ensure this by using the
``glFlush`` and ``glFinish`` commands in their rendering code. In the case of
Native Client this is usually unnecessary. The ``SwapBuffers`` command does an
implicit flush, and the Chrome team is continually tweaking the GPU code to
consume the OpenGL ES 2.0 FIFO as fast as possible.

Sometimes a 3D application can write to the FIFO in a way that's difficult to
handle. The command pipeline may fill up and your code will have to wait for the
GPU to flush the FIFO. If this is the case, you may be able to add ``glFlush``
calls to speed up the flow of the OpenGL ES 2.0 command FIFO. Before you start
to add your own flushes, first try to determine if pipeline saturation is really
the problem by monitoring the rendering time per frame and looking for irregular
spikes that do not consistently fall on the same OpenGL ES 2.0 call. If you're
convinced the pipeline needs to be accelerated, insert ``glFlush`` calls in your
code before starting blocks of processing that do not generate OpenGL ES 2.0
commands. For example, issue a flush before you begin any multithreaded particle
work, so that the command buffer will be clear when you start doing OpenGL ES
2.0 calls again. Determining where and how often to call ``glFlush`` can be
tricky, you will need to experiment to find the sweet spot.

Rendering and inactive tabs
===========================

Users will often switch between tabs in a multi-tab browser. A well-behaved
application that's performing 3D rendering should pause any real-time processing
and yield cycles to other processes when its tab becomes inactive.

In Chrome, an inactive tab will continue to execute timed functions (such as
``setInterval`` and ``setTimeout``) but the timer interval will be automatically
overridden and limited to not less than one second while the tab is inactive. In
addition, any callback associated with a ``SwapBuffers`` call will not be sent
until the tab is active again. You may receive asynchronous callbacks from
functions other than ``SwapBuffers`` while a tab is inactive. Depending on the
design of your application, you might choose to handle them as they arrive, or
to queue them in a buffer and process them when the tab becomes active.

The time that passes while a tab is inactive can be considerable. If your main
thread pulse is based on the ``SwapBuffers`` callback, your app won't update
while a tab is inactive. A Native Client module should be able to detect and
respond to the state of the tab in which it's running. For example, when a tab
becomes inactive, you can set an atomic flag in the Native Client thread that
will skip the 3D rendering and ``SwapBuffers`` calls and continue to call the
main thread every 30 msec or so. This provides time to update features that
should still run in the background, like audio. It may also be helpful to call
``sched_yield`` or ``usleep`` on any worker threads to release resources and
cede cycles to the OS.

Handling tab activation from the main thread
--------------------------------------------

You can detect and respond to the activation or deactivation of a tab with
JavaScript on your hosting page. Add an EventListener for ``visibilitychange``
that sends a message to the Native Client module, as in this example:

.. naclcode::

  document.addEventListener('visibilitychange', function(){
    if (document.hidden) {
      // PostMessage to your Native Client module
      document.nacl_module.postMessage('INACTIVE');
    } else {
      // PostMessage to your Native Client module
      document.nacl_module.postMessage('ACTIVE');
    }

  }, false);

Handling tab activation from the Native Client thread
-----------------------------------------------------

You can also detect and respond to the activation or deactivation of a tab
directly from your Native Client module by including code in the function
``pp::Instance::DidChangeView``, which is called whenever a change in the
module's view occurs. The code can call ``ppb::View::IsPageVisible`` to
determine if the page is visible or not. The most common cause of invisible
pages is that the page is in a background tab.

Tips and best practices
=======================

Here are some suggestions for writing safe code and getting the maximum
performance with the Pepper 3D API.

Do's
----

* **Make sure to enable attrib 0.** OpenGL requires that you enable attrib 0,
  but OpenGL ES 2.0 does not. For example, you can define a vertex shader with 2
  attributes, numbered like this:

  .. naclcode::

    glBindAttribLocation(program, "positions", 1);
    glBindAttribLocation(program, "normals", 2);

  In this case the shader is not using attrib 0 and Chrome may have to perform
  some additional work if it is emulating OpenGL ES 2.0 on top of OpenGL. It's
  always more efficient to enable attrib 0, even if you do not use it.

* **Check how shaders compile.** Shaders can compile differently on different
  systems, which can result in ``glGetAttrib*`` functions returning different
  results. Be sure that the vertex attribute indices match the corresponding
  name each time you recompile a shader.

* **Update indices sparingly.** For security reasons, all indices must be
  validated. If you change indices, Native Client will validate them
  again. Structure your code so indices are not updated often.

* **Use a smaller plugin and let CSS scale it.** If you're running into fillrate
  issues, it may be beneficial to perform scaling via CSS. The size your plugin
  renders is determined by the width and height attributes of the ``<embed>``
  element for the module. The actual size displayed on the web page is
  controlled by the CSS styles applied to the element.

* **Avoid matrix-to-matrix conversions.** With some versions of Mac OS, there is
  a driver problem when compiling shaders. If you get compiler errors for matrix
  transforms, avoid matrix-to-matrix conversions. For instance, upres a vec3 to
  a vec4 before transforming it by a mat4, rather than converting the mat4 to a
  mat3.

Don'ts
------

* **Don't use client side buffers.** OpenGL ES 2.0 can use client side data with
  ``glVertexAttribPointer`` and ``glDrawElements``, but this is really slow. Try
  to avoid client side buffers. Use Vertex Buffer Objects (VBOs) instead.

* **Don't mix vertex data and index data.** By default, Pepper 3D binds buffers
  to a single point. You could create a buffer and bind it to both
  ``GL_ARRAY_BUFFER`` and ``GL_ELEMENT_ARRAY_BUFFER``, but that would be
  expensive overhead and it is not recommended.

* **Don't call glGet* or glCheck* during rendering.** This is normal
  advice for OpenGL programs, but is particularly important for 3D on
  Chrome. Calls to any OpenGL ES 2.0 function whose name begins with these
  strings blocks the Native Client thread. This includes ``glGetError``; avoid
  calling it in release builds.

* **Don't use fixed point (GL_FIXED) vertex attributes.** Fixed point
  attributes are not supported in OpenGL ES 2.0, so emulating them in OpenGL ES
  2.0 is slow. By default, ``GL_FIXED`` support is turned off in the Pepper 3D
  API.

* **Don't read data from the GPU.** Don't call ``glReadPixels``, as it is slow.

* **Don't update a small portion of a large buffer.** In the current OpenGL ES
  2.0 implementation when you update a portion of a buffer (with
  ``glSubBufferData`` for example) the entire buffer must be reprocessed. To
  avoid this problem, keep static and dynamic data in different buffers.

* **Don't call glDisable(GL_TEXTURE_2D).** This is an OpenGL ES 2.0
  error. Each time it is called, an error messages will appear in Chrome's
  ``about:gpu`` tab.