Elements and Interaction Sequences

This folder contains primitives for locating named elements in different application windows as well as following specific sequences of user interactions with the UI. It is designed to support things like User Education Tutorials and interactive UI testing, and to work with multiple UI presentation frameworks.

See the Supported Frameworks section for a list of currently-supported frameworks.

Named elements

Named elements are the fundamental building blocks of these systems. Each supported framework must define how a UI element can be assigned an ElementIdentifier which its framework implementation must be able to read. A UI element with a non-null ElementIdentifier assigned to it is known as a named element.

Each ElementIdentifier contains a globally unique opaque handle, with its underlying value based on the address of a block of memory allocated at compile time.

Two named elements with the same identifier are not necessarily equivalent. In an application with multiple primary windows (such as a browser with several windows and a PWA open), each primary window and all of its secondary UI - menus, dialogs, bubbles, etc. - are represented by a single ElementContext. Like identifiers, contexts are opaque handles that are unique to each primary window. To get the context associated with a UI element or window, you will need to call a method specific to your framework; see the relevant section below.

ElementIdentifier and ElementContext both support the methods one might expect from a handle or opaque pointer:

• Assignment (=)
• Equality (==, !=)
• Ordering (<) - for use in std::set and std::map
• Boolean value (! and explicit operator bool)
• Conversion to and from an integer type (intptr_t) - for platforms written in languages that don't support pointers

The only falsy/null/zero value is the default-constructed value. No guarantees are made about the ordering produced by the < operator; only that there is one.

Creating ElementIdentifier values for your application

You will want to create named constants representing each identifier you want to use in your code. These constants are defined using macros found in element_identifier.h.

Each declaration creates a compile-time constant that can be copied and used anywhere in your application - they are valid from the time your application starts up until it exits.

Furthermore, every ElementIdentifier you declare in code will either be default-constructed (and therefore null), or be assigned a copy of one of these compile-time constants. You should never construct an ElementIdentifier or assign a value from anything other than another ElementIdentifier.

To create a public, unique ElementIdentifier value in a global scope:

// This goes in the .h file:
DECLARE_ELEMENT_IDENTIFIER_VALUE(kMyIdentifierName);

// This goes in the .cc file:
DEFINE_ELEMENT_IDENTIFIER_VALUE(kMyIdentifierName);

To create a class member that is a unique identifier, use:

// This declares a unique identifier, MyClass::kClassMemberIdentifier. You could
// also make the identifier protected or private if you wanted.
class MyClass {
 public:
  DECLARE_CLASS_ELEMENT_IDENTIFIER_VALUE(MyClass, kClassMemberIdentifier);
};

// This goes in the .cc file:
DEFINE_CLASS_ELEMENT_IDENTIFIER_VALUE(MyClass, kClassMemberIdentifier);

And finally, if you want to create an identifier that's local to a .cc file or class method, there is a single-line declaration available, as shown in these examples:

// This declares a module-local identifier. The anonymous namespace is optional
// (though recommended) as kModuleLocalIdentifier will also be marked 'static'.
namespace {
DEFINE_LOCAL_ELEMENT_IDENTIFIER_VALUE(kModuleLocalIdentifier);
}

// This declares a method-local identifier.
/* static */ ui::ElementIdentifier MyClass::GetIdentifier() {
  DEFINE_LOCAL_ELEMENT_IDENTIFIER_VALUE(kMethodLocalIdentifier);
  return kMethodLocalIdentifier;
}

Please note that since all ElementIdentifiers created using these macros are constexpr/compile-time constants, copies of these values can be used outside of their originating classes or modules - the value of the ElementIdentifier is valid anywhere in the application.

UI Elements and element events

ElementTracker is a global singleton object that allows you to:

• Retrieve a visible element or elements from a particular context by identifier
• Register for callbacks when elements with a given identifier and context become visible
• Register for callbacks when elements with a given identifier and context become hidden or are destroyed
• Register for callbacks when the user interacts with an element with a given identifier and context (known as activation)

TrackedElement, a polymorphic class defined in element_tracker.h, represents a platform-agnostic UI element with an identifier and context. There must be a 1:1 correspondence between a visible named UI element and an TrackedElement; the TrackedElement is what is passed to callbacks when the UI element is shown, hidden, or activated.

Each framework has its own derived version of TrackedElement that may provide additional information about the element. If you know what platform the element is from you may use the AsA() template method to dynamically downcast to the platform-specific element type. If you are working in an environment with multiple presentation frameworks, you can use the IsA() method to determine if the element is of the expected type.

Here is an example that shows some of the functionality of ElementTracker and TrackedElement. Note that you must specify the ElementContext in which you are listening:

void ListenForShowEvent(ui::ElementIdentifier id, ui::ElementContext context) {
  auto callback =
      base::BindRepeating(&MyClass::OnElementShown, base::Unretained(this)));
  subscription_ = ui::ElementTracker::GetElementTracker()
      ->AddElementShownCallback(id, context, callback);
}

void OnElementShown(ui::TrackedElement* element) {
// Technically you don't need the IsA() call here, since AsA() returns null if
// the object is the wrong type.
if (element->IsA<views::TrackedElementViews>()) {
  views::View* const view =
      element->AsA<views::TrackedElementViews>()->view();
  // Do something with the view that was shown here.
}

Then, in your production code, assign an element identifier to the element you want to track:

// Note: matching DECLARE macro must go in header file.
DEFINE_CLASS_ELEMENT_IDENTIFIER_VALUE(MyView, kMyElementIdentifier);

MyView::MyView() {
  auto* const child_view = AddChildView(std::make_unique<ChildViewType>());

  // This child view will now generate events with the given identifier.
  // The context will be derived from the widget the parent view is added to.
  child_view->SetProperty(views::kElementIdentifierKey, kMyElementIdentifier);
}

Defining and following user interaction sequences

The InteractionSequence class provides a way to describe a sequence of interactions between the application and the user (real or simulated). Sequences are useful for e.g. creating user education tutorials guiding the user through the steps of using a new feature, or for simulating user input during interaction testing.

Each sequence consists of a series of steps of the following types:

• Shown - a UI element becomes visible to the user
• Activated - the user clics on or otherwise interacts with the UI element
• Hidden - a UI element stops being visible to the user

These are equivalent to the corresponding events provided by ElementTracker. All of the steps must be followed in order, or the sequence is aborted. Callbacks may be registered at the start and end of each step, and for when the sequence completes or aborts.

Events not in the sequence (other UI elements appearing, focus changes, mouse hover, scroll) are ignored unless they result in a required UI element being dismissed (such as if a dialog or menu is closed).

To create an interaction sequence, use a InteractionSequence::Builder. To add steps to the builder, use an InteractionSequence::StepBuilder, or call a convenience method like WithInitialElement(). Here is an example that expects the user to interact with a feature entry point and then displays a help bubble on the resulting dialog:

initial_element =
    ElementTracker::GetFirstMatchingElement(kFeatureEntryPointID, context());
sequence_ = InteractionSequence::Builder()
    .SetCompletedCallback(base::BindOnce(
        &MyClass::OnSequenceComplete,
        base::Unretained(this)))
    .AddStep(InteractionSequence::WithInitialElement(initial_element))
    .AddStep(InteractionSequence::StepBuilder()
        .SetElementID(initial_element->identifier())
        .SetType(StepType::kActivated)
        .Build())
    .AddStep(InteractionSequence::StepBuilder()
        .SetElementID(kFeatureDialogID)
        .SetType(StepType::kShown)
        .SetStartCallback(&MyClass::ShowHelpBubble, base::Unretained(this))
        .Build())
    .Build();
sequence_->Start();

Interaction steps

Each Step has properties that can be set via its StepBuilder:

• Type - whether this is a show, activate, or hide step
• Element ID - the identifier of the element involved in the step
• Must be visible at start - the element must be visible at the start of the step or else the sequence is aborted
• Must remain visible - the element must remain visible until the next step begins; not compatible with hidden steps
• Start callback - called as soon as the step is started
• End callback - called as soon as the next step is started, or the sequence aborts or completes

Of these, only Type and Element ID are required. If you do not specify whether the element must be visible at start or remain visible, default values will be assigned according to the type of step. All callbacks are optional.

Instead of using StepBuilder, for the initial step you can call InteractionSequence::WithInitialElement(). This creates a default shown step for an element that is already visible; it expects the element to be visible when Start() is called or the sequence will abort. You may pass optional step start and end callbacks to WithInitialElement(); these are useful for displaying an initial prompt to the user (in the case of a tutorial).

There is an additional method on StepBuilder, SetContext(), but it is only used by helper methods and for testing. You should instead use Builder::SetContext() or InteractionSequence::WithInitialElement(). There is currently no support for cross-context sequences and setting conflicting contexts in a sequence is an error and will crash if DCHECK is enabled.

Step callbacks

Each step callback (start or end) has three parameters:

• Element - the element involved in the step; null if the element is not available (i.e. was hidden before the callback could be called)
• Identifier - the ElementIdentifier associated with the step, which is always valid even if the element is null
• Type - the step type; shown, activated, or hidden

The element can be used to retrieve the UI element in your framework by downcasting via AsA() - see UI Elements and element events above.

Typically, when using a sequence to run a tutorial, this will be the code that shows or hides a tutorial dialog or prompt. When using the sequence for interaction testing, the callback will contain the code to simulate the next input to the UI.

Best practices

In general, it will be pretty obvious how to construct your sequence, because you know the steps you need to perform in the UI to get where you want to go. However, keep the following in mind:

• Try to start the sequence with a step generated by WithInitialElement(), keyed to a UI element you know will be visible when the sequence starts.
• Do not assume the order in which elements will become visible when a surface is shown.
• Do not assume that interacting with a button or menu item will bring up a resulting surface (another menu, a dialog) before the initial button or menu item disappears.

To elaborate on the third point: it is better to have the following steps in the case where a menu item brings up a dialog:

1. Menu item shown
2. Menu item activated (does not need to remain visible; default)
3. Dialog element shown (does not need to be visible at start; default)

If you specify that the menu item must stay visible or that the dialog element must be visible at step start, the sequence could fail depending on the order in which the presentation framework dismisses the menu and displays the dialog.

However, in the case where you want the user to navigate a series of submenus, if the platform supports menu-open-via-hover you may not receive the activated signal and a sequence like the following might work better:

1. Menu item shown
2. Submenu item shown (triggers as soon as the submenu is opened, regardless of how)
3. Submenu item activated
4. ...

Known limitations

• Cannot nest sequences (might be able to in some cases via callbacks)
• Cannot provide alternate sets of steps in the same sequence
• Cannot skip ahead (e.g. if the user uses a shortcut key to bypass a menu)
• Cannot restart steps (e.g. if the user hovers a submenu containing the next element, then un-hovers it, then hovers it again)

All of these can be addressed if a relevant, concrete need is found.

Supporting additional UI frameworks

If you want to use ElementTracker with a framework that isn't supported yet, you must at minimum do the following:

1. Derive a class from TrackedElement representing visual elements in your framework.
2. Determine how ElementContexts are defined in your framework.
3. Implement code to create and register your derived element objects with ElementTracker when UI elements become visible to the user, send events when they are activated by the user (however you choose to define “activation”), and to unregister them when the element is no longer visible.

See ElementTrackerViews for an example implementation.

When you are done, please add the folder containing the implementation code to the Supported Frameworks section below.

1. Derive a class from TrackedElement

When you derive a class from TrackedElement to use for your UI framework, you are obliged to declare specific metadata in order to support IsA() and AsA(). To do this, add the following to the class definition:

class TrackedElementMyPlatform {
 public:
  // This provides the required TrackedElement metadata support.
  DECLARE_ELEMENT_TRACKER_METADATA();
}

// In the corresponding .cc file:
DEFINE_ELEMENT_TRACKER_METADATA(TrackedElementMyPlatform)

You will also be expected to pass an immutable identifier and context into the constructor, and if the element object stores a pointer or handle to the associated UI element in your framework, that reference should not change for the life of the element (and the element should not outlive the corresponding framework object).

2. Define how ElementContext works in your framework

You will need a method to generate an ElementContext from a window or UI element. The context identifies the primary window associated with the UI, such as a browser or PWA window, the taskbar of an operating system's GUI, a file browser window, etc. Good candidates are the handle of the primary window or the address of the framework object that represents it. The value of the handle must not change over the lifetime of that window.

If you do not already have one, you will probably want a helper method that finds the primary window given a UI element that might be in secondary UI (such as a dialog or menu).

In Views, we use the address of the primary window's Widget to construct the ElementContext and we provide the ElementTrackerViews::GetContextForView() method to fetch it. We also added the Widget::GetPrimaryWindowWidget() helper method for finding the primary window.

3. Managing the lifetime of your elements and sending events

How your platform manages the lifetime of elements is entirely up to you. You could create an TrackedElement whenever a named UI element in your framework becomes visible to the user, or you could have every UI element with an associated ElementIdentifier hold a permanent TrackedElement.

The one requirement is that a single TrackedElement must be associated with any named UI element, and must remain associated with that implementation as long as the element remains visible.

To register or unregister an element or send activation events, get the ElementTrackerFrameworkDelegate from the ElementTracker class and call the appropriate method:

auto* const delegate = ui::ElementTracker::GetFrameworkDelegate();

// Register a visible element:
delegate->NotifyElementShown(my_element);

// Notify that the element was activated by the user:
delegate->NotifyElementActivated(my_element);

// Unregister the element on hide:
delegate->NotifyElementHidden(my_element);

“Activation” requires some special discussion here, as what it means to be activated is extremely context-specific. For example:

• A button is activated when the user clicks it
• A menu item is activated when the user performs the item's action
• A browser tab is activated when the user selects the tab with the mouse or keyboard

Activation should be the default action that occurs when the user directly interacts with that UI element. So for example, the Back button in a browser can be clicked to return to the previous page, or long-pressed/dragged to open a menu containing a list of previous pages. The single-click is the default action, and therefore should be the action that results in NotifyElementActivated() being called.

How you proxy events from UI elements to calls to ElementTrackerFrameworkDelegate is entirely up to you. In Views we go through a mediator object - ElementTrackerViews - which first maps the Button, MenuItem, etc. to an TrackedElementViews before passing that object to the appropriate delegate method.

Supported Frameworks

The following UI frameworks support ElementTracker and InteractionSequence. Please add additional frameworks to this list as they become supported.

• Views:
  • ElementTrackerViews
  • InteractionSequenceViews