third_party/blink/renderer/modules/content_extraction/readme.md

# Annotated Page Content (APC)

## 1\. Introduction

### What is APC?

Annotated Page Content (APC) is a structured and actionable representation of a
webpage's content and layout. Its primary function is to enable a deep
understanding of page structure, content, and interactive elements by
downstream clients, who can receive the information as a protobuf tree.

### Core Principles

APC is designed with the following principles in mind:

* **Completeness:** Capture all relevant page information, including text,
images, forms, and tables. This encompasses content that is both visible in the
viewport and findable through user actions like scrolling or searching.
* **Actionability:** Allow systems to not only parse content but also identify
and support interactions with elements like buttons, links, and form fields.
* **Consistency:** Provide a stable representation of the page, even as it
changes, to support reliable multi-step interactions.
* **Efficiency:** Minimize the computational cost and data size required.
* **Extensibility:** Support a wide variety of current and future features.
* **Privacy & Security:** Prevent the leakage of sensitive user information and
protect against security threats like cross-origin attacks.
* **Safety:** Support pre-action verification before performing requested tasks
on a page.

## 2\. The APC Data Structure

The foundation of APC is the `AnnotatedPageContent` protobuf message, which
organizes page content into a hierarchical tree.

### A Tree of `ContentNode`s

The representation is a tree of `ContentNode`s. These nodes can represent
layout containers on the page, grouping related information in a structure
derived from the layout tree. This includes:

* Content sectioning elements (`<article>`, `<nav>`, `<section>`)
* Lists, tables, and forms
* Text, images, paragraphs, headings, and links
* Interactive elements
* Iframes, with origin information

### Key Information in Each Node (`ContentAttributes`)

Each `ContentNode` contains attributes that describe the element in detail:

* **General Metadata:** Includes a unique content node ID, DOM Node ID(s), the
role of the content (e.g., header, main), and the node's type.
* **Geometry:** Bounding box coordinates for each node are provided, allowing
its content to be mapped to visual representations of the page like screenshots.
* **Text (`TextInfo`):** The text content, along with styling information like
size, emphasis, and color.
* **Images (`ImageInfo`):** The image's alt text or caption, its URL, and
  security origin. SVG image titles and descriptions supply captions. For HTML
  images without an `alt` attribute, `title` supplies the caption. These caption
  sources do not also supply fallback labels, even when redundant label removal
  is disabled. Separate tooltips and ARIA labels can still supply labels.
* **Links (`AnchorData`):** The destination URL and the link's `rel` attribute.
* **Forms (`FormInfo`, `FormControlData`):** Includes the form's name/ID and
data for individual controls like field name, value, and type. Password field
values are omitted unless the user has made them visible on the page.
* **Interaction (`InteractionInfo`):** Describes the node's interactivity
(e.g., clickable, editable, focusable).

### What's Not Currently Included

The following elements are under consideration for future inclusion but are not
currently part of the APC structure:

* Multimedia (`<audio>`, `<video>`)
* Canvas (`<canvas>`) and SVG (`<svg>`)
* Scripts (explicitly excluded as they are not user-visible content)
* Structured PDF content (currently, only raw bytes are sent)

## 3\. How APC is Generated

APC is generated by traversing Blink's **layout tree**, not the DOM tree. This
is a critical distinction because the layout tree only includes content that is
actually rendered on the page.

The generation algorithm recursively traverses the layout tree, creating a
`ContentNode` for each rendered object with structured content or a significant
semantic role. It extracts relevant data and organizes the nodes into a
hierarchy that preserves the visual order of the page.

## 4\. Using APC: Formats and Interactions

### Available Formats ("Views")

On the browser side, the raw APC proto can be converted into various consumable
formats, including:

* **Structured Markdown:** A Markdown representation of the page that preserves
structure and visual order. Elements in the Markdown can be labeled with unique
IDs (`{#ID}`) that link back to the original `ContentNode`.
* **Passage Chunks:** The visible content of the page broken down into
consistently sized passages, useful for citing specific sections of the page.

### Enabling Page Interactions

A key goal of APC is to enable reliable interactions with webpages, even when
they change dynamically.

To handle dynamic page changes, an algorithm robustly identifies the target
element by matching key properties like its type, interactivity, and location.
If needed, it can further verify the element by comparing its text content to
ensure the correct action is taken.

### Selective Node ID Emission

APC now supports a selective node-id policy in `AIPageContentOptions`:

* `node_id_allowlist`: Optional list of `AIPageContentAttributeType` values
  that should emit `dom_node_id`.

The main goal is to avoid assigning DOM node ids to more nodes than needed.
Over-emitting ids grows Blink's DOM node id hash map, which hurts overall
renderer performance after extraction finishes.

Semantics:

* If `node_id_allowlist` is unset, APC preserves legacy behavior and emits ids
  broadly.
* If `node_id_allowlist` is set (empty or non-empty), APC always emits ids for
  required override cases (for example actionable targets and metadata-linked
  nodes such as focus/selection/label-for references).
* If `node_id_allowlist` is non-empty, APC also emits ids for the listed
  attribute types.

## 5\. Critical Considerations for Implementation

Using APC requires careful attention to privacy and security. While APC
provides data to help mitigate risks, feature owners bear ultimate
responsibility.

* **Data Exfiltration and Origin Tracking:** Webpages often contain content
from multiple origins (e.g., in iframes). APC tags all data with its source
origin, allowing consumers to detect and handle cross-origin information
appropriately.
* **Handling Password-Like Fields:** Values from password fields (and other
  password-like text inputs detected via heuristics such as
  `-webkit-text-security`) are removed from the APC representation to help
  prevent sensitive credential leakage.
* **Paywalled Content:** APC's design helps exclude most paywalled content.
Websites can also use specific markup
([`isAccessibleForFree=false`](https://developers.google.com/search/docs/appeara
nce/structured-data/paywalled-content)) to flag paid content, and APC includes
this signal.
* **Data from Protected Environments:** Systems using APC should be aware that
content may originate from sources with special data handling requirements.
Consumers of APC data are responsible for enforcing all applicable data
protection and access control rules.
* **Guidelines for Storing APC Data:** Due to the potential for private
information, APC data or its derivatives should not be persisted beyond the
scope of a user's immediate task without explicit user consent.

## 6. Testing

### Unit Tests

To run the unit tests for content extraction, use the following command:

```bash
autoninja -C out/Default blink_unittests && out/Default/blink_unittests --gtest_filter=AIPageContentAgentTest.*
```

### Web Tests

The web tests for content extraction are located in
`third_party/blink/web_tests/content_extraction/`.

To run the web tests:

```bash
third_party/blink/tools/run_web_tests.py -C out/Default content_extraction
```

To update the web test expectations:

```bash
third_party/blink/tools/run_web_tests.py -C out/Default content_extraction --reset-results
```

### Feature Flags and APC-on-load

`AIPageContentOuterBoxMapToAncestorSpace` reuses the GeometryMapper mapping for
both the `outer_bounding_box` and `visible_bounding_box`. It is enabled by
default because it is a stable Blink runtime feature.

`AIPageContentBuildOnLoadForTesting` forces every local root frame to build APC
(in actionable mode) immediately after load. This mirrors the behaviour used by
the AnnotatedPageContentExtraction Finch trial.

For example, to launch content_shell with APC-on-load enabled:

```bash
out/Default/content_shell \
  --enable-blink-features=AIPageContentBuildOnLoadForTesting
```

A dedicated virtual test suite (`content-extraction`) exercises existing
layout tests with APC-on-load enabled while reusing the default geometry path:

```bash
third_party/blink/tools/run_web_tests.py \
    -C out/Default \
    --virtual-test-suite content-extraction
```