docs/memory-infra/memory_benchmarks.md - chromium/src.git - Git at Google

 # Memory Benchmarks

 This document describes benchmarks available to track Chrome's and
 WebView's memory usage, where they live, what they measure, how to run them,
 and on how to diagnose regressions.

 [TOC]

 ## Glossary

 *   **User story:** a set of actions to perform on a browser or device (e.g.
     open google homepage, type "foo", click search, scroll down, visit first
     result, etc.).
 *   **Metric:** a data aggregation process that takes a Chrome trace as input
     (produced by a [Telemetry][] run) and produces a set of summary numbers as
     output (e.g. total GPU memory used).
 *   **Benchmark:** a combination of (one or more) user stories and (one or
     more) metrics.

 [Telemetry]: https://github.com/catapult-project/catapult/blob/master/telemetry/README.md

 ## System Health

 *System health* is an effort to unify top-level benchmarks (as opposite to
 micro-benchmarks and regression tests) that are suitable to capture
 representative user stories.

 ### Benchmarks

 System health memory benchmarks are:

 *   [system_health.memory_mobile][system_health] -
     user stories running on Android devices.
 *   [system_health.memory_desktop][system_health] -
     user stories running on desktop platforms.

 These benchmarks are run continuously on the [chrome.perf][] waterfall,
 collecting and reporting results on the
 [Chrome Performance Dashboard][chromeperf].

 [system_health]: https://chromium.googlesource.com/chromium/src/+/main/tools/perf/page_sets/system_health/
 [chrome.perf]: https://ci.chromium.org/p/chrome/g/chrome.perf/console
 [chromeperf]: https://chromeperf.appspot.com/report

 ### User stories

 System health user stories are classified by the kind of interactions they
 perform with the browser:

 *   `browse` stories navigate to a URL and interact with the page; e.g.
     scroll, click on elements, navigate to subpages, navigate back.
 *   `load` stories just navigate to a URL and wait for the page to
     load.
 *   `background` stories navigate to a URL, possibly interact with the
     page, and then bring another app to the foreground (thus pushing the
     browser to the background).
 *   `long_running` stories interact with a page for a longer period
     of time (~5 mins).
 *   `multitab` loads different web sites in several tabs, then cycles through
     them.
 *   `play` loads a web site and plays some media (e.g. a song).

 The full name of a story has the form `{interaction}:{category}:{site}[:{year}]`
 where:

 *   `interaction` is one the labels given above;
 *   `category` is used to group together sites with a similar purpose,
     e.g. `news`, `social`, `tools`;
 *   `site` is a short name identifying the website in which the story mostly
     takes place, e.g. `cnn`, `facebook`, `gmail`.
 *   `year` indicates the year in which the web page recording for the story
     was most recently updated.

 For example `browse:news:cnn:2018` and `background:social:facebook` are two
 system health user stories. The list of all current stories can be found at
 [bit.ly/csh-stories](http://bit.ly/csh-stories).

 Today, for most stories, a garbage collection is forced at the end of the
 story and a memory dump is then triggered. Metrics report the values
 obtained from this single measurement.

 ## Continuous monitoring

 ![Chrome Performance Dashboard](https://storage.googleapis.com/chromium-docs.appspot.com/79d08f59cf497c761f7099ea427704c14e9afc03.png)

 To view data from one of the benchmarks on the
 [Chrome Performance Dashboard][chromeperf] you should select:

 *   **Test suite:** The name of a *[benchmark](#Benchmarks)*.
 *   **Bot:** The name of a *platform or device configuration*. Sign in to also
     see internal bots.
 *   **Subtest (1):** The name of a *[metric](#Understanding-memory-metrics)*.
 *   **Subtest (2):** The name of a *story group*; these have the form
     `{interaction}_{category}` for system health stories.
 *   **Subtest (3):** The name of a *[user story](#User-stories)*
     (with `:` replaced by `_`).

 Clicking on any point of the graph will give you the commit range, links to the
 builder that ran the benchmark, and a trace file collected during the story
 run. See below for details on how to interpret these traces when
 [debugging memory related issues](#debugging-memory-regressions).

 Many of the high level memory measurements are automatically tracked and the
 Performance Dashboard will generate alerts when a memory regression is detected.
 These are triaged by [perf sheriffs][] who create bugs and start bisect jobs
 to find the root cause of regressions.

 [perf sheriffs]: /docs/speed/perf_regression_sheriffing.md

 ![Chrome Performance Dashboard Alert](https://storage.googleapis.com/chromium-docs.appspot.com/perfdashboard_alert.png)

 ## Debugging memory regressions

 If you are investigating a memory regression, chances are, a [pinpoint][]
 job identified one of your CLs as a possible culprit.

 ![Pinpoint Regression](https://storage.googleapis.com/chromium-docs.appspot.com/pinpoint_regression.png)

 Note the "chart" argument identifies the memory metric that regressed. The
 pinpoint results page also gives you easy access to traces before and after
 your commit landed. It's useful to look at both and compare them to identify what
 changed. The documentation on [memory-infra][memory-infra] explains how to dig
 down into details and interpret memory measurements. Also note that pinpoint
 runs each commit multiple times, so you can access more traces by clicking on
 a different "repeat" of either commit.

 Sometimes it's also useful to follow the link to "Analyze benchmark results"
 which will bring up the [Metrics Results UI][results-ui] to compare all
 measurements (not just the one caught by the alert) before and after your
 CL landed. Make sure to select the "before" commit as reference column, show
 absolute changes (i.e. "Δavg") instead of relative, and sort by the column
 with changes on the "after" commit to visualize them more easily. This can be
 useful to find a more specific source of the regression, e.g.
 `renderer_processes:reported_by_chrome:v8:heap:code_space:effective_size`
 rather than just `all_processes:reported_by_chrome:effective_size`, and help
 you pin down the source of the regression.

 To confirm whether a revert of your CL would fix the regression you can run
 a [pinpoint try job](#How-to-run-a-pinpoint-try-job) with a patch containing
 the revert. Finally, **do not close the bug** even if you suspect that your CL
 may not be the cause of the regression; instead follow the more general
 guidance on how to [address performance regressions][addressing-regressions].
 Bugs should only be closed if the regression has been fixed or justified.

 [results-ui]: https://chromium.googlesource.com/catapult.git/+/HEAD/docs/metrics-results-ui.md
 [memory-infra]: /docs/memory-infra/README.md
 [addressing-regressions]: /docs/speed/addressing_performance_regressions.md

 ## How to run the benchmarks

 Benchmarks may be run on a local platform/device or remotely on a pinpoint
 try job.

 ### How to run a pinpoint try job

 Given a patch already uploaded to code review, try jobs provide a convenient
 way to evaluate its memory implications on devices or platforms which
 may not be immediately available to developers.

 ![New pinpoint try job dialog](https://storage.googleapis.com/chromium-docs.appspot.com/yHRMmUqraqJ.png)

 To start a try job go to the [pinpoint][] website, click on the `+` button to
 create a new job, and fill in the required details:

 [pinpoint]: https://pinpoint-dot-chromeperf.appspot.com/

 * **Bug ID** (optional): The id of a crbug.com issue where pinpoint can post
   updates when the job finishes.
 * **Gerrit URL**: URL to the patch you want to test. Note that your patch can
   live in chromium or any of its sub-repositories!
 * **Bot**: Select a suitable device/platform from the drop-down menu on which
   to run your job.
 * **Benchmark**: The name of the benchmark to run. If you are interested in
   memory try `system_health.memory_mobile` or `system_health.memory_desktop`
   as appropriate.
 * **Story** (optional): A pattern (Python regular expression) passed to
   Telemetry's `--story-filter` option to only run stories that match the
   pattern.
 * **Extra Test Arguments** (optional): Additional command line arguments for
   Telemetry's `run_benchmark`. Of note, if you are interested in running a
   small but representative sample of system health stories you can pass
   `--story-tag-filter health_check`.

 If you have more specific needs, or need to automate the creation of jobs, you
 can also consider using [pinpoint_cli][].

 [pinpoint_cli]: https://cs.chromium.org/chromium/src/tools/perf/pinpoint_cli

 ### How to run locally

 After building, e.g. `ChromePublic.apk`, you can run a specific system health
 story with the command:

 ```
 $SRC/tools/perf/run_benchmark run system_health.memory_mobile \
     --browser android-chromium --story-filter load:search:google
 ```

 This will run the story with a default of 3 repetitions and produce a
 `results.html` file comparing results from this and any previous benchmark
 runs. In addition, you'll also get individual [trace files][memory-infra]
 for each story run by the benchmark. **Note:** by default only high level
 metrics are shown, you may need to tick the "Show all" check box in order to
 view some of the lower level memory metrics.

 ![Example results.html file](https://storage.googleapis.com/chromium-docs.appspot.com/ea60207d9bb4809178fe75923d6d1a2b241170ef.png)

 Other useful options for this command are:

 *   `--pageset-repeat [n]` - override the default number of repetitions
 *   `--reset-results` - clear results from any previous benchmark runs in the
     `results.html` file.
 *   `--results-label [label]` - give meaningful names to your benchmark runs,
     this way it is easier to compare them.

 For WebView make sure to [replace the system WebView][webview_install]
 on your device and use `--browser android-webview`.

 [memory-infra]: /docs/memory-infra/README.md
 [webview_install]: https://www.chromium.org/developers/how-tos/build-instructions-android-webview

 ## Understanding memory metrics

 There is a large number of [memory-infra][] metrics, breaking down usage
 attributed to different components and processes.

 ![memory-infra metrics](https://storage.googleapis.com/chromium-docs.appspot.com/a73239c6367ed0f844500e51ce1e04556cb99b4f.png)

 Most memory metrics have the form
 `memory:{browser}:{processes}:{source}:{component}:{kind}`
 where:

 *   **browser:** One of `chrome` or `webview`.
 *   **processess:** One of `browser_process`, `renderer_processess`,
     `gpu_process`, or `all_processess`.
 *   **source:** One of `reported_by_chrome` or `reported_by_os`
 *   **component:** May be a Chrome component, e.g. `skia` or `sqlite`;
     details about a specific component, e.g. `v8:heap`; or a class of memory
     as seen by the OS, e.g. `system_memory:native_heap` or `gpu_memory`. If
     reported by chrome, the metrics are gathered by `MemoryDumpProvider`s,
     probes placed in the specific components' codebase. For example, in
     "memory:chrome:all_processes:reported_by_chrome:net:effective_size_avg,"
     the component is "net" which is Chrome's network stack and
     "reported_by_chrome" means that this metric is gathered via probes in
     the network stack.
 *   **kind:** The kind of memory being reported. For metrics reported by
     Chrome this usually is `effective_size` (others are `locked_size`
     and `allocated_objects_size`); for metrics by the OS this usually is
     `proportional_resident_size` (others are `peak_resident_size` and
     `private_dirty_size`).

 Read the [memory-infra documentation][memory-infra] for more details on them.

 [memory-infra]: /docs/memory-infra/README.md
	# Memory Benchmarks

	This document describes benchmarks available to track Chrome's and
	WebView's memory usage, where they live, what they measure, how to run them,
	and on how to diagnose regressions.

	[TOC]

	## Glossary

	* User story: a set of actions to perform on a browser or device (e.g.
	open google homepage, type "foo", click search, scroll down, visit first
	result, etc.).
	* Metric: a data aggregation process that takes a Chrome trace as input
	(produced by a [Telemetry][] run) and produces a set of summary numbers as
	output (e.g. total GPU memory used).
	* Benchmark: a combination of (one or more) user stories and (one or
	more) metrics.

	[Telemetry]: https://github.com/catapult-project/catapult/blob/master/telemetry/README.md

	## System Health

	System health is an effort to unify top-level benchmarks (as opposite to
	micro-benchmarks and regression tests) that are suitable to capture
	representative user stories.

	### Benchmarks

	System health memory benchmarks are:

	* [system_health.memory_mobile][system_health] -
	user stories running on Android devices.
	* [system_health.memory_desktop][system_health] -
	user stories running on desktop platforms.

	These benchmarks are run continuously on the [chrome.perf][] waterfall,
	collecting and reporting results on the
	[Chrome Performance Dashboard][chromeperf].

	[system_health]: https://chromium.googlesource.com/chromium/src/+/main/tools/perf/page_sets/system_health/
	[chrome.perf]: https://ci.chromium.org/p/chrome/g/chrome.perf/console
	[chromeperf]: https://chromeperf.appspot.com/report

	### User stories

	System health user stories are classified by the kind of interactions they
	perform with the browser:

	* `browse` stories navigate to a URL and interact with the page; e.g.
	scroll, click on elements, navigate to subpages, navigate back.
	* `load` stories just navigate to a URL and wait for the page to
	load.
	* `background` stories navigate to a URL, possibly interact with the
	page, and then bring another app to the foreground (thus pushing the
	browser to the background).
	* `long_running` stories interact with a page for a longer period
	of time (~5 mins).
	* `multitab` loads different web sites in several tabs, then cycles through
	them.
	* `play` loads a web site and plays some media (e.g. a song).

	The full name of a story has the form `{interaction}:{category}:{site}[:{year}]`
	where:

	* `interaction` is one the labels given above;
	* `category` is used to group together sites with a similar purpose,
	e.g. `news`, `social`, `tools`;
	* `site` is a short name identifying the website in which the story mostly
	takes place, e.g. `cnn`, `facebook`, `gmail`.
	* `year` indicates the year in which the web page recording for the story
	was most recently updated.

	For example `browse:news:cnn:2018` and `background:social:facebook` are two
	system health user stories. The list of all current stories can be found at
	[bit.ly/csh-stories](http://bit.ly/csh-stories).

	Today, for most stories, a garbage collection is forced at the end of the
	story and a memory dump is then triggered. Metrics report the values
	obtained from this single measurement.

	## Continuous monitoring

	![Chrome Performance Dashboard](https://storage.googleapis.com/chromium-docs.appspot.com/79d08f59cf497c761f7099ea427704c14e9afc03.png)

	To view data from one of the benchmarks on the
	[Chrome Performance Dashboard][chromeperf] you should select:

	* Test suite: The name of a [benchmark](#Benchmarks).
	* Bot: The name of a platform or device configuration. Sign in to also
	see internal bots.
	* Subtest (1): The name of a [metric](#Understanding-memory-metrics).
	* Subtest (2): The name of a story group; these have the form
	`{interaction}_{category}` for system health stories.
	* Subtest (3): The name of a [user story](#User-stories)
	(with `:` replaced by `_`).

	Clicking on any point of the graph will give you the commit range, links to the
	builder that ran the benchmark, and a trace file collected during the story
	run. See below for details on how to interpret these traces when
	[debugging memory related issues](#debugging-memory-regressions).

	Many of the high level memory measurements are automatically tracked and the
	Performance Dashboard will generate alerts when a memory regression is detected.
	These are triaged by [perf sheriffs][] who create bugs and start bisect jobs
	to find the root cause of regressions.

	[perf sheriffs]: /docs/speed/perf_regression_sheriffing.md

	![Chrome Performance Dashboard Alert](https://storage.googleapis.com/chromium-docs.appspot.com/perfdashboard_alert.png)

	## Debugging memory regressions

	If you are investigating a memory regression, chances are, a [pinpoint][]
	job identified one of your CLs as a possible culprit.

	![Pinpoint Regression](https://storage.googleapis.com/chromium-docs.appspot.com/pinpoint_regression.png)

	Note the "chart" argument identifies the memory metric that regressed. The
	pinpoint results page also gives you easy access to traces before and after
	your commit landed. It's useful to look at both and compare them to identify what
	changed. The documentation on [memory-infra][memory-infra] explains how to dig
	down into details and interpret memory measurements. Also note that pinpoint
	runs each commit multiple times, so you can access more traces by clicking on
	a different "repeat" of either commit.

	Sometimes it's also useful to follow the link to "Analyze benchmark results"
	which will bring up the [Metrics Results UI][results-ui] to compare all
	measurements (not just the one caught by the alert) before and after your
	CL landed. Make sure to select the "before" commit as reference column, show
	absolute changes (i.e. "Δavg") instead of relative, and sort by the column
	with changes on the "after" commit to visualize them more easily. This can be
	useful to find a more specific source of the regression, e.g.
	`renderer_processes:reported_by_chrome:v8:heap:code_space:effective_size`
	rather than just `all_processes:reported_by_chrome:effective_size`, and help
	you pin down the source of the regression.

	To confirm whether a revert of your CL would fix the regression you can run
	a [pinpoint try job](#How-to-run-a-pinpoint-try-job) with a patch containing
	the revert. Finally, do not close the bug even if you suspect that your CL
	may not be the cause of the regression; instead follow the more general
	guidance on how to [address performance regressions][addressing-regressions].
	Bugs should only be closed if the regression has been fixed or justified.

	[results-ui]: https://chromium.googlesource.com/catapult.git/+/HEAD/docs/metrics-results-ui.md
	[memory-infra]: /docs/memory-infra/README.md
	[addressing-regressions]: /docs/speed/addressing_performance_regressions.md

	## How to run the benchmarks

	Benchmarks may be run on a local platform/device or remotely on a pinpoint
	try job.

	### How to run a pinpoint try job

	Given a patch already uploaded to code review, try jobs provide a convenient
	way to evaluate its memory implications on devices or platforms which
	may not be immediately available to developers.

	![New pinpoint try job dialog](https://storage.googleapis.com/chromium-docs.appspot.com/yHRMmUqraqJ.png)

	To start a try job go to the [pinpoint][] website, click on the `+` button to
	create a new job, and fill in the required details:

	[pinpoint]: https://pinpoint-dot-chromeperf.appspot.com/

	* Bug ID (optional): The id of a crbug.com issue where pinpoint can post
	updates when the job finishes.
	* Gerrit URL: URL to the patch you want to test. Note that your patch can
	live in chromium or any of its sub-repositories!
	* Bot: Select a suitable device/platform from the drop-down menu on which
	to run your job.
	* Benchmark: The name of the benchmark to run. If you are interested in
	memory try `system_health.memory_mobile` or `system_health.memory_desktop`
	as appropriate.
	* Story (optional): A pattern (Python regular expression) passed to
	Telemetry's `--story-filter` option to only run stories that match the
	pattern.
	* Extra Test Arguments (optional): Additional command line arguments for
	Telemetry's `run_benchmark`. Of note, if you are interested in running a
	small but representative sample of system health stories you can pass
	`--story-tag-filter health_check`.

	If you have more specific needs, or need to automate the creation of jobs, you
	can also consider using [pinpoint_cli][].

	[pinpoint_cli]: https://cs.chromium.org/chromium/src/tools/perf/pinpoint_cli

	### How to run locally

	After building, e.g. `ChromePublic.apk`, you can run a specific system health
	story with the command:

	```
	$SRC/tools/perf/run_benchmark run system_health.memory_mobile \
	--browser android-chromium --story-filter load:search:google
	```

	This will run the story with a default of 3 repetitions and produce a
	`results.html` file comparing results from this and any previous benchmark
	runs. In addition, you'll also get individual [trace files][memory-infra]
	for each story run by the benchmark. Note: by default only high level
	metrics are shown, you may need to tick the "Show all" check box in order to
	view some of the lower level memory metrics.

	![Example results.html file](https://storage.googleapis.com/chromium-docs.appspot.com/ea60207d9bb4809178fe75923d6d1a2b241170ef.png)

	Other useful options for this command are:

	* `--pageset-repeat [n]` - override the default number of repetitions
	* `--reset-results` - clear results from any previous benchmark runs in the
	`results.html` file.
	* `--results-label [label]` - give meaningful names to your benchmark runs,
	this way it is easier to compare them.

	For WebView make sure to [replace the system WebView][webview_install]
	on your device and use `--browser android-webview`.

	[memory-infra]: /docs/memory-infra/README.md
	[webview_install]: https://www.chromium.org/developers/how-tos/build-instructions-android-webview

	## Understanding memory metrics

	There is a large number of [memory-infra][] metrics, breaking down usage
	attributed to different components and processes.

	![memory-infra metrics](https://storage.googleapis.com/chromium-docs.appspot.com/a73239c6367ed0f844500e51ce1e04556cb99b4f.png)

	Most memory metrics have the form
	`memory:{browser}:{processes}:{source}:{component}:{kind}`
	where:

	* browser: One of `chrome` or `webview`.
	* processess: One of `browser_process`, `renderer_processess`,
	`gpu_process`, or `all_processess`.
	* source: One of `reported_by_chrome` or `reported_by_os`
	* component: May be a Chrome component, e.g. `skia` or `sqlite`;
	details about a specific component, e.g. `v8:heap`; or a class of memory
	as seen by the OS, e.g. `system_memory:native_heap` or `gpu_memory`. If
	reported by chrome, the metrics are gathered by `MemoryDumpProvider`s,
	probes placed in the specific components' codebase. For example, in
	"memory:chrome:all_processes:reported_by_chrome:net:effective_size_avg,"
	the component is "net" which is Chrome's network stack and
	"reported_by_chrome" means that this metric is gathered via probes in
	the network stack.
	* kind: The kind of memory being reported. For metrics reported by
	Chrome this usually is `effective_size` (others are `locked_size`
	and `allocated_objects_size`); for metrics by the OS this usually is
	`proportional_resident_size` (others are `peak_resident_size` and
	`private_dirty_size`).

	Read the [memory-infra documentation][memory-infra] for more details on them.

	[memory-infra]: /docs/memory-infra/README.md