This document describes benchmarks available to track Chrome‘s and WebView’s memory usage, where they live, what they measure, how to run them, and on how to diagnose regressions.
System health is an effort to unify top-level benchmarks (as opposite to micro-benchmarks and regression tests) that are suitable to capture representative user stories.
System health memory benchmarks are:
These benchmarks are run continuously on the chromium.perf waterfall, collecting and reporting results on the Chrome Performance Dashboard.
Other benchmarks maintained by the memory-infra team are discussed in the appendix.
System health user stories are classified by the kind of interactions they perform with the browser:
browse
stories navigate to a URL and interact with the page; e.g. scroll, click on elements, navigate to subpages, navigate back.load
stories just navigate to a URL and wait for the page to load.background
stories navigate to a URL, possibly interact with the page, and then bring another app to the foreground (thus pushing the browser to the background).long_running
stories interact with a page for a longer period of time (~5 mins).blank
has a single story that just navigates to about:blank.The full name of a story has the form {interaction}:{category}:{site}
where:
interaction
is one the labels given above;category
is used to group together sites with a similar purpose, e.g. news
, social
, tools
;site
is a short name identifying the website in which the story mostly takes place, e.g. cnn
, facebook
, gmail
.For example browse:news:cnn
and background:social:facebook
are two system health user stories.
Today, for most stories a garbage collection is forced at the end of the story and a memory dump is then triggered. Metrics report the values obtained from this single measurement.
To view data from one of the benchmarks on the Chrome Performance Dashboard you should select:
{interaction}_{category}
for system health stories.:
replaced by _
).If you are investigating a Perf dashboard alert and would like to see the details, you can click on any point of the graph. It gives you the commit range, buildbot output and a link to the trace file taken during the buildbot run. (More information about reading trace files here)
Benchmarks may be run on a local platform/device or remotely on a pinpoint try job.
Given a patch already uploaded to code review, try jobs provide a convenient way to evaluate its memory implications on devices or platforms which may not be immediately available to developers.
To start a try job go to the pinpoint website, click on the +
button to create a new job, and fill in the required details:
system_health.memory_mobile
or system_health.memory_desktop
as appropriate.--story-filter
option to only run stories that match the pattern.run_benchmark
. Of note, if you are interested in running a small but representative sample of system health stories you can pass --story-tag-filter health_check
.If you have more specific needs, or need to automate the creation of jobs, you can also consider using pinpoint_cli.
After building, e.g. ChromePublic.apk
, you can run a specific system health story with the command:
$SRC/tools/perf/run_benchmark run system_health.memory_mobile \ --browser android-chromium --story-filter load:search:google
This will run the story with a default of 3 repetitions and produce a results.html
file comparing results from this and any previous benchmark runs. In addition, you'll also get individual trace files for each story run by the benchmark. Note: by default only high level metrics are shown, you may need to tick the “Show all” check box in order to view some of the lower level memory metrics.
Other useful options for this command are:
--pageset-repeat [n]
- override the default number of repetitions--reset-results
- clear results from any previous benchmark runs in the results.html
file.--results-label [label]
- give meaningful names to your benchmark runs, this way it is easier to compare them.For WebView make sure to replace the system WebView on your device and use --browser android-webview
.
There is a large number of memory-infra metrics, breaking down usage attributed to different components and processes.
Most memory metrics have the form memory:{browser}:{processes}:{source}:{component}:{kind}
where:
chrome
or webview
.browser_process
, renderer_processess
, gpu_process
, or all_processess
.reported_by_chrome
or reported_by_os
skia
or sqlite
; details about a specific component, e.g. v8:heap
; or a class of memory as seen by the OS, e.g. system_memory:native_heap
or gpu_memory
. If reported by chrome, the metrics are gathered by MemoryDumpProvider
s, probes placed in the specific components' codebase. For example, in “memory:chrome:all_processes:reported_by_chrome:net:effective_size_avg,” the component is “net” which is Chrome's network stack and “reported_by_chrome” means that this metric is gathered via probes in the network stack.effective_size
(others are locked_size
and allocated_objects_size
); for metrics by the OS this usually is proportional_resident_size
(others are peak_resident_size
and private_dirty_size
).There are a few other benchmarks maintained by the memory-infra team. These also use the same set of metrics as system health, but have differences on the kind of stories that they run.
The memory.top_10_mobile benchmark is in the process of being deprecated in favor of system health benchmarks. This process, however, hasn't been finalized and currently they are still the reference benchmark used for decision making in the Android release process. Therefore, it is important to diagnose and fix regressions caught by this benchmark.
The benchmark's work flow is:
Cycle between:
Repeat for each of 10 pages without closing the browser.
Close the browser, re-open and repeat the full page set a total of 5 times.
Story groups are either foreground
or background
depending on the state of the browser at the time of measurement.
The main difference to watch out between this and system health benchmarks is that, since a single browser instance is kept open and shared by many individual stories, they are not independent of each other. In particular, do not use the --story-filter
argument when trying to reproduce regressions on these benchmarks, as doing so will affect the results.
Dual browser benchmarks are intended to assess the memory implications of shared resources between Chrome and WebView.
memory.dual_browser_test - cycle between doing Google searches on a WebView-based browser (a stand-in for the Google Search app) and loading pages on Chrome. Runs on Android devices only.
Story groups are either on_chrome
or on_webview
, indicating the browser in foreground at the moment when the memory measurement was made.
memory.long_running_dual_browser_test - same as above, but the test is run for 60 iterations keeping both browsers alive for the whole duration of the test and without forcing garbage collection. Intended as a last-resort net to catch memory leaks not apparent on shorter tests.