Addressing Performance Regressions

The bisect bot just picked your CL as the culprit in a performance regression and assigned a bug to you! What should you do? Read on...

About our performance tests

The chromium.perf waterfall is a continuous build which runs performance tests on dozens of devices across Windows, Mac, Linux, and Android Chrome and WebView. Often, a performance regression only affects a certain type of hardware or a certain operating system, which may be different than what you tested locally before landing your CL.

Each test has an owner, named in this spreadsheet, who you can cc on a performance bug if you have questions.

Understanding the bisect results

The bug comment

The bisect service spits out a comment on the bug that looks like this:

📍 Found significant differences after each of 2 commits.
https://pinpoint-dot-chromeperf.appspot.com/job/148a8d4e840000

Add smcgruer as an animations OWNER by flackr@chromium.org
https://chromium.googlesource.com/chromium/src/+/b091c264862d26ac12d932e84eef7bd5f674e62b

Roll src/third_party/depot_tools/ 0f7b2007a..fd4ad2416 (1 commit) by depot-tools-roller@chromium.org
https://chromium.googlesource.com/chromium/src/+/14fc99e3fd3614096caab7c7a8362edde8327a5d

Understanding performance regressions:
  http://g.co/ChromePerformanceRegressions

The bug comment gives a summary of that commits that caused improvements or regressions. For more details, click the link at the beginning of the comment to go to the Pinpoint Job details page.

The Job details page

Clicking the Pinpoint link in the bug comment brings you to the Job details page.

Pinpoint Job page

Down the left you can see some details about the bisect configuration, including the benchmark (loading.desktop) and story (Pantip) that ran, the bot it ran on (chromium-rel-mac11-pro), and the metric that was measured (cpuTimeToFirstMeaningfulPaint). If you're not familiar with the benchmark or metric, you can cc the benchmark owner to ask for help.

The graph in the middle of the page shows a summary of the commits that were tested across the x-axis and their results on the y-axis. The dots show the medians, and the bars show the min and max. These can be used to estimate the size of the regression. The units are not available on this page, but are on the performance graphs linked on the bug in comment #1.

Click the + button in the bottom-right corner of the page to test a patch with the current configuration.

The alerts page

Comment 1 on the bug will have a link to the perf dashboard graphs for the regression. (https://chromeperf.appspot.com/group_report?bug_id=XXX)