To investigate why a Render Test is failing on the trybots:
results_details
link under the chrome_public_test_apk
step to go to the Suites Summary page.<test class>.<render id>.<device details>.png
.Now you will see a Render Results page, showing:
At this point, decide whether the UI change was intentional. If it was, follow the steps below to update the golden files stored in the repository. If not, go and fix your code! If there's some other error or flakiness, file a bug to peconn@chromium.org
.
Link to Golden
link to determine where in the repository the golden was stored.Download Failure Image
link to save the failure image in the appropriate place in your local repository.//chrome/test/data/android/manage_render_test_goldens.py upload
to upload the new goldens to Google Storage and update the hashes used to download them.When putting a change up for review that changes goldens, please include links to the results_details/Render Results pages that you grabbed the new goldens from. This will help reviewers confirm that the changes to the goldens are acceptable.
If you add a new device/SDK combination that you expect golden images for, be sure to add it to ALLOWED_DEVICE_SDK_COMBINATIONS
in //chrome/test/data/android/manage_render_test_goldens.py
, otherwise the goldens for it will not be uploaded.
Follow the steps in Running the tests locally below to generate renders.
You can rename the renders as appropriate and move them to the correct place in the repository, or you can open the locally generated Render Results pages and follow steps 2-3 in the second part of the Failing on trybots section.
To write a new test, start with the example in the javadoc for RenderTestRule.
When running instrumentation tests locally, pass the --local-output
option to the test runner to generate a folder in your output directory (in the example out/Debug
) looking like TEST_RESULTS_2017_11_09T13_50_49
containing the failed renders, eg:
./out/Debug/bin/run_chrome_public_test_apk -A Feature=RenderTest --local-output
The golden images should be downloaded as part of the gclient sync
process, but if there appear to be goldens missing that should be there, try running //chrome/test/data/android/manage_render_test_goldens.py download
to ensure that the downloaded goldens are current for the git revision.
New golden images may be downloaded from the trybots or retrieved locally. This section elaborates how to do the latter.
You should always create your reference images on the same device type as the one running the tests. This is because each device/API version may produce a slightly different image, eg. due to different screen dimensions, DPI setting, or styling used across OS versions. This is also why each golden image name includes the device name and API version.
When running a test with no goldens on the correct device, your tests should fail with an exception:
RenderTest Goldens missing for: <reference>. See RENDER_TESTS.md for how to fix this failure.
You will be able to find the images the device captured on the device's SD card.
adb -d shell ls /sdcard/chromium_tests_root/chrome/test/data/android/render_tests/failures
How a View is rendered depends on both the device model and the version of Android it is running. We only want to maintain golden files for model/SDK pairs that occur on the trybots, otherwise the golden files will get out of date as changes occur and render tests will either fail on the Testers with no warning, or be useless.
Currently, chrome_public_test_apk
is only run on Nexus 5s running Android Lollipop, so that is the only model/sdk combination for which we store goldens. There is work to expand this to include Nexus 5Xs running Marshmallow as well.
Certain features lead to flaky tests, for example any sort of animation we don't take into account while writing the tests. To help deal with this, you can use RenderTestRule.sanitize
to modify the View hierarchy and remove some of the more troublesome attributes (for example, it disables the blinking cursor in EditText
s).