docs/testing/chromeos_debugging_tips.md - chromium/src - Git at Google


            
            # Chrome OS Debugging Instructions

            
            Chrome on Chrome OS is tested using a handful of frameworks, each of which

            
            you'll find running on Chrome's CQ and waterfalls. If you're investigating

            
            failures in these tests, below are some tips for debugging and identifying the

            
            cause.


            *** note


            This doc outlines tests running in true Chrome OS environments (ie: on virtual

            
            machines or real devices). [linux-chromeos] tests, on the other hand, can be

            
            debugged like any other linux test.

            
            ***


            ## Tast


            [Tast] is Chrome OS's integration testing framework. Since Chrome itself is

            
            instrumental to the Chrome OS system, it's equally important that we run some

            
            of these integration tests on Chrome's waterfalls. If you find one of these

            
            tests failing (likely in the `chrome_all_tast_tests` step), you can:


            - **Inspect the failed test's log snippet**: There should be a log snippet for

            
            each failed test in the `Test Results` tab in the build UI. eg: For this

            
            [failed build], clicking on the `policy.IncognitoModeAvailability` expands to

            
            include stack traces and error messages.


            - **View browser & system logs**: A common cause of failure on Chrome's builders

            
            are browser crashes. When this happens, each test's log snippets will simply

            
            contain warnings like "[Chrome probably crashed]". To debug these crashes,

            
            expand the list of attached artifacts for the test by clicking the `Artifacts`

            
            link under the failed test in the `Test Results` tab. There you'll find an

            
            extended log for the test under `log.txt`. Additionally, you can find system

            
            logs included in that list. To find a system log for a particular test, match

            
            the timestamps printed in the test's log with the timestamps present in the

            
            system log filename. For instance, the previous `example.ChromeFixture` failure

            
            matches the [chrome/chrome_20210920-051805] browser log, which contains the

            
            culprit Chrome crash and backtrace.


            - **Symbolizing a browser crash dump**: See [below](#symbolizing-a-crash-dump).


            ### Disabling a test


            If you are a Chrome Sheriff, please read the sheriff documentation

            
            [here](http://go/chrome-sheriff-tast) before disabling any tests.


            Tast tests are run under both Chrome's builders and CrOS's builders. They can be

            
            disabled either completely (in all builders), or in Chrome's builders alone. The

            
            latter should be used only for changes which are not expected to occur on CrOS's

            
            builders.


            - **Disabling in all builders**: If you have a full CrOS checkout, you can add

            
            the `informational` [attribute] to the test's definition. (You may be able to

            
            bypass the need for a full CrOS checkout by using the `Edit code` button in

            
            codesearch UI, but this flow is unverified.) This can take time (ie: many hours)

            
            to land and propagate onto Chrome's builders. So if you need the test disabled

            
            ASAP, consult the next option.

            
            - **Disabling in only Chrome's builders**: You can add the test to the list of

            
            disabled tests for the step's GN target. For example, to disable a test in the

            
            `chrome_all_tast_tests` step, add it to [this list]. **Note**: If the test is

            
            failing consistently, and you only disable it here, it will likely start to fail

            
            in the next [Chrome uprev] on CrOS's builders, which can lead to further

            
            problems down the road. So please make sure you pursue the first option as well

            
            in that case.


            In both cases, please make sure a bug is filed for the test, and route it to

            
            the appropriate owners.


            ### Symbolizing a crash dump


            If a test fails due to a browser crash, there should be a Minidump crash report

            
            present in the test's isolated out under the prefix `crashes/chrome...`. These

            
            reports aren't very useful by themselves, but with a few commands you can

            
            symbolize the report locally to get insight into what conditions caused Chrome

            
            to crash.


            If you are running a locally compiled [Simple Chrome] binary on a device or VM,

            
            you can can build `minidump_stackwalk` and download the

            
            `/home/chronos/crash/chrome*.dmp` file.

            
            ```

            
            autoninja -C out/Release minidump_stackwalk dump_syms


            rsync -r -e "ssh -p 9222" root@localhost:/home/chronos/crash /tmp

            
            ```


            For a crash on a bot, download both the task's input files (this provides the

            
            symbols and the symbolizing tools) as well as the task's output results (this

            
            provides the crash reports). See the commands listed under the *Reproducing the

            
            task locally* section on the task page. For example, to download them for

            
            [this task](https://chrome-swarming.appspot.com/task?id=5cc272e0a839b311), `cd`

            
            into a tmp directory and run:

            
            ```

            
            cipd install "infra/tools/luci/cas/\${platform}" -root bar

            
            ./bar/cas login

            
            ./bar/cas download -cas-instance projects/chrome-swarming/instances/default_instance -digest 1ad29e201e4ae7e3056a8b17935edbcd62fb54befdfeba221f2e82e54f150c86/812 -dir foo


            cipd install "infra/tools/luci/swarming/\${platform}" -root bar

            
            ./bar/swarming login

            
            ./bar/swarming collect -S chrome-swarming.appspot.com -output-dir=foo 5cc272e0a839b311

            
            ```


            Generate the breakpad symbols by pointing the `generate_breakpad_symbols.py` script to

            
            your local binary, or the downloaded input build dir:

            
            ```

            
            cd foo

            
            vpython3 components/crash/content/tools/generate_breakpad_symbols.py --symbols-dir symbols --build-dir out/Release/ --binary out/Release/chrome --platform chromeos

            
            ```


            That will generate the symbols in the `symbols/` dir. Then to symbolize a Chrome

            
            crash report (either in the tasks's output, or the `/tmp/crash` dir):

            
            ```

            
            ./out/Release/minidump_stackwalk 5cc272e0a839b311/crashes/chrome.20220816.214251.44917.24579.dmp symbols/

            
            ```


            ### Running a test locally


            To run a Tast test the same way it's ran on Chrome's builders:


            - Decide which Chrome OS device type or VM to test on.


            - Build Chrome via the [Simple Chrome] workflow for that board.


            - Deploy your Chrome to the device via the [deploy_chrome.py] tool.


            - Finally, run the Tast test on the device via the `cros_run_test` tool under

            
              `//third_party/chromite/bin/`. eg:

            
              `cros_run_test --device $DEVICE_IP --tast login.Chrome`. See [here] for more

            
              info on cros_run_test.


            ## Telemetry


            >TODO: Add instructions for debugging telemetry failures.


            ## GTest


            >TODO: Add instructions for debugging GTest failures.


            ## Rerunning these tests locally


            >TODO: Add instructions for rerunning these tests locally.


            [linux-chromeos]: https://chromium.googlesource.com/chromium/src/+/HEAD/docs/chromeos_build_instructions.md

            
            [Tast]: https://chromium.googlesource.com/chromiumos/platform/tast/+/HEAD/README.md

            
            [failed build]: https://ci.chromium.org/ui/p/chromium/builders/ci/chromeos-kevin-rel/37300/test-results

            
            [Chrome probably crashed]: https://luci-milo.appspot.com/ui/inv/build-8835572137562508161/test-results?q=example.ChromeFixture

            
            [chrome/chrome_20210920-051805]: https://luci-milo.appspot.com/ui/artifact/raw/invocations/task-chromium-swarm.appspot.com-561bed66572a9411/artifacts/chrome%2Fchrome_20210920-051805

            
            [attribute]: https://chromium.googlesource.com/chromiumos/platform/tast/+/HEAD/docs/test_attributes.md

            
            [this list]: https://codesearch.chromium.org/chromium/src/chromeos/tast_control.gni

            
            [Chrome uprev]: https://chromium.googlesource.com/chromiumos/docs/+/HEAD/chrome_commit_pipeline.md#the-chrome-os-commit-pipeline-for-chrome-changes

            
            [Simple Chrome]: https://chromium.googlesource.com/chromiumos/docs/+/HEAD/simple_chrome_workflow.md

            
            [deploy_chrome.py]: https://chromium.googlesource.com/chromiumos/docs/+/HEAD/simple_chrome_workflow.md#Deploying-Chrome-to-the-device

            
            [here]: https://chromium.googlesource.com/chromiumos/docs/+/HEAD/cros_vm.md#in-simple-chrome
	# Chrome OS Debugging Instructions
	Chrome on Chrome OS is tested using a handful of frameworks, each of which
	you'll find running on Chrome's CQ and waterfalls. If you're investigating
	failures in these tests, below are some tips for debugging and identifying the
	cause.

	*** note

	This doc outlines tests running in true Chrome OS environments (ie: on virtual
	machines or real devices). [linux-chromeos] tests, on the other hand, can be
	debugged like any other linux test.
	***

	## Tast

	[Tast] is Chrome OS's integration testing framework. Since Chrome itself is
	instrumental to the Chrome OS system, it's equally important that we run some
	of these integration tests on Chrome's waterfalls. If you find one of these
	tests failing (likely in the `chrome_all_tast_tests` step), you can:

	- Inspect the failed test's log snippet: There should be a log snippet for
	each failed test in the `Test Results` tab in the build UI. eg: For this
	[failed build], clicking on the `policy.IncognitoModeAvailability` expands to
	include stack traces and error messages.

	- View browser & system logs: A common cause of failure on Chrome's builders
	are browser crashes. When this happens, each test's log snippets will simply
	contain warnings like "[Chrome probably crashed]". To debug these crashes,
	expand the list of attached artifacts for the test by clicking the `Artifacts`
	link under the failed test in the `Test Results` tab. There you'll find an
	extended log for the test under `log.txt`. Additionally, you can find system
	logs included in that list. To find a system log for a particular test, match
	the timestamps printed in the test's log with the timestamps present in the
	system log filename. For instance, the previous `example.ChromeFixture` failure
	matches the [chrome/chrome_20210920-051805] browser log, which contains the
	culprit Chrome crash and backtrace.

	- Symbolizing a browser crash dump: See [below](#symbolizing-a-crash-dump).

	### Disabling a test

	If you are a Chrome Sheriff, please read the sheriff documentation
	[here](http://go/chrome-sheriff-tast) before disabling any tests.

	Tast tests are run under both Chrome's builders and CrOS's builders. They can be
	disabled either completely (in all builders), or in Chrome's builders alone. The
	latter should be used only for changes which are not expected to occur on CrOS's
	builders.

	- Disabling in all builders: If you have a full CrOS checkout, you can add
	the `informational` [attribute] to the test's definition. (You may be able to
	bypass the need for a full CrOS checkout by using the `Edit code` button in
	codesearch UI, but this flow is unverified.) This can take time (ie: many hours)
	to land and propagate onto Chrome's builders. So if you need the test disabled
	ASAP, consult the next option.
	- Disabling in only Chrome's builders: You can add the test to the list of
	disabled tests for the step's GN target. For example, to disable a test in the
	`chrome_all_tast_tests` step, add it to [this list]. Note: If the test is
	failing consistently, and you only disable it here, it will likely start to fail
	in the next [Chrome uprev] on CrOS's builders, which can lead to further
	problems down the road. So please make sure you pursue the first option as well
	in that case.

	In both cases, please make sure a bug is filed for the test, and route it to
	the appropriate owners.

	### Symbolizing a crash dump

	If a test fails due to a browser crash, there should be a Minidump crash report
	present in the test's isolated out under the prefix `crashes/chrome...`. These
	reports aren't very useful by themselves, but with a few commands you can
	symbolize the report locally to get insight into what conditions caused Chrome
	to crash.

	If you are running a locally compiled [Simple Chrome] binary on a device or VM,
	you can can build `minidump_stackwalk` and download the
	`/home/chronos/crash/chrome*.dmp` file.
	```
	autoninja -C out/Release minidump_stackwalk dump_syms

	rsync -r -e "ssh -p 9222" root@localhost:/home/chronos/crash /tmp
	```

	For a crash on a bot, download both the task's input files (this provides the
	symbols and the symbolizing tools) as well as the task's output results (this
	provides the crash reports). See the commands listed under the *Reproducing the
	task locally* section on the task page. For example, to download them for
	[this task](https://chrome-swarming.appspot.com/task?id=5cc272e0a839b311), `cd`
	into a tmp directory and run:
	```
	cipd install "infra/tools/luci/cas/\${platform}" -root bar
	./bar/cas login
	./bar/cas download -cas-instance projects/chrome-swarming/instances/default_instance -digest 1ad29e201e4ae7e3056a8b17935edbcd62fb54befdfeba221f2e82e54f150c86/812 -dir foo

	cipd install "infra/tools/luci/swarming/\${platform}" -root bar
	./bar/swarming login
	./bar/swarming collect -S chrome-swarming.appspot.com -output-dir=foo 5cc272e0a839b311
	```

	Generate the breakpad symbols by pointing the `generate_breakpad_symbols.py` script to
	your local binary, or the downloaded input build dir:
	```
	cd foo
	vpython3 components/crash/content/tools/generate_breakpad_symbols.py --symbols-dir symbols --build-dir out/Release/ --binary out/Release/chrome --platform chromeos
	```

	That will generate the symbols in the `symbols/` dir. Then to symbolize a Chrome
	crash report (either in the tasks's output, or the `/tmp/crash` dir):
	```
	./out/Release/minidump_stackwalk 5cc272e0a839b311/crashes/chrome.20220816.214251.44917.24579.dmp symbols/
	```


	### Running a test locally

	To run a Tast test the same way it's ran on Chrome's builders:

	- Decide which Chrome OS device type or VM to test on.

	- Build Chrome via the [Simple Chrome] workflow for that board.

	- Deploy your Chrome to the device via the [deploy_chrome.py] tool.

	- Finally, run the Tast test on the device via the `cros_run_test` tool under
	`//third_party/chromite/bin/`. eg:
	`cros_run_test --device $DEVICE_IP --tast login.Chrome`. See [here] for more
	info on cros_run_test.

	## Telemetry

	>TODO: Add instructions for debugging telemetry failures.

	## GTest

	>TODO: Add instructions for debugging GTest failures.

	## Rerunning these tests locally

	>TODO: Add instructions for rerunning these tests locally.


	[linux-chromeos]: https://chromium.googlesource.com/chromium/src/+/HEAD/docs/chromeos_build_instructions.md
	[Tast]: https://chromium.googlesource.com/chromiumos/platform/tast/+/HEAD/README.md
	[failed build]: https://ci.chromium.org/ui/p/chromium/builders/ci/chromeos-kevin-rel/37300/test-results
	[Chrome probably crashed]: https://luci-milo.appspot.com/ui/inv/build-8835572137562508161/test-results?q=example.ChromeFixture
	[chrome/chrome_20210920-051805]: https://luci-milo.appspot.com/ui/artifact/raw/invocations/task-chromium-swarm.appspot.com-561bed66572a9411/artifacts/chrome%2Fchrome_20210920-051805
	[attribute]: https://chromium.googlesource.com/chromiumos/platform/tast/+/HEAD/docs/test_attributes.md
	[this list]: https://codesearch.chromium.org/chromium/src/chromeos/tast_control.gni
	[Chrome uprev]: https://chromium.googlesource.com/chromiumos/docs/+/HEAD/chrome_commit_pipeline.md#the-chrome-os-commit-pipeline-for-chrome-changes
	[Simple Chrome]: https://chromium.googlesource.com/chromiumos/docs/+/HEAD/simple_chrome_workflow.md
	[deploy_chrome.py]: https://chromium.googlesource.com/chromiumos/docs/+/HEAD/simple_chrome_workflow.md#Deploying-Chrome-to-the-device
	[here]: https://chromium.googlesource.com/chromiumos/docs/+/HEAD/cros_vm.md#in-simple-chrome