docs/testing/how_to_repro_bot_failures.md - chromium/src - Git at Google

 # How to repro bot failures

 If you're looking for repro a CI/CQ compile or test bot failures locally, then
 this is the right doc. This doc tends to getting you to the exact same
 environment as bot for repro. Most likely you don't need to follow the exact
 steps here. But if you have a hard time for repro, then it's good to check each
 steps. Also keep in mind that even you have the exact same environment, some
 failures are not reproduceable locally.

 When trying to repro and fix a bot failure, the easier approach is tryjobs, or
 [debug with swarming](../workflow/debugging-with-swarming.md).
 This doc is more useful when it requires more local debugging and testing.

 This doc assumes you're familiar with Chrome development on at least 1 platform.

 ## Identify the platform

 Usually this is easy by looking at the builder name. e.g. linux-rel means it's
 on linux platform.

 ## Understand the platform system requirements

 Usually you should be able to follow
 [Chromium Get Code](https://www.chromium.org/developers/how-tos/get-the-code/)
 and understand the platform.

 ## Prepare the compile and test devices

 For compile devices, if you see a "compilator steps"(usually for CQ jobs),

 ![compilator step]

 then
 click the "compilator build" and then bot on the right.

 ![bot link]

 Otherwise just click bot on right.

 For test devices, for most builders, click the shard and then click on bot.

 ![test shard]

 The bot page would look like this

 ![bot page]

 You would want to prepare the exact same environment when possible. e.g.
 many tests are using e2-standard-8 GCE bot, you will want to create a
 e2-standard-8 GCE VM. If you see python 3.8, then install python 3.8.

 ## Checkout code

 Follow the platform specific workflow to checkout the code.

 ## Revision

 On bot, you can see revision like this.

 ![revision]

 First sync to that revision. For tryjobs, you also need to cherry-pick the
 changelist with correct patchset after sync to correct revision.

 Note: Don't just download a patch from gerrit to repro tryjob failures.
 On bot, it will first checkout the branch HEAD, then patch the patchset.
 But on gerrit, what you see is your local revision when you upload with
 the patchset. On gerrit you see code at an older base revision.

 ## Change gclient config
 On bot, there is a step named 'gclient config'.

 ![gclient]

 Apply the same gclient config and then rerun 'gclient sync'.

 ## gn args

 On bot, there is a step named 'lookup GN args'.

 ![gn args]

 Apply the same gn args locally.
 Note: most of the time, some path based flags can be omitted. Examples:
 * coverage_instrumentation_input_file

 At the time of writing, some gn args can't be applied locally, like
 use_remoteexec.

 Note: ChromeOS builders(or any platform that involves 'import' gn rules)
 may show complicated gn args. For these builders, you need to search the
 builder name in //tools/mb or //internal/tools/mb for internal builders to
 find out the gn args.

 ## Compile

 On bot, there is a step named 'compile'. Go to 'execution_details' and compile
 all the targets listed in there.

 ## Run test

 Most of the time, on bot the compile device is not the same as the test device.
 But locally for desktop platforms, they are the same device and in the same
 //out folder. There is no easy way to mimic bot behavior. A common issue on bot
 is you see some missing files but can't repro locally. This is because on bot
 we calculate what files are needed for testing using build maps and copy them
 from compile machine to test machine. But locally you don't have this step.
 Usually the fix is to add some 'data' or 'data_deps' in BUILD.gn.

 Open the bot test result page

 ![test page]

 You should see all the information you need including system environment and
 commandlines.

 Note: Set system environment. Especially important for gtest GTEST_SHARD_INDEX
 GTEST_TOTAL_SHARDS.

 Note: On bot, the CURRENT_DIR is in the //out dir. Say you compile test target
 in //out/Default, then enter //out/Default and then run tests.

 Note: For gtest, if you're not using the same machine type, you might want to
 override some flags like --test-launcher-jobs.

 [bot link]: images/bot_repro_bot_link.png
 [bot page]: images/bot_repro_bot_page.png
 [compilator step]: images/bot_repro_compilator_step.png
 [gclient]: images/bot_repro_gclient.png
 [gn args]: images/bot_repro_gn_args.png
 [revision]: images/bot_repro_revision.png
 [test page]: images/bot_repro_test_page.png
 [test shard]: images/bot_repro_test_shard.png
	# How to repro bot failures

	If you're looking for repro a CI/CQ compile or test bot failures locally, then
	this is the right doc. This doc tends to getting you to the exact same
	environment as bot for repro. Most likely you don't need to follow the exact
	steps here. But if you have a hard time for repro, then it's good to check each
	steps. Also keep in mind that even you have the exact same environment, some
	failures are not reproduceable locally.

	When trying to repro and fix a bot failure, the easier approach is tryjobs, or
	[debug with swarming](../workflow/debugging-with-swarming.md).
	This doc is more useful when it requires more local debugging and testing.

	This doc assumes you're familiar with Chrome development on at least 1 platform.

	## Identify the platform

	Usually this is easy by looking at the builder name. e.g. linux-rel means it's
	on linux platform.

	## Understand the platform system requirements

	Usually you should be able to follow
	[Chromium Get Code](https://www.chromium.org/developers/how-tos/get-the-code/)
	and understand the platform.

	## Prepare the compile and test devices

	For compile devices, if you see a "compilator steps"(usually for CQ jobs),

	![compilator step]

	then
	click the "compilator build" and then bot on the right.

	![bot link]

	Otherwise just click bot on right.

	For test devices, for most builders, click the shard and then click on bot.

	![test shard]

	The bot page would look like this

	![bot page]

	You would want to prepare the exact same environment when possible. e.g.
	many tests are using e2-standard-8 GCE bot, you will want to create a
	e2-standard-8 GCE VM. If you see python 3.8, then install python 3.8.

	## Checkout code

	Follow the platform specific workflow to checkout the code.

	## Revision

	On bot, you can see revision like this.

	![revision]

	First sync to that revision. For tryjobs, you also need to cherry-pick the
	changelist with correct patchset after sync to correct revision.

	Note: Don't just download a patch from gerrit to repro tryjob failures.
	On bot, it will first checkout the branch HEAD, then patch the patchset.
	But on gerrit, what you see is your local revision when you upload with
	the patchset. On gerrit you see code at an older base revision.

	## Change gclient config
	On bot, there is a step named 'gclient config'.

	![gclient]

	Apply the same gclient config and then rerun 'gclient sync'.

	## gn args

	On bot, there is a step named 'lookup GN args'.

	![gn args]

	Apply the same gn args locally.
	Note: most of the time, some path based flags can be omitted. Examples:
	* coverage_instrumentation_input_file

	At the time of writing, some gn args can't be applied locally, like
	use_remoteexec.

	Note: ChromeOS builders(or any platform that involves 'import' gn rules)
	may show complicated gn args. For these builders, you need to search the
	builder name in //tools/mb or //internal/tools/mb for internal builders to
	find out the gn args.

	## Compile

	On bot, there is a step named 'compile'. Go to 'execution_details' and compile
	all the targets listed in there.

	## Run test

	Most of the time, on bot the compile device is not the same as the test device.
	But locally for desktop platforms, they are the same device and in the same
	//out folder. There is no easy way to mimic bot behavior. A common issue on bot
	is you see some missing files but can't repro locally. This is because on bot
	we calculate what files are needed for testing using build maps and copy them
	from compile machine to test machine. But locally you don't have this step.
	Usually the fix is to add some 'data' or 'data_deps' in BUILD.gn.

	Open the bot test result page

	![test page]

	You should see all the information you need including system environment and
	commandlines.

	Note: Set system environment. Especially important for gtest GTEST_SHARD_INDEX
	GTEST_TOTAL_SHARDS.

	Note: On bot, the CURRENT_DIR is in the //out dir. Say you compile test target
	in //out/Default, then enter //out/Default and then run tests.

	Note: For gtest, if you're not using the same machine type, you might want to
	override some flags like --test-launcher-jobs.

	[bot link]: images/bot_repro_bot_link.png
	[bot page]: images/bot_repro_bot_page.png
	[compilator step]: images/bot_repro_compilator_step.png
	[gclient]: images/bot_repro_gclient.png
	[gn args]: images/bot_repro_gn_args.png
	[revision]: images/bot_repro_revision.png
	[test page]: images/bot_repro_test_page.png
	[test shard]: images/bot_repro_test_shard.png