tast: Diagnose kernel thread hung up.

When kernel thread hung up, kernel leaves logs in the following format:

[ 6718.241446] INFO: task kswapd0:32 blocked for more than 120 seconds.
[ 6718.241452]       Not tainted 4.4.178-16473-g2e7bd078f16a #1
[ 6718.241455] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 6718.241457] kswapd0         D 9c661abcf5c83990     0    32      2 0x00000000
[ 6718.241464]  ffff8801659b52c0 ffff88017ed15900 ffff8800766cbb00 ffff88017983e740
[ 6718.241470]  0000000000000000 ffff88017904b9d8 ffffffffa0ea988d 0000000000000000
[ 6718.241476]  0000000000000286 9c661abcf5c83990 ffff88017983e740 ffff88017a11c0c0
[ 6718.241482] Call Trace:
[ 6718.241490]  [<ffffffffa0ea988d>] ? __schedule+0x3c4/0x868
[ 6718.241494]  [<ffffffffa0ea9490>] schedule+0x40/0x79
[ 6718.241499]  [<ffffffffa08c2fd1>] jbd2_log_wait_commit+0xb9/0x13c
[ 6718.241503]  [<ffffffffa079d551>] ? prepare_to_wait_event+0xd4/0xd4
[ 6718.241507]  [<ffffffffa08c36f8>] jbd2_complete_transaction+0x8a/0x9f
[ 6718.241510]  [<ffffffffa0879f89>] ext4_evict_inode+0x1f5/0x51b
[ 6718.241514]  [<ffffffffa081bfb6>] dispose_list+0xdf/0x1f3
[ 6718.241517]  [<ffffffffa081c272>] prune_icache_sb+0x56/0x76
[ 6718.241522]  [<ffffffffa080cee8>] super_cache_scan+0x139/0x184
[ 6718.241525]  [<ffffffffa07ef52f>] shrink_slab+0x3c2/0x4e8
[ 6718.241529]  [<ffffffffa07f0d64>] shrink_zone+0x6a6/0x786
[ 6718.241532]  [<ffffffffa0eae800>] ? return_from_execve+0xb/0x23
[ 6718.241536]  [<ffffffffa07b9d31>] kswapd+0x71c/0x934
[ 6718.241543]  [<ffffffffa06e8318>] kthread+0xf7/0xff
[ 6718.241545]  [<ffffffffa0eae8a4>] ? __switch_to_asm+0x34/0x70
[ 6718.241549]  [<ffffffffa07b9615>] ? kswapd_run+0x9b/0x9b
[ 6718.241552]  [<ffffffffa06e8221>] ? flush_kthread_worker+0x91/0x91
[ 6718.241554]  [<ffffffffa0eae92e>] ret_from_fork+0x4e/0x80
[ 6718.241558]  [<ffffffffa06e8221>] ? flush_kthread_worker+0x91/0x91
...
[ 6718.247108] Kernel panic - not syncing: hung_task: blocked tasks

On seeing this log, scrape the kernel thread name (kswapd0:32) and
the innermost non-scheduleing stack frame (jbd2_log_wait_commit)
to build an informative error message.

Also put real ramoops files to testdata directory with links to
original files.

BUG=chromium:930239
TEST=fast_build.sh -T

Change-Id: Ifac792fad5f605e1aa0303628e221449ec892636
Reviewed-on: https://chromium-review.googlesource.com/1575187
Commit-Ready: Shuhei Takahashi <nya@chromium.org>
Tested-by: Shuhei Takahashi <nya@chromium.org>
Reviewed-by: Dan Erat <derat@chromium.org>
5 files changed
tree: 5aa0b214c46d31855526193f144c7446e3d16c7c
  1. docs/
  2. src/
  3. tools/
  4. .gitignore
  5. fast_build.sh
  6. OWNERS
  7. PRESUBMIT.cfg
  8. README.md
README.md

Tast (go/tast)

Tast logo with Go gopher with green rubber PASS stamp

Tast is an integration-testing system for Chrome OS. Its focus is on maintainability, speed, and ease of interpreting and reproducing test results. It supports building, deploying, and running tests. It doesn't implement other functionality like managing labs of devices used for testing, scheduling tests, or storing test results.

To try it out, see the quickstart document. The overview is a good starting point for learning more about how the system is structured.

Directory structure

This repository is organized in accordance with the Go in Chromium OS suggestions.

  • src/chromiumos/
    • cmd/local_test_runner/ - main package for the local_test_runner executable used to execute local test bundles on-device.
    • cmd/remote_test_runner/ - main package for the remote_test_runner executable used to execute remote test bundles on the host system.
    • cmd/tast/ - main package for the tast executable used to build and run tests.
      • ... - Packages used only by the tast executable.
    • tast/... - Common packages shared by multiple executables.

Tests are located in the tast-tests repository.

Documentation

For more information, see the list of documentation.

GoDoc

Mailing list

The tast-users mailing list is used for discussions about writing and running tests.

Bug reports

Please file bug reports and feature requests in the Chromium issue tracker using the Tests>Tast component.

Other notes

The Go gopher concept and design was originally created by Renee French and is covered by the Creative Commons Attribution 3.0 license. The Tast logo above is also covered by the Creative Commons Attribution 3.0 license.