docs/linux/debugging_minidump.md - chromium/src - Git at Google


            
            # Linux Minidump Code Lab


            [TOC]


            ## About Minidumps


            Minidump is a file format for storing parts of a program's state for later

            
            inspection. [Microsoft's

            
            Documentation](https://docs.microsoft.com/en-us/windows/desktop/api/minidumpapiset/)

            
            defines the format though the [Rust

            
            Documentation](https://docs.rs/minidump/latest/minidump/format/index.html)

            
            is sometimes easier to navigate. The minidump implementation and tools used by

            
            Chrome are

            
            [Breakpad](https://chromium.googlesource.com/breakpad/breakpad/+/refs/heads/main/README.md)

            
            and

            
            [Crashpad](https://chromium.googlesource.com/crashpad/crashpad/+/main/README.md).

            
            However, the tools of interest here are from the Breakpad project.


            ## Create a Minidump


            When Chrome crashes it writes out a minidump file. The minidump file is written

            
            under the application product directory. On Linux this is

            
            `<XDG_CONFIG_HOME>/<app-name>/Crash Reports`. The default for `XDG_CONFIG_HOME`

            
            is `~.config`. Common `<app-name>`s are `chromium`, `google-chrome`,

            
            `google-chrome-beta`, and `google-chrome-unstable`. A typical example is

            
            `~.config/google-chrome/Crash Reports`. When a minidump is uploaded it will be

            
            moved between the `new`, `pending`, and `completed` subdirectories. The minidump

            
            file is named something like `<uuid>.dmp`. If the minidump is uploaded to the

            
            crash reporting system, the `<uuid>.meta` file will contain the crash report id.

            
            Those with access can find the uploaded report at `go/crash/<report-id>`, where

            
            the minidump file will be available with a name like

            
            `upload_file_minidump-<report-id>.dmp`.


            To create a minidump, you can use a local build of Chromium or a release

            
            version of Chrome. Run the browser with the environment variable

            
            `CHROME_HEADLESS=1`, which enables crash dumping but prevents crash dumps from

            
            being uploaded and deleted. Something like `$ env CHROME_HEADLESS=1

            
            ./out/debug/chrome-wrapper` or `$ env CHROME_HEADLESS=1

            
            /opt/google/chrome/google-chrome`. Navigate to `chrome://crash` to trigger a

            
            crash in the renderer process or reproduce your current crash bug. A crash dump

            
            file should appear in the `Crash Reports` directory.


            ## Inspect the Minidump


            To get an idea about what is in a minidump file, install the Okteta hex editor

            
            and add the [Minidump Structure

            
            Definition](https://github.com/bungeman/structures/tree/main/okteta-minidump).

            
            Open the minidump previously created and explore the information it contains.


            One quirk to notice is that there is a `ThreadListStream` which contains

            
            `MINIDUMP_THREAD`s which contain a `MINIDUMP_THREAD_CONTEXT` and an

            
            `ExceptionStream` which also contains a `MINIDUMP_THREAD_CONTEXT`. The thread

            
            list contains the thread contexts as they existed when the crash reporter was

            
            running. The exception's thread context is the state of the crashing thread at

            
            the time that it crashed, which is generally the most interesting thread

            
            context. When using the Breakpad tools for Linux (like `minidump_stackwalk` and

            
            `minidump-2-core`) the thread context from the exception record is used in place

            
            of the thread context associated with the corresponding thread.


            Each `MINIDUMP_THREAD` contains a `StackMemoryRva` which is a reference to to a

            
            copy of the stack on that thread at the time the crash handler was running.

            
            Parsing a stack usefully requires additional debug information.

            
            `minidump_stackwalk` or a debugger may be used to parse the stack memory to

            
            create a usable trace.


            ## Get the Tools


            From a Chromium checkout `ninja -C out/release minidump-2-core

            
            minidump_stackwalk dump_syms`. From a [Breakpad

            
            checkout](https://chromium.googlesource.com/breakpad/breakpad/) `make`. It can

            
            be useful to use Breakpad directly on machines where one does not already have

            
            a Chromium checkout.


            When working at this level, one will also want to have `readelf` and

            
            `objdump` available, which are available from most distributions.


            ## Get Executables and Symbols


            In addition to the minidump, you will need the exact executables of Chromium or

            
            Chrome which produced the minidump and those executable's symbols. If the

            
            minidump was created locally, you already have the executables. Symbols for

            
            Google Chrome's official builds are available from

            
            `https://edgedl.me.gvt1.com/chrome/linux/symbols/google-chrome-debug-info-linux64-${VERSION}.zip`

            
            where `${VERSION}` is any version of Google Chrome that has recently been served

            
            to Stable, Beta, or Unstable (Dev) channels on Linux, like `114.0.5696.0`. Those

            
            with access can find both executables and symbols for unreleased builds at

            
            `go/chrome-symbols`.


            For symbols outside of Chrome (like when the crash is happening in a shared

            
            library) then symbols for the files of interest must be found. If the minidump

            
            was created locally then install the symbol packages from your distribution.  If

            
            not, you will need to track down the exact symbol files, which can be an

            
            interesting exercise. For some distributions using the

            
            [debuginfod](https://sourceware.org/elfutils/Debuginfod.html) system can be

            
            quite helpful.


            To ensure the correct binaries and debug symbols are used, the minidump contains

            
            the build-id for each loaded module in the `ModuleListStream` in the

            
            `CvRecordRva`'s `Signature`. This build-id is matched against a note

            
            section of type `NT_GNU_BUILD_ID`, usually named `.note.gnu.build-id` in the

            
            executable and symbol files. This note can be inspected with `readelf -n

            
            <file>` like `readelf -n chrome` or `readelf -n chrome.debug` and looking for

            
            the `.note.gnu.build-id` section. `readelf` reports the `Build ID` as the flat

            
            bytes in the note, but Breakpad binaries like `stackwalk_minidimp` and

            
            `dump_syms` will report and expect this truncated to a formatted Type 2 GUID

            
            (without dashes). This means `readelf` will output a `<build-id>` like

            
            33221100554477668899AABBCCDDEEFFXXXXXXXX but crashpad binaries will expect and

            
            report this as a `<build-uuid>` of 00112233445566778899AABBCCDDEEFF.


            The `.gnu_debuglink` section states which debug symbol file to use with a

            
            striped binary. For example `readelf --string-dump=.gnu_debuglink chrome`

            
            produces `chrome.debug`. This can be helpful to know for libraries with

            
            interesting debug symbol setup, like libc.so.6.


            ## Create Symbolized Stack


            Given a minidump with the name `mini.dmp`


            `minidump_stackwalk mini.dmp > mini.stackwalk.nosym`


            This will produce a mostly unsymbolized summary of the crash. To symbolize, look

            
            toward the bottom of the output for `WARNING: No symbols, <file>, <build-uuid>`.

            
            For each `<file>` which is of interest, `mkdir -p symbols/<file>/<build-uuid>` then

            
            `dump_syms <file> <directory-with-file.debug> >

            
            symbols/<file>/<build-uuid>/<file>.sym`. Ensure this output `<file>.sym`

            
            contains the expected `<build-uuid>`. Then re-run `minidump_stackwalk` but

            
            with the symbols directory, like `minidump_stackwalk mini.dmp symbols/ >

            
            mini.stackwalk`.


            The output of `minidump_stackwalk` is often quite useful and enough to track

            
            down many issues. However, it does not fully use all of the information from

            
            DWARF, so it is possible sometimes to get much better stack traces from a full

            
            debugger like gdb. This is particularly true when functions have been

            
            aggressively inlined.


            ## Create Core File


            `minidump-2-core mini.dmp > mini.core`


            ## Loading into GDB


            This works best if the binaries, symbols, and core files are all in different

            
            directories to prevent gdb from automatically loading them into the wrong

            
            locations. This is also generally necessary when using a system installed

            
            version of Chrome. For full details see [the gdb

            
            manual](https://sourceware.org/gdb/onlinedocs/gdb/Separate-Debug-Files.html).

            
            The easiest way is to rename and move the .debug files to a directory structure

            
            like `<debugdir>/debug/.build-id/nn/nnnnnnnn.debug` where `nn` are the first

            
            two hex characters of the `build-id`, and `nnnnnnnn` are the rest of the hex

            
            characters of the `build-id`. Note that this `build-id` is exactly what is

            
            reported by `readelf -n <binary> | grep "Build ID"` and not the `build-uuid`

            
            used by Breakpad. Then in gdb use `show debug-file-directory` to get the

            
            `<previous-directories>` and `set debug-file-directory

            
            <previous-directories>:<debugdir>/debug`.


            The `offset`s used here are the offsets of the corresponding module from the

            
            output of `minidump_stackwalk` or (equivalently) the value of

            
            `ModuleListStream::Modules[]::BaseOfImage` from the minidump file (which can be

            
            read with the structure definition).


            ```

            
            $ gdb

            
            (gdb) file <executable>

            
            (gdb) show debug-file-directory

            
            <previous directories>

            
            (gdb) set debug-file-directory <previous directories>:<debugdir>/debug

            
            (gdb) symbol-file <executable> -o <executable-offset>

            
            (gdb) core-file <mini.core>

            
            ```


            Running the commands in this order avoids needing to load the symbols twice and

            
            maps the `<executable>` to the expected location.


            To add an additional shared library it is possible to

            
            `(gdb) add-symbol-file <shared-library> -o <shared-library-offset>`


            Source paths in Chrome builds are relative to the `out/<build>` directory. If you

            
            have a Chromium checkout at or around when the Chrome build was created, it can

            
            be added to the debugger search path, like


            ```

            
            (gdb) directory <path-to-chromium>/chromium/src/out/<build>/

            
            ```
	# Linux Minidump Code Lab

	[TOC]

	## About Minidumps

	Minidump is a file format for storing parts of a program's state for later
	inspection. [Microsoft's
	Documentation](https://docs.microsoft.com/en-us/windows/desktop/api/minidumpapiset/)
	defines the format though the [Rust
	Documentation](https://docs.rs/minidump/latest/minidump/format/index.html)
	is sometimes easier to navigate. The minidump implementation and tools used by
	Chrome are
	[Breakpad](https://chromium.googlesource.com/breakpad/breakpad/+/refs/heads/main/README.md)
	and
	[Crashpad](https://chromium.googlesource.com/crashpad/crashpad/+/main/README.md).
	However, the tools of interest here are from the Breakpad project.

	## Create a Minidump

	When Chrome crashes it writes out a minidump file. The minidump file is written
	under the application product directory. On Linux this is
	`<XDG_CONFIG_HOME>/<app-name>/Crash Reports`. The default for `XDG_CONFIG_HOME`
	is `~.config`. Common `<app-name>`s are `chromium`, `google-chrome`,
	`google-chrome-beta`, and `google-chrome-unstable`. A typical example is
	`~.config/google-chrome/Crash Reports`. When a minidump is uploaded it will be
	moved between the `new`, `pending`, and `completed` subdirectories. The minidump
	file is named something like `<uuid>.dmp`. If the minidump is uploaded to the
	crash reporting system, the `<uuid>.meta` file will contain the crash report id.
	Those with access can find the uploaded report at `go/crash/<report-id>`, where
	the minidump file will be available with a name like
	`upload_file_minidump-<report-id>.dmp`.

	To create a minidump, you can use a local build of Chromium or a release
	version of Chrome. Run the browser with the environment variable
	`CHROME_HEADLESS=1`, which enables crash dumping but prevents crash dumps from
	being uploaded and deleted. Something like `$ env CHROME_HEADLESS=1
	./out/debug/chrome-wrapper` or `$ env CHROME_HEADLESS=1
	/opt/google/chrome/google-chrome`. Navigate to `chrome://crash` to trigger a
	crash in the renderer process or reproduce your current crash bug. A crash dump
	file should appear in the `Crash Reports` directory.

	## Inspect the Minidump

	To get an idea about what is in a minidump file, install the Okteta hex editor
	and add the [Minidump Structure
	Definition](https://github.com/bungeman/structures/tree/main/okteta-minidump).
	Open the minidump previously created and explore the information it contains.

	One quirk to notice is that there is a `ThreadListStream` which contains
	`MINIDUMP_THREAD`s which contain a `MINIDUMP_THREAD_CONTEXT` and an
	`ExceptionStream` which also contains a `MINIDUMP_THREAD_CONTEXT`. The thread
	list contains the thread contexts as they existed when the crash reporter was
	running. The exception's thread context is the state of the crashing thread at
	the time that it crashed, which is generally the most interesting thread
	context. When using the Breakpad tools for Linux (like `minidump_stackwalk` and
	`minidump-2-core`) the thread context from the exception record is used in place
	of the thread context associated with the corresponding thread.

	Each `MINIDUMP_THREAD` contains a `StackMemoryRva` which is a reference to to a
	copy of the stack on that thread at the time the crash handler was running.
	Parsing a stack usefully requires additional debug information.
	`minidump_stackwalk` or a debugger may be used to parse the stack memory to
	create a usable trace.

	## Get the Tools

	From a Chromium checkout `ninja -C out/release minidump-2-core
	minidump_stackwalk dump_syms`. From a [Breakpad
	checkout](https://chromium.googlesource.com/breakpad/breakpad/) `make`. It can
	be useful to use Breakpad directly on machines where one does not already have
	a Chromium checkout.

	When working at this level, one will also want to have `readelf` and
	`objdump` available, which are available from most distributions.

	## Get Executables and Symbols

	In addition to the minidump, you will need the exact executables of Chromium or
	Chrome which produced the minidump and those executable's symbols. If the
	minidump was created locally, you already have the executables. Symbols for
	Google Chrome's official builds are available from
	`https://edgedl.me.gvt1.com/chrome/linux/symbols/google-chrome-debug-info-linux64-${VERSION}.zip`
	where `${VERSION}` is any version of Google Chrome that has recently been served
	to Stable, Beta, or Unstable (Dev) channels on Linux, like `114.0.5696.0`. Those
	with access can find both executables and symbols for unreleased builds at
	`go/chrome-symbols`.

	For symbols outside of Chrome (like when the crash is happening in a shared
	library) then symbols for the files of interest must be found. If the minidump
	was created locally then install the symbol packages from your distribution. If
	not, you will need to track down the exact symbol files, which can be an
	interesting exercise. For some distributions using the
	[debuginfod](https://sourceware.org/elfutils/Debuginfod.html) system can be
	quite helpful.

	To ensure the correct binaries and debug symbols are used, the minidump contains
	the build-id for each loaded module in the `ModuleListStream` in the
	`CvRecordRva`'s `Signature`. This build-id is matched against a note
	section of type `NT_GNU_BUILD_ID`, usually named `.note.gnu.build-id` in the
	executable and symbol files. This note can be inspected with `readelf -n
	<file>` like `readelf -n chrome` or `readelf -n chrome.debug` and looking for
	the `.note.gnu.build-id` section. `readelf` reports the `Build ID` as the flat
	bytes in the note, but Breakpad binaries like `stackwalk_minidimp` and
	`dump_syms` will report and expect this truncated to a formatted Type 2 GUID
	(without dashes). This means `readelf` will output a `<build-id>` like
	33221100554477668899AABBCCDDEEFFXXXXXXXX but crashpad binaries will expect and
	report this as a `<build-uuid>` of 00112233445566778899AABBCCDDEEFF.

	The `.gnu_debuglink` section states which debug symbol file to use with a
	striped binary. For example `readelf --string-dump=.gnu_debuglink chrome`
	produces `chrome.debug`. This can be helpful to know for libraries with
	interesting debug symbol setup, like libc.so.6.

	## Create Symbolized Stack

	Given a minidump with the name `mini.dmp`

	`minidump_stackwalk mini.dmp > mini.stackwalk.nosym`

	This will produce a mostly unsymbolized summary of the crash. To symbolize, look
	toward the bottom of the output for `WARNING: No symbols, <file>, <build-uuid>`.
	For each `<file>` which is of interest, `mkdir -p symbols/<file>/<build-uuid>` then
	`dump_syms <file> <directory-with-file.debug> >
	symbols/<file>/<build-uuid>/<file>.sym`. Ensure this output `<file>.sym`
	contains the expected `<build-uuid>`. Then re-run `minidump_stackwalk` but
	with the symbols directory, like `minidump_stackwalk mini.dmp symbols/ >
	mini.stackwalk`.

	The output of `minidump_stackwalk` is often quite useful and enough to track
	down many issues. However, it does not fully use all of the information from
	DWARF, so it is possible sometimes to get much better stack traces from a full
	debugger like gdb. This is particularly true when functions have been
	aggressively inlined.

	## Create Core File

	`minidump-2-core mini.dmp > mini.core`

	## Loading into GDB

	This works best if the binaries, symbols, and core files are all in different
	directories to prevent gdb from automatically loading them into the wrong
	locations. This is also generally necessary when using a system installed
	version of Chrome. For full details see [the gdb
	manual](https://sourceware.org/gdb/onlinedocs/gdb/Separate-Debug-Files.html).
	The easiest way is to rename and move the .debug files to a directory structure
	like `<debugdir>/debug/.build-id/nn/nnnnnnnn.debug` where `nn` are the first
	two hex characters of the `build-id`, and `nnnnnnnn` are the rest of the hex
	characters of the `build-id`. Note that this `build-id` is exactly what is
	reported by `readelf -n <binary> \| grep "Build ID"` and not the `build-uuid`
	used by Breakpad. Then in gdb use `show debug-file-directory` to get the
	`<previous-directories>` and `set debug-file-directory
	<previous-directories>:<debugdir>/debug`.

	The `offset`s used here are the offsets of the corresponding module from the
	output of `minidump_stackwalk` or (equivalently) the value of
	`ModuleListStream::Modules[]::BaseOfImage` from the minidump file (which can be
	read with the structure definition).

	```
	$ gdb
	(gdb) file <executable>
	(gdb) show debug-file-directory
	<previous directories>
	(gdb) set debug-file-directory <previous directories>:<debugdir>/debug
	(gdb) symbol-file <executable> -o <executable-offset>
	(gdb) core-file <mini.core>
	```

	Running the commands in this order avoids needing to load the symbols twice and
	maps the `<executable>` to the expected location.

	To add an additional shared library it is possible to
	`(gdb) add-symbol-file <shared-library> -o <shared-library-offset>`

	Source paths in Chrome builds are relative to the `out/<build>` directory. If you
	have a Chromium checkout at or around when the Chrome build was created, it can
	be added to the debugger search path, like

	```
	(gdb) directory <path-to-chromium>/chromium/src/out/<build>/
	```