i#2260 web: Add wiki content to doxygen docs (#2409) Adds all of the Github wiki content to new doxygen docs pages integrated into the web site. Updates references to wiki pages in other files in the repository. Issue: #2260
diff --git a/.github/ISSUE_TEMPLATE/bug_report.md b/.github/ISSUE_TEMPLATE/bug_report.md index 9057049..533426f 100644 --- a/.github/ISSUE_TEMPLATE/bug_report.md +++ b/.github/ISSUE_TEMPLATE/bug_report.md
@@ -18,7 +18,7 @@ 2. Precise command line for running the application. 3. Exact output or incorrect behavior. -Please also answer these questions drawn from https://github.com/DynamoRIO/drmemory/wiki/Debugging#narrowing-down-the-source-of-the-problem : +Please also answer these questions drawn from https://drmemory.org/page_help.html#sec_narrow : - Does the problem go away when running in light mode (pass `-light` to Dr. Memory)? - Does the problem go away when running with the options `-leaks_only -no_count_leaks -no_track_allocs`? - Does the problem go away when running under plain DynamoRIO? Do this by running `dynamorio/bin32/drrun -- <application and args>` or `dynamorio/bin64/drrun -- <application and args>` depending on the bitwidth of your applicaiton. (Ignore warnings about "incomplete installation".) @@ -38,7 +38,7 @@ **Versions** - What version of Dr. Memory are you using? - Does the latest build from -https://github.com/DynamoRIO/drmemory/wiki/Latest-Build solve the problem? +https://drmemory.org/page_download.html#sec_latest_build solve the problem? - What operating system version are you running on? ("Windows 10" is *not* sufficient: give the release number.) - Is your application 32-bit or 64-bit?
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 3993ff6..dadeb7b 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md
@@ -1,20 +1,11 @@ # Contributing to Dr. Memory We welcome contributions to Dr. Memory. See our [list of project ideas for -contributors](http://drmemory.org/projects.html) and also [our list that -includes DynamoRIO -projects](https://github.com/DynamoRIO/drmemory/wiki/Projects). +contributors](http://drmemory.org/projects.html). -If you would like to contribute code to Dr. Memory, you will need to first sign a -[Contributor License Agreement](https://developers.google.com/open-source/cla/individual). - -Our wiki contains further information on policies, how to check out the -code, and how to add new code: - -- [Contribution policies and suggestions](https://github.com/DynamoRIO/drmemory/wiki/Contributing) -- [Git workflow](https://github.com/DynamoRIO/drmemory/wiki/Workflow) -- [Code style guide](https://github.com/DynamoRIO/drmemory/wiki/Code-Style) -- [Code reviews](https://github.com/DynamoRIO/drmemory/wiki/Code-Reviews) +[Our web site](https://drmemory.org/page_contribute.html) +contains further information on policies, how to check out the +code, and how to add new code. ## Reporting issues @@ -38,7 +29,7 @@ -no_count_leaks -no_track_allocs"? Does the problem go away when using the most recent build from -https://github.com/DynamoRIO/drmemory/wiki/Latest-Build? +https://drmemory.org/Latest-Build? Try the debug version of Dr. Memory and of its underlying engine DynamoRIO by passing "-debug -dr_debug -pause_at_assert" to drmemory.exe. Are any @@ -46,7 +37,7 @@ Please provide any additional information below. Please also see the "Narrowing Down the Source of the Problem" section -of https://github.com/DynamoRIO/drmemory/wiki/Debugging. +of https://drmemory.org/Debugging. ``` ### Including code in issues
diff --git a/README b/README index 942521c..7ed3c67 100644 --- a/README +++ b/README
@@ -1,5 +1,5 @@ # ********************************************************** -# Copyright (c) 2010-2016 Google, Inc. All rights reserved. +# Copyright (c) 2010-2021 Google, Inc. All rights reserved. # Copyright (c) 2008-2009 VMware, Inc. All rights reserved. # ********************************************************** @@ -88,8 +88,7 @@ Dr. Memory's source code and issue tracker live at https://github.com/DynamoRIO/drmemory -If you would like to submit a patch, you will need to first sign a -Contributor License Agreement. See -https://github.com/DynamoRIO/drmemory/wiki/Contributing for more information. +We welcome contributions to Dr. Memory. Please see our +instructions for contributing at https://drmemory.org/page_contribute.html. ===========================================================================
diff --git a/README.md b/README.md index ef2e1c7..61ae2ca 100644 --- a/README.md +++ b/README.md
@@ -13,8 +13,7 @@ Linux, Mac, or Android on commodity IA-32, AMD64, and ARM hardware. Dr. Memory is released under an LGPL license and binary packages are -[available for -download](https://github.com/DynamoRIO/drmemory/wiki/Downloads). +[available for download](https://drmemory.org/page_download.html). Dr. Memory is built on the [DynamoRIO dynamic instrumentation tool plaform](http://dynamorio.org).
diff --git a/docs/CMake_doxyfile.cmake b/docs/CMake_doxyfile.cmake index c632fe2..bc2efa3 100644 --- a/docs/CMake_doxyfile.cmake +++ b/docs/CMake_doxyfile.cmake
@@ -217,9 +217,9 @@ if (TOOL_DR_MEMORY) string(REGEX REPLACE "using.dox" - "using.dox errors.dox reports.dox light.dox chinese.dox fuzzer.dox coverage.dox tools.dox ${headers} debugging.dox" + "download.dox using.dox errors.dox reports.dox light.dox chinese.dox fuzzer.dox coverage.dox tools.dox ${headers} debugging.dox" string "${string}") - string(REGEX REPLACE "drfuzz.dox" "drfuzz.dox license.dox" string "${string}") + string(REGEX REPLACE "drfuzz.dox" "drfuzz.dox license.dox contribute.dox build.dox projects.dox new_release.dox test.dox design_docs.dox arm_port.dox" string "${string}") else () string(REGEX REPLACE "main.dox" "main.dox ${headers}" string "${string}")
diff --git a/drmemory/docs/arm_port.dox b/drmemory/docs/arm_port.dox new file mode 100644 index 0000000..8b21bea --- /dev/null +++ b/drmemory/docs/arm_port.dox
@@ -0,0 +1,198 @@ +/* ********************************************************** + * Copyright (c) 2010-2021 Google, Inc. All rights reserved. + * **********************************************************/ + +/* Dr. Memory: the memory debugger + * + * This library is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; + * version 2.1 of the License, and no later version. + + * This library is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Library General Public License for more details. + + * You should have received a copy of the GNU Lesser General Public + * License along with this library; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +/** + **************************************************************************** + **************************************************************************** +\page page_arm_port ARM Port + +# ARM Port Design Document + +## Pattern Mode + +### Instrumentation to compare a memory value to an immediate + +We can't easily use our x86 instrumentation: + + cmp <memval>, 0xf1fdf1fd + +We may have to do sthg like: + + spill r0 + spill r1 + ldr r0, <memval> + movw 0xf1fd, r1 + movt 0xf1fd, r1 + cmp r0, r1 + restore r0 + restore r1 + +#### Thumb mode: can repeat single byte + +The expanded-immeds do allow: + + cmp r0, 0xf100f100 + +Or + + cmp r0, 0x00fd00fd + +Or + + cmp r0, 0xf1f1f1f1 + +For Thumb, anyway. + +Probably it's worth changing the pattern to avoid extra spills and instrs. + +What it looks like with 0xf1f1f1f1: + + +22 m4 @0x5291e120 <label> + +22 m4 @0x5291db00 f8ca c084 str %r12 -> +0x00000084(%r10)[4byte] + +26 m4 @0x5291e088 f8d3 c004 ldr +0x04(%r3)[4byte] -> %r12 + +30 m4 @0x5291e03c f1bc 3ff1 cmp %r12 $0xf1f1f1f1 + +34 m4 @0x5291dfa4 e7fe b.ne @0x5291e0d4[4byte] + +36 m4 @0x5291df58 de00 udf $0x00000000 + +38 m4 @0x5291e0d4 <label> + +38 L3 f843 1f04 str %r1 $0x00000004 %r3 -> +0x04(%r3)[4byte] %r3 + +With flags save: + + +12 m4 @0x550b2408 f8ca 0084 str %r0 -> +0x00000084(%r10)[4byte] + +16 m4 @0x550b2920 f3ef 8000 mrs %cpsr -> %r0 + +20 m4 @0x550b2454 f8ca 0080 str %r0 -> +0x00000080(%r10)[4byte] + +24 m4 @0x550b1e98 f8d1 00e4 ldr +0x000000e4(%r1)[4byte] -> %r0 + +28 m4 @0x550b24a0 f1b0 3ff1 cmp %r0 $0xf1f1f1f1 + +32 m4 @0x550b231c e7fe b.ne @0x550b26fc[4byte] + +34 m4 @0x550b22d0 de00 udf $0x00000000 + +36 m4 @0x550b26fc <label> + +36 L3 f8c1 20e4 str.hi %r2 -> +0x000000e4(%r1)[4byte] + +40 m4 @0x550b1bd8 f8da 0080 ldr +0x00000080(%r10)[4byte] -> %r0 + +44 m4 @0x550b252c f380 8c00 msr $0x0c %r0 -> %cpsr + +48 m4 @0x550b26a8 f8da 0084 ldr +0x00000084(%r10)[4byte] -> %r0 + +#### To avoid spilling flags, try sub+cbnz in thumb mode + +Our scratch reg must be r0-r7 for cbnz though. + +And we'd have to add an IT block for sub (but cbnz cannot be inside it). + +So maybe we should only do it when the flags are live? Thus +adding more complexity to the fault identification code. + +#### ARM mode: cannot repeat an immmed byte! Use OP_sub x4? + +But what about ARM? ARM immediates in GPR instrs are just an 8-bit value +rotated: no repeating. Even the SIMD and VFP immeds aren't much help, +except maybe the cmode=1111 combined with cmode=1100? Subtract one and +then the other? + +We could use mvn if most bits are 1's: sthg like 0xfff1ffff, but we still +need to spill a reg, and if we do that we may as well use movw,movt. + +#### Do 4 subtracts? + +Faster than a spill, though not if we have a (2nd) dead reg. + +So we'd do: + + sub r0, 0xf1000000 + sub r0, 0x00f10000 + sub r0, 0x0000f100 + sub r0, 0x000000f1 + cmp r0, 0 (cbnz is Thumb-only) + jne skip + udf + skip: + +We could use 0xf1fdf1fd here -- but maybe simplest to still limit to +single-byte for consistency w/ Thumb? + +Vs the movw,movt: 2 extra instrs if reg dead, same # and no mem access if +live. Can we ask drreg whether dead or not? +=> +add drreg_is_register_dead() + +However, having 2 different versions complicates the fault handling. + +Double-checking the compiler doesn't have some trick: + + if (argc == 0xf1fdf1fd) + return 1; + => + gcc thumb -O3: + 8372: f24f 13fd movw r3, #61949 ; 0xf1fd + 8376: f2cf 13fd movt r3, #61949 ; 0xf1fd + 837a: 4298 cmp r0, r3 + gcc arm -O3: + 8374: e30f31fd movw r3, #61949 ; 0xf1fd + 8378: e34f31fd movt r3, #61949 ; 0xf1fd + 837c: e1500003 cmp r0, r3 + +Real example: + + +4 m4 @0x4f8c5be8 e58a1084 str %r1 -> +0x00000084(%r10)[4byte] + +8 m4 @0x4f8c60a8 e10f1000 mrs %cpsr -> %r1 + +12 m4 @0x4f8c5c34 e58a1080 str %r1 -> +0x00000080(%r10)[4byte] + +16 m4 @0x4f8c6134 e5901000 ldr (%r0)[4byte] -> %r1 + +20 m4 @0x4f8c6180 e24114f1 sub %r1 $0xf1000000 -> %r1 + +24 m4 @0x4f8c5ff4 e24118f1 sub %r1 $0x00f10000 -> %r1 + +28 m4 @0x4f8c5b10 e2411cf1 sub %r1 $0x0000f100 -> %r1 + +32 m4 @0x4f8c5f28 e24110f1 sub %r1 $0x000000f1 -> %r1 + +36 m4 @0x4f8c5f68 e3510000 cmp %r1 $0x00000000 + +40 m4 @0x4f8c5b50 1afffffe b.ne @0x4f8c60f4[4byte] + +44 m4 @0x4f8c6250 e7f000f0 udf $0x00000000 + +48 m4 @0x4f8c60f4 e7f000f0 <label> + +48 L3 e5900000 ldr (%r0)[4byte] -> %r0 + +52 m4 @0x4f8c6290 e59a1080 ldr +0x00000080(%r10)[4byte] -> %r1 + +56 m4 @0x4f8c62dc e12cf001 msr $0x0c %r1 -> %cpsr + +60 m4 @0x4f8c6334 e59a1084 ldr +0x00000084(%r10)[4byte] -> %r1 + +#### Switch to thumb mode just for the cmp? + +Breaks DR's rules: would mess up decode_fragment. + +Instead of inlining, could jump to separate gencode (need 15 forms one for +each scratch reg) -- if already in cache maybe ok that it's not local. + +#### Load immed from TLS slot + +If TLS in data cache and have L1 hit, may be as fast as movw,movt, and +it's shorter code. + +#### Go w/ unified ARM+Thumb same approach for simpler code? + +#### Permanently steal another reg? + +Very complex w/ interactions w/ r10 though + +#### Put the optimizations under an option and under option switch to single-byte pattern val + +#### For 2 spills, have drreg use ldm or ldrd? + +For 2 spills, is ldm or ldrd faster? Qin's initial tests showed no faster +than separate ldr x2, so even though instr density is better, if it makes drreg +really complex it's prob not worth doing. + + +**************************************************************************** +**************************************************************************** +*/
diff --git a/drmemory/docs/build.dox b/drmemory/docs/build.dox new file mode 100644 index 0000000..0cfb95a --- /dev/null +++ b/drmemory/docs/build.dox
@@ -0,0 +1,470 @@ +/* ********************************************************** + * Copyright (c) 2010-2021 Google, Inc. All rights reserved. + * **********************************************************/ + +/* Dr. Memory: the memory debugger + * + * This library is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; + * version 2.1 of the License, and no later version. + + * This library is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Library General Public License for more details. + + * You should have received a copy of the GNU Lesser General Public + * License along with this library; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +/** + **************************************************************************** + **************************************************************************** +\page page_build Building from Sources + +# Source Code + +The source code can be obtained from our git repository. Be sure to run the `devsetup.sh` script to set up the git aliases for [our workflow](\ref page_contribute). + +``` +$ git clone https://github.com/DynamoRIO/drmemory.git +$ cd drmemory +$ make/git/devsetup.sh +``` + +# Pre-Built Packages + +If you don't need to modify Dr. Memory and just want a build more recent than the last official release, we provide [weekly package builds](Latest-Build). + +# GitHub Zip Download Not Supported + +We do not support building from the "Download Zip" GitHub feature, as it A) fails to support submodules and B) thwarts our patchlevel setting mechanism (xref https://github.com/DynamoRIO/dynamorio/issues/1565). + +# Components + +The Dr. Memory code base contains two tools: +-# Dr. Memory: memory debugging tool +-# Dr. Heapstat: memory profiling tool + +Which tool to build is determined by whether the TOOL_DR_HEAPSTAT CMake +variable is enabled. + +Both tools use the [DynamoRIO dynamic instrumentation tool platform](http://dynamorio.org). The sources for the version of DynamoRIO needed is included as a git submodule in the dynamorio/ subdirectory. By default, a Dr. Memory build also builds DynamoRIO from these embedded sources. You can also point at a separate DynamoRIO build if you have built it or downloaded it separately, using the DynamoRIO_DIR CMake variable. + +# Windows Setup + +You'll need to install a compiler, the Debugging Tools for Windows, CMake, and a few other tools in order to build Dr. Memory +on Windows. Since Dr. Memory also builds DynamoRIO, you can follow the +[setup instructions for DynamoRIO](https://dynamorio.org/page_building.html). We summarize what is needed here for the specific DynamoRIO build built by Dr. Memory. + +## Compiler + +For the compiler, install Visual Studio 2013. We have never tested building from within the +Visual Studio IDE, but it should work (CMake can generate Visual Studio +project files as well as Makefiles). If you try it, please update these +instructions. From the command line, set up the path and environment +variables for Visual Studio as normal: run the `vcvars` batch file, either manually in a cmd shell or by running the Visual Studio Command Prompt which has a Start Menu entry under Visual Studio Tools, +or translate the vcvars environment variables into cygwin bash. + +## Debugging Tools for Windows + +Dr. Memory needs redistributable copies of dbghelp.dll and symsrv.dll. +These are available in a Visual Studio or SDK installation that has the +Remote Debugger installed, in the Windows 8 SDK, and in the WDK or DDK. +Visual Studio Express seems to have the right files but has not been tested +for all versions. + +If you choose to use the WDK or DDK, we have tested the Windows +Driver Kit version 7.1.0 (7600.16385.1), the Windows Vista WDK (6000), and +the Windows 2003 DDK (3790.1830). If you install into a non-default +location (i.e., other than `$SYSTEMDRIVE/WINDDK/3790.1830` or +`$SYSTEMDRIVE/WINDDK/6000`) then you must point the DDKROOT environment +variable at the base of the DDK/WDK: + +``` +export DDKROOT=`cygpath -dm ~/WINDDK/6000/` +``` + +Alternatively, you can point at a redistributable copy of dbghelp.dll of at +least version 6.3 with the CMake variable DBGHELP_DLL and at a version of +symsrv.dll with the CMake variable SYMSRV_DLL. + +## CMake + +You need CMake for Windows (*not* for Cygwin, as we need to pass paths with +drive letters to our compiler), at least version 3.7. You can download a +binary installation here: + + http://www.cmake.org/cmake/resources/software.html + +Install CMake and put its bin directory on the path for your shell. + +## Other Tools + +To build the Dr. Heapstat data visualizer, the Flex SDK and Java are +required. We have tested versions 3.3, 3.4, and 4.1 of the Flex SDK. It +can be obtained from http://www.adobe.com/go/flex4_sdk. Set the CMake +variable BUILD_VISUALIZER to ON, and then set the CMake variable FLEX_SDK +to point at the base of the directory where you unzipped the SDK files, or +set the FLEXROOT environment variable to point there: + +``` +export FLEXROOT=/home/bruening/work/flex_sdk_4.1 +``` + +For Java, if the java interpreter is not found by CMake, set the JAVA CMake +variable. On Windows all that's needed is the Java runtime environment +(e.g., from http://www.oracle.com/technetwork/java/javase/downloads/index.html). + +You can build the Dr. Heapstat data gathering tool without either Flex or +Java. By default, that's what is built: only if the BUILD_VISUALIZER CMake +variable is enabled is the visualizer built. Be aware that the visualizer +is a prototype built several years ago and it may not work out of the box +on modern browsers, in particular on Windows. We are looking for +contributors to build a new visualizer. + +In order to build the documentation, you will additionally need doxygen, +which can be either a Cygwin package or native Windows. Currently the +documentation is always built, so this is required. + +To build a Windows installation packagage (with an installer, instead of +just a zip file for local installs), you will need to have +[NSIS](http://nsis.sourceforge.net) installed. If NSIS is not found, there +will not be an error, but only the zip package on Windows will be built. +Your NSIS install must be patched with the ["Large Strings" build](http://nsis.sourceforge.net/Special_Builds) in order to support long PATHs. A configure-time check ensures this is in place. See [Issue 1029](http://code.google.com/p/drmemory/issues/detail?id=1029) for more information. + +In order to build DynamoRIO you must have a version of perl. It can be +either a Cygwin perl or a native Windows perl. + +---------------- + +# Linux Setup + +In order to build you'll need the following packages: + + - gcc + - binutils + - cmake (at least version 3.7) + - perl + +To build Dr. Heapstat you will additionally need: + + - Flex SDK + - Java + +In order to build the documentation, you will additionally need: + + - doxygen + +If your machine does not have support for running 32-bit applications and +its version of binutils is older than 2.18.50 then you'll need to set the +`CMAKE_ASM_COMPILER` CMake cache variable to point at a GNU assembler of at +least version 2.18.50. + +If you're using a 64-bit Linux distribution, be sure to install 32-bit +development support (for Ubuntu, the `ia32-libs` and `g++-multilib` +packages). + +---------------- + +# Building with CMake + +If you have all the required tools installed in standard locations, CMake +will find them automatically. Only the Flex SDK and the DDK (which is only +needed with non-full installations of Visual Studio) are likely to +not be found as they do not normally live in system install directories. +If you set the FLEXROOT and DDKROOT environment variables as specified +above, then CMake should find all your tools without any custom CMake +variable settings. + +## Configuring Your Build + +First, configure your compiler. On Windows this means setting up the +environment variables (including `PATH`). On Linux, if your compiler +supports both 64-bit and 32-bit targets, you can select the non-default +type by setting the `CFLAGS` and `CXXFLAGS` environment variables to the +flags used to change your compiler's target. For example, for gcc: + +``` +export CXXFLAGS=-m32 +export CFLAGS=-m32 +``` + +Next, in your shell (which you have set up for building with your +compiler), create a build directory. Out-of-source builds are supported +and recommended. + +You can configure using the CMake GUI or directly from the command line. +We describe the GUI method first. To use the GUI, invoke it from the build +directory, pointing at the source directory. For example, if your current +directory is the base of the DynamoRIO tree: + +On Windows, from a Cygwin bash shell: +``` +mkdir build +cd build +CMakeSetup .. +``` + +On Linux: +``` +mkdir build +cd build +cmake-gui .. +``` + +Alternatively, `ccmake`, the curses-based GUI, can be used on Linux. + +On Windows, press "Configure" and select "Visual Studio 9 2008" or "Visual +Studio 8 2005); on Linux, press Configure and then select "Unix Makefiles" +(or, in `ccmake`, press `c` once). + +Now you can select the parameters to configure your build. The main +parameters are as follows: + + - TOOL_DR_HEAPSTAT = whether to build Dr. Heapstat instead of Dr. Memory + - CMAKE_BUILD_TYPE = whether to build Debug or RelWithDebInfo + +Note that the DynamoRIO build options are present as advanced options but +in general should be ignored as Dr. Memory sets the appropriate settings. + +Press Configure and then OK (or `c` and then `g`) to complete the +configuration step and generate the Makefiles. It is normal to see +messages like this for various types (uint, ushort, bool, byte, sbyte, +uint32, uint64, int32, int64): + +``` +-- Check size of bool +-- Check size of bool - failed +``` + +This is really a check for the existence of a typedef and it is normal for +some such checks to print a "failure" message. + +On Windows, if you have both `cl` and `gcc` on your path, you can ensure +that CMake selects `cl` by setting the `CC` and `CXX` environment +variables. For example: + +``` +CC=cl CXX=cl cmake .. +``` + +To improve compilation speed on Windows, if you have CMake 2.8 or later, +you can turn off the progress and status messages with this CMake define: + +``` +-DCMAKE_RULE_MESSAGES:BOOL=OFF +``` + +## Building and Installing + +After the configuration step is complete, type `cmake --build .` +on Windows or `make` on Linux. + +For a release build on Windows, you'll need to use: +``` +cmake --build . --config RelWithDebInfo +``` + +## Configuring in Batch Mode + +Instead of interactive configuration you can pass all your parameter +changes to CMake on the command line. For example: + +On Windows, with code checked out into the current directory named `src`: +``` +mkdir ../build +cd ../build +cmake -G"Visual Studio 9 2008" -DCMAKE_BUILD_TYPE=Debug ../src +cmake --build . +``` + +On Linux, with code checked out into the current directory named `src`: +``` +mkdir ../build +cd ../build +cmake -DCMAKE_BUILD_TYPE=Debug ../src +make +``` + +Developers may wish to pass `-DCMAKE_VERBOSE_MAKEFILE=1` to CMake in order +to see the compilation command lines. They can alternatively be seen for +any individual build with `make VERBOSE=1`. + +## Re-Configuring + +You can re-run the configure step, using the same parameters that you +originally used, in two ways: + +``` + make rebuild_cache +``` + +Or: + +``` + cmake . +``` + +With the latter command you can also change the configuration, but be sure +to do a `make clean` prior to building as CMake does not perform +dependence checking on this type of modification. As an example: + +``` + cmake -DCMAKE_BUILD_TYPE=RelWithDebInfo . + make clean +``` + +## Using Ninja For Better Error Messages, Faster Builds, and Proper Incremental Builds + +Ninja is a new build system that provides a much better build experience +from the command line. We highly recommend using it on Windows. Follow +the +[DynamoRIO instructions for using Ninja](https://dynamorio.org/page_building.html#sec_ninja). + +## Using MSBuild For Better Error Messages And Faster Builds + +We recommend Ninja (see above) over MSBuild, but we're leaving the +information on MSBuild here as it is recommended over devenv. + +By default, CMake will use devenv when building for Visual Studio generators. +An exception is CMake 2.8.4, which uses MSBuild by default, but does not build +in parallel without passing the "/m" flag to MSBuild: + +``` +cmake --build . --target RelWithDebInfo -- /m +``` + +MSBuild summarizes all build warnings and errors at the end of the run, unlike +devenv. It is also a little faster than devenv (when parallel building is +enabled). There are some general outstanding issues with MSBuild, but it seems +to work well for Dr. Memory with Visual Studio 2010. To request MSBuild set the +CMAKE_MAKE_PROGRAM variable at configuration time. In a Visual Studio Command +Prompt, MSBuild is on the PATH, so the absolute path is not needed (here the code is checked out into the current directory named `src`): + +``` +cmake -G"Visual Studio 10" -DCMAKE_MAKE_PROGRAM:FILEPATH=MSBuild ../src +``` + +Don't forget to pass the "/m" flag (after "--") as shown above when building. + +To request verbose build information, use the "/v:diag" flag: + +``` +cmake --build . --target RelWithDebInfo -- /m /v:diag +``` + +Here is a sample bash function to make building easier within Cygwin: + +``` +function build { + config=`grep ^CMAKE_CONFIGURATION_TYPES:S CMakeCache.txt | sed 's/^.*=//'` + /usr/bin/time cmake --build . --config $config -- /m +} +``` + +## Building the Qt Dr. Heapstat visualizer + +See the [Qt DrGUI extension build instructions for DynamoRIO](https://dynamorio.org/page_building.html#sec_drgui). + +---------------- + +## Cross-Compiling for ARM on Linux + +Install the cross compiler for the `gnueabihf` target: + +``` +$ sudo apt-get install gcc-arm-linux-gnueabihf binutils-arm-linux-gnueabihf g++-arm-linux-gnueabihf +``` + +Check out the sources as normal, and point at our toolchain CMake file: + +``` +$ git clone https://github.com/DynamoRIO/drmemory.git +$ cd drmemory +$ make/git/devsetup.sh +$ cd .. +$ mkdir build_arm +$ cd build_arm +$ cmake -DCMAKE_TOOLCHAIN_FILE=../drmemory/dynamorio/make/toolchain-arm32.cmake ../drmemory +$ make -j +``` + +---------------- + +## Cross-Compiling for ARM Android + +Install the Android NDK and configure it for the androideabi standalone toolchain. Something like this: + +``` +/myndkpath/android-ndk-r10e/build/tools/make-standalone-toolchain.sh --arch=arm --platform=android-21 --install-dir=/mytooldir/android-ndk-21 --toolchain=arm-linux-androideabi-4.9 +``` + +Switch from the gold linker to bfd: + +``` +ln -sf arm-linux-androideabi-ld.bfd /mytooldir/android-ndk-21/bin/arm-linux-androideabi-ld +``` + +You'll need a cmake version greater than 3.0.2 -- otherwise you'll see this error: + +``` +System is unknown to cmake, create: +Platform/Android to use this system, please send your config file to ... +``` + +Now check out the sources as normal, and point at our toolchain CMake file. + +``` +$ git clone https://github.com/DynamoRIO/drmemory.git +$ cd drmemory +$ make/git/devsetup.sh +$ cd .. +$ mkdir build_android +$ cd build_android +$ cmake -DCMAKE_TOOLCHAIN_FILE=../drmemory/dynamorio/make/toolchain-android.cmake -DANDROID_TOOLCHAIN=/mytooldir/android-ndk-21 ../drmemory +$ make -j +``` + +If your Android system does not have a writable `/data/local/tmp` directory, you will need to either set `DYNAMORIO_CONFIGDIR` to a writable location or always run from a writable current directory. + +If you have an Android device set up for access through the `adb shell` utility, the Android build is capable of automatically copying binaries to the device and running tests. First, ensure that `adb` is on your PATH, and that `adb status` indicates an attached device. Next, set the cmake variable `DRM_COPY_TO_DEVICE`. This sets up post-build steps for each target needed by the tests to copy binaries over to the device. Now `ctest` should work when run from the host machine. + +---------------- + +# Building a Release Package + +We use Github Actions to build official release packages of +Dr. Memory. This section describes how to build a local package. + +Use the package.cmake CTest script to build a release package. On Linux, +this is a tarball; on Windows, both a zip file and an installation +executable are built. The Windows installer is only built, however, if +[NSIS](http://nsis.sourceforge.net) is installed on your machine. See the instructions under Other Tools for how to appropriately patch NSIS. + +Launch package.cmake using `ctest -S`. Here is an example of building a +Dr. Memory package from the current sources: + +``` + git clone https://github.com/DynamoRIO/drmemory.git + cd drmemory + make/git/devsetup.sh + cd .. + mkdir build_package + cd build_package + ctest -S ../drmemory/package.cmake,drmem_only\;build=1 +``` + +The arguments to a ctest script are ;-separated (but the ; must be escaped +for most shells) and separated from the script name by a comma. + +To set a CMake variable for a CTest script, create a file and pass it +in via the preload argument to package.cmake. + +To diagnose build failures, look at the detailed output in the log files for each step (configure, build, test) in the `Testing/Temporary/` directory inside each build directory. + +**************************************************************************** +**************************************************************************** +*/
diff --git a/drmemory/docs/contribute.dox b/drmemory/docs/contribute.dox new file mode 100644 index 0000000..f8e6601 --- /dev/null +++ b/drmemory/docs/contribute.dox
@@ -0,0 +1,89 @@ +/* ********************************************************** + * Copyright (c) 2010-2021 Google, Inc. All rights reserved. + * **********************************************************/ + +/* Dr. Memory: the memory debugger + * + * This library is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; + * version 2.1 of the License, and no later version. + + * This library is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Library General Public License for more details. + + * You should have received a copy of the GNU Lesser General Public + * License along with this library; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +/** + **************************************************************************** + **************************************************************************** +\page page_developers Developer Documentation + +- \subpage page_contribute +- \subpage page_build +- \subpage page_submodule +- \subpage page_projects +- \subpage page_new_release +- \subpage page_test +- \subpage page_design_docs + +\page page_contribute Contributing to Dr. Memory + +We welcome contributions to Dr. Memory. We do not require a formal contributor license agreement or transfer of copyright. Contributions are implicitly assumed to be offered under terms of [Dr. Memory's primary LGPL 2.1 license](https://github.com/DynamoRIO/drmemory/blob/master/license.txt) as well as a [BSD license](https://github.com/DynamoRIO/drmemory/blob/master/license.txt#L485). + +We use the [same code review, code style, and workflow as for DynamoRIO](https://dynamorio.org/page_contributing.html). The `git pullall` alias is extended for Dr. Memory to properly update the [DynamoRIO submodule](\ref page_submodule). + + +\page page_submodule Updating the DynamoRIO Submodule + +# DynamoRIO Inside Dr. Memory + +DynamoRIO is included inside the Dr. Memory source tree as a [git submodule](http://git-scm.com/docs/git-submodule). We support both building that embedded DR along with pointing to an already-built separate DR export tree. + +# Submodule Rollback + +Be careful with submodules when pulling upstream changes. If a submodule +was modified upstream and only a `git pull` is run without a subsequent +`git submodule update`, that upstream change will be reverted upon the next +`git push`. Please use the `pullall` alias to avoid +accidentally clobbering submodule changes. + +We do have a pre-commit hook in place that will warn you if you try to roll +back the dynamorio submodule while checking in at least one other file: + +``` +% git commit -a +Error: the dynamorio submodule is being rolled back. +This is likely a mistake: did you pull but not run git submodule update? +Aborting commit. +``` + +# Updating the DynamoRIO Version + +Run `git checkout <hash>` to update DynamoRIO to that hash: + +``` +cd dynamorio +git pull +git checkout 0c81acfc9aaea949e6f6ecfe0b4157c06a014dda +``` + +If Dr. Memory's sources are being changed to require this new version of DR, apply the Dr. Memory changes such that they will be included in the same commit that updates the submodule version. Also add to the commit message something like "Relies on DR <hash>. Updates DR to <hash>.". + +If Dr. Memory won't build, or shouldn't be built, with any older version of +DR, also update the set of the "DynamoRIO_VERSION_REQUIRED" variable in +trunk/CMakeLists.txt (use the patchlevel printed by `cmake .`). + +Now run the Dr. Memory test suite **with an embedded DR, not with an already-built DR** to ensure it's happy with the updated DR. + +In your commit log message state to which version you're updating DR, and why (typically to pull in a certain bug fix or new feature). + + +**************************************************************************** +**************************************************************************** +*/
diff --git a/drmemory/docs/debugging.dox b/drmemory/docs/debugging.dox index 54f618c..c0a90e4 100644 --- a/drmemory/docs/debugging.dox +++ b/drmemory/docs/debugging.dox
@@ -25,12 +25,23 @@ **************************************************************************** \page page_help Obtaining Help and Reporting Bugs +# Discussion Email List + For questions, bug reports, and discussion, join the <a href="http://groups.google.com/group/drmemory-users/">Dr. Memory Users group</a>. +# Issue Tracker + The Dr. Memory issue tracker is located at: -http://drmemory.org/issues +https://github.com/DynamoRIO/drmemory/issues + +\section sec_debugging Debugging Dr. Memory + +For general information about debugging DynamoRIO, please see [its docs](https://dynamorio.org/page_debugging.html). +This page contains a couple of how-tos with Dr. Memory specifics. + +## General Tips If a problem in the Dr. Memory tool itself is encountered, try the following steps to help diagnose the issue: @@ -40,9 +51,47 @@ drmemory.exe. Are any messages reported? - Try running light mode ("-light"). Does the issue appear there? - Try running "-leaks_only -no_count_leaks -no_track_allocs". Does that work? - - Try a recent build from https://github.com/DynamoRIO/drmemory/wiki/Latest-Build + - Try the [latest build](\ref sec_latest_build) - Try disabling anti-virus or other invasive software, as it may not interoperate well with Dr. Memory (see \ref sec_interop). + - Look in the log file, which is in the same process as the results.txt file but is named `global.<pid>.log` for warnings and other messages. You can increase verbosity via the -verbose flag, but only use higher levels for short-running applications. + +\subsection sec_narrow Narrowing Down the Source of the Problem + +First, as with all debugging scenarios, try to reduce the application workload that shows the problem to the smallest and shortest scenario that you can, to make it easier to analyze upon re-execution. + +If you have a reproducible problem where an application does not work properly under Dr. Memory, try running it with each of these sets of runtime options to determine which component of Dr. Memory is responsible. Try these in order, and as soon as your application works, you can then report that it works with one set of options but not with another, which will make it easier for a developer to further diagnose the issue: + +``` +-no_count_leaks +``` +``` +-light +``` +``` +-leaks_only +``` +``` +-leaks_only -no_count_leaks +``` +``` +-leaks_only -no_count_leaks -no_track_allocs +``` +If the problem persists with all of the above options to Dr. Memory, try running under plain DynamoRIO. You can run the version in the Dr. Memory package like this (ignore warnings about "incomplete installation"): + +``` +dynamorio/bin32/drrun -- <application and args> +``` + +For a 64-bit application run this instead: + +``` +dynamorio/bin64/drrun -- <application and args> +``` + +Alternatively, download [the latest standalone DynamoRIO](https://dynamorio.org/page_weekly_builds.html). + +## Gathering Additional Data To supply data for analyzing and fixing a bug, reproduce the problem using these options to Dr. Memory: @@ -67,6 +116,67 @@ and attach the resulting <tt>global.<pid>.log</tt> file from the same directory as the results.txt file (compress it if large). +## Windows + +To attach to a Dr. Memory process with the debugger, it's usually easiest to use the DynamoRIO -msgbox_mask flag which pops up a message box and pauses the app: +``` +drmemory.exe -dr_ops "-msgbox_mask 15" <others flags> -- <command> +``` +`-msgbox_mask 15` shows message boxes on all DR internal events, including the application start. + +Dr. Memory also has options -pause_at_exit, -pause_at_assert (which I normally have on all the time for development with the Dr. Memory Debug build), -pause_at_uninitialized, and -pause_at_unaddressable, each of which pop up message boxes at the appropriate point, allowing for a debugger to be attached. + +When the desired message box appears, open windbg and attach (F6 or File->Attach) to the PID mentioned in the message. + +You'll need to tell windbg where the DynamoRIO, Dr. Memory, and private libraries are located. DynamoRIO provides a script for this. You can tell windbg to execute it at startup: + +``` +"C:\Program Files (x86)\Debugging Tools for Windows\windbg.exe" -pt 1 -c "$><c:\src\dynamorio\tools\windbg-scripts\load_syms" +``` + +Or you can execute the command directly at the windbg prompt: +``` +$><c:\src\dynamorio\tools\windbg-scripts\load_syms +``` + +## Private Symbols + +Dr. Memory will load in private symbols from paths specified by the `_NT_SYMBOL_PATH` environment variable. Point it at the directory you downloaded the symbols to within windbg. Eventually Dr. Memory will support downloading them but probably never online (xref [issue 143](https://github.com/DynamoRIO/drmemory/issues#issue/143)). + +E.g., inside windbg: +``` +.symfix c:\src\symbols +.reload +x **!FOO # force lazy loading for each module +lm # check that "(pdb symbols)" is mentioned for each system library +``` + +And outside in the shell or systemwide or wherever: +``` +set _NT_SYMBOL_PATH=c:\src\symbols +``` + +## DMP files + +If you can reproduce a Dr. Memory assert in the Debug build, a DMP file would be very helpful. +Re-run Dr. Memory with `-pause_at_assert` and attach as described above, then type this in windbg: +``` +.dump /ma c:\mypath\issue543.dmp +``` +and get the DMP file. It may take quite a lot of time to write and the file may be pretty large but it's not a big problem as DMP files compress very well. + +If you're debugging a custom build, please ZIP the DMP file together with your `<debug>/bin` and `<debug_or_release>/dynamorio` directories so we can have your DLL and PDB files as well. Then, attach the ZIP file to the issue you've filed + +## Application Callstack + +You can get a callstack of the application by pointing windbg at the frame pointer, stack pointer, and pc to use. A common scenario is when the current windbg frame has a `dr_mcontext_t**` variable named "mc", and the pc is inside an app_loc_t var named "loc": + +``` +kb =@@(mc->ebp) @@(mc->esp) @@(loc->u.addr.pc) +``` + +The windbg documentation is very good, so look up these commands to learn more. Note that "@@" switches to C expression mode. + **************************************************************************** **************************************************************************** */
diff --git a/drmemory/docs/design_docs.dox b/drmemory/docs/design_docs.dox new file mode 100644 index 0000000..e80c2da --- /dev/null +++ b/drmemory/docs/design_docs.dox
@@ -0,0 +1,31 @@ +/* ********************************************************** + * Copyright (c) 2010-2021 Google, Inc. All rights reserved. + * **********************************************************/ + +/* Dr. Memory: the memory debugger + * + * This library is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; + * version 2.1 of the License, and no later version. + + * This library is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Library General Public License for more details. + + * You should have received a copy of the GNU Lesser General Public + * License along with this library; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +/** + **************************************************************************** + **************************************************************************** +\page page_design_docs Design Documents + +- \subpage page_arm_port + +**************************************************************************** +**************************************************************************** +*/
diff --git a/drmemory/docs/download.dox b/drmemory/docs/download.dox new file mode 100644 index 0000000..a90a36e --- /dev/null +++ b/drmemory/docs/download.dox
@@ -0,0 +1,88 @@ +/* ********************************************************** + * Copyright (c) 2010-2021 Google, Inc. All rights reserved. + * **********************************************************/ + +/* Dr. Memory: the memory debugger + * + * This library is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; + * version 2.1 of the License, and no later version. + + * This library is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Library General Public License for more details. + + * You should have received a copy of the GNU Lesser General Public + * License along with this library; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +/** + **************************************************************************** + **************************************************************************** +\page page_download Download + +\section sec_latest_build Latest Build + +For more recent bug fixes and features prior to the next official release, [weekly builds are available](https://github.com/DynamoRIO/drmemory/releases) (for Windows, only zip packages for portable/local installation are available: no installer is built) for recent commits. If there were no commits during the week, no build is generated that week. + +# Official Releases + +Official Dr. Memory packages are hosted as [GitHub releases](https://github.com/DynamoRIO/drmemory/releases). + +The [2.3.0 release](https://github.com/DynamoRIO/drmemory/releases/tag/release_2.3.0): + + - [Dr. Memory installer for Windows (DrMemory-Windows-2.3.0-1.msi)](https://github.com/DynamoRIO/drmemory/releases/download/release_2.3.0/DrMemory-Windows-2.3.0-1.msi) + + - [Dr. Memory zip file for portable/local installation on Windows (DrMemory-Windows-2.3.0-1.zip)](https://github.com/DynamoRIO/drmemory/releases/download/release_2.3.0/DrMemory-Windows-2.3.0-1.zip) + + - [Dr. Memory for Linux (DrMemory-Linux-2.3.0-1.tar.gz)](https://github.com/DynamoRIO/drmemory/releases/download/release_2.3.0/DrMemory-Linux-2.3.0-1.tar.gz) + + - [Dr. Memory for Mac -- alpha release (DrMemory-MacOS-2.3.0-1.tar.gz)](https://github.com/DynamoRIO/drmemory/releases/download/release_2.3.0/DrMemory-MacOS-2.3.0-1.tar.gz) + +The prior [2.2.0 release](https://github.com/DynamoRIO/drmemory/releases/tag/release_2.2.0): + + - [Dr. Memory installer for Windows (DrMemory-Windows-2.2.0-1.msi)](https://github.com/DynamoRIO/drmemory/releases/download/release_2.2.0/DrMemory-Windows-2.2.0-1.msi) + + - [Dr. Memory zip file for portable/local installation on Windows (DrMemory-Windows-2.2.0-1.zip)](https://github.com/DynamoRIO/drmemory/releases/download/release_2.2.0/DrMemory-Windows-2.2.0-1.zip) + + - [Dr. Memory for Linux (DrMemory-Linux-2.2.0-1.tar.gz)](https://github.com/DynamoRIO/drmemory/releases/download/release_2.2.0/DrMemory-Linux-2.2.0-1.tar.gz) + + - [Dr. Memory for Mac -- alpha release (DrMemory-MacOS-2.2.1-alpha2.tar.gz)](https://github.com/DynamoRIO/drmemory/releases/download/release_2.2.1_alpha2/DrMemory-MacOS-2.2.1-alpha2.tar.gz) + +The prior [2.1.0 release](https://github.com/DynamoRIO/drmemory/releases/tag/release_2.1.0), with 64-bit uninitialized read support for Linux and Windows, and Windows auto-syscall-generation: + + - [Dr. Memory installer for Windows (DrMemory-Windows-2.1.0-1.msi)](https://github.com/DynamoRIO/drmemory/releases/download/release_2.1.0/DrMemory-Windows-2.1.0-1.msi) + + - [Dr. Memory zip file for portable/local installation on Windows (DrMemory-Windows-2.1.0-1.zip)](https://github.com/DynamoRIO/drmemory/releases/download/release_2.1.0/DrMemory-Windows-2.1.0-1.zip) + + - [Dr. Memory for Linux (DrMemory-Linux-2.1.0-1.tar.gz)](https://github.com/DynamoRIO/drmemory/releases/download/release_2.1.0/DrMemory-Linux-2.1.0-1.tar.gz) + +The prior Linux-only [2.0.1 release](https://github.com/DynamoRIO/drmemory/releases/tag/release_2.0.1), with 64-bit uninitialized read support: + + - [Dr. Memory for Linux (DrMemory-Linux-2.0.1-2.tar.gz)](https://github.com/DynamoRIO/drmemory/releases/download/release_2.0.1/DrMemory-Linux-2.0.1-2.tar.gz) + +The older multi-platform [1.11.0 release](https://github.com/DynamoRIO/drmemory/releases/tag/release_1.11.0): + + - [Dr. Memory installer for Windows (DrMemory-Windows-1.11.0-2.msi)](https://github.com/DynamoRIO/drmemory/releases/download/release_1.11.0/DrMemory-Windows-1.11.0-2.msi) + + - [Dr. Memory zip file for portable/local installation on Windows (DrMemory-Windows-1.11.0-2.zip)](https://github.com/DynamoRIO/drmemory/releases/download/release_1.11.0/DrMemory-Windows-1.11.0-2.zip) + + - [Dr. Memory for Linux (DrMemory-Linux-1.11.0-2.tar.gz)](https://github.com/DynamoRIO/drmemory/releases/download/release_1.11.0/DrMemory-Linux-1.11.0-2.tar.gz) + + - [Dr. Memory for Mac (DrMemory-MacOS-1.11.0-2.tar.gz)](https://github.com/DynamoRIO/drmemory/releases/download/release_1.11.0/DrMemory-MacOS-1.11.0-2.tar.gz) + + - [Dr. Memory for Android/ARM (DrMemory-ARM-Android-EABI-1.11.0-2.tar.gz)](https://github.com/DynamoRIO/drmemory/releases/download/release_1.11.0/DrMemory-ARM-Android-EABI-1.11.0-2.tar.gz) + + - [Dr. Memory for Linux/ARM (DrMemory-ARM-Linux-EABIHF-1.11.0-2.tar.gz)](https://github.com/DynamoRIO/drmemory/releases/download/release_1.11.0/DrMemory-ARM-Linux-EABIHF-1.11.0-2.tar.gz) + +# Tutorials and talks + + - [Dr. Fuzz/Dr. Memory/DynamoRIO tutorial at SecDec Nov 2016] +(http://dynamorio.org/tutorial-secdev16.html) ([Online slides](https://docs.google.com/presentation/d/1cpOvQ16AZZ674E5EPz0H_PHFeFII1z22xof2yCKZhjI/edit?usp=sharing)) + +**************************************************************************** +**************************************************************************** +*/
diff --git a/drmemory/docs/main.dox b/drmemory/docs/main.dox index d9cb5ab..75fcf23 100644 --- a/drmemory/docs/main.dox +++ b/drmemory/docs/main.dox
@@ -41,7 +41,7 @@ # Downloading Dr. Memory -Dr. Memory is released under [an LGPL license](page_license.html). Windows, Linux, and Mac packages [are available for download](https://github.com/DynamoRIO/drmemory/wiki/Downloads). The sources are also [browsable](https://github.com/DynamoRIO/drmemory). +Dr. Memory is released under [an LGPL license](page_license.html). Windows, Linux, and Mac packages [are available for download](\ref page_download). The sources are also [browsable](https://github.com/DynamoRIO/drmemory). # Dr. Memory Performance
diff --git a/drmemory/docs/new_release.dox b/drmemory/docs/new_release.dox new file mode 100644 index 0000000..e08ff44 --- /dev/null +++ b/drmemory/docs/new_release.dox
@@ -0,0 +1,185 @@ +/* ********************************************************** + * Copyright (c) 2010-2021 Google, Inc. All rights reserved. + * **********************************************************/ + +/* Dr. Memory: the memory debugger + * + * This library is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; + * version 2.1 of the License, and no later version. + + * This library is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Library General Public License for more details. + + * You should have received a copy of the GNU Lesser General Public + * License along with this library; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +/** + **************************************************************************** + **************************************************************************** +\page page_new_release New Release + +# Versioning + +We use the same [versioning scheme as DynamoRIO](https://dynamorio.org/page_new_release.html). + +# Steps for Publishing a New Release + +## Sanity Testing + +It is a good idea to perform some sanity tests on all platforms beyond what our CI tests to catch issues before a new release. Minimal tests on each platform include: + +- Installing from the .msi installer +- Verifying that all of the Start Menu links work properly +- Verifying that Dr. Memory is added to the user's path +- Dragging calc.exe onto the Dr. Memory desktop icon (on Win10, drag notepad instead as calc is launched via service) +- Installing from the zip file +- Running something like tests/free.exe from the cygwin and cmd command line +- Ensure installer set up as a Visual Studio External Tool. Try running on a sample project that contains bugs. Try to test this with as many versions of Visual Studio ({2005, 2008, 2010, 2012, 2013} x {Professional, Express}) as possible. +- Run drstrace: `drstrace -- calc` and look at the log file +- Run symquery: `symquery -e free.exe -s main` + +## Commit the Version and Changelist Changes + +Update the following in the code and commit the change: + ++ Update `VERSION_NUMBER_DEFAULT` ++ Potentially update `DynamoRIO_VERSION_REQUIRED` ++ Potentially update `DRMF_VERSION_DEFAULT` ++ Update the default version number in `ci-package.yml` ++ Update the release changelist. + +## Trigger a Package Build + +To create a new build: + +1. Go to the Actions tab for the DynamoRIO/drmemory repository. +2. Click on "ci-package" on the list of workflows on the left. +3. Click on "Run workflow" on the right. +4. Select the branch. +5. Fill in the version number. For a test build or periodic build (i.e., not an official new-version-number release), use the current version major.minor and the date-based patchlevel. The patchlevel can be found from CMake's configuration output by running `cmake .`: +``` +-- Dr. Memory version number: 2.3.18609 +``` +For the build number, start at 0 for any given version number and add 1 for each attempt to produce a successful build. If the build number is 0 it is not appended to the version number. + +Wait for the workflow to finish. It should create a new tag, a new Release, upload 4 binary packages to the Release, and update the online documentation. + +## Verify Package and Documentation Uploads + +Double-check that all binary package files were uploaded to the new release on Github. + +## Trigger a Documentation Build + +Run the "ci-docs" Action in the same manner as the "ci-package" action above. +Double-check that the documentation was pushed to http://drmemory.org. + +## Update the Release Description + +Once the build is finished and posted to https://github.com/DynamoRIO/drmemory/releases, edit the description to list the highlights of the new release. You can use markdown to make a bulleted list. + +## Update download.dox + +Make a new list of direct links for the new version. + +## Add a New Changelist Section + +Update the changelist in release.dox: create a new entry for the next release. + +# Old Instructions for Locally Building Packages + +## Setup for Building Release Packages + +First, set up a symlink or NTFS junction such that the source path to Dr. Memory looks "professional", for the replace_malloc frames. For example, here is what I did on Windows: + +``` +cd /d +cmd /c mklink /J drmemory_package 'd:\derek\drmemory\trunk' +``` + +And on Linux: + +``` +ln -s /work/drmemory/drmemory/src /work/drmemory_package +``` + +When I build pointing at d:/drmemory_package/package.cmake, error messages then look like this: + +``` +Error #4: INVALID HEAP ARGUMENT to free 0x00001230 +# 0 replace_free [# 1 main [d:\derek\drmemory\git\src\tests\malloc.c:175](d:\drmemory_package\common\alloc_replace.c:2352]) +Note: @0:00:00.344 in thread 3224 +``` + +Next, be sure to use doxygen >= 1.8.1 for high-quality documentation. On Windows, use a non-Cygwin build (I use ftp://ftp.stack.nl/pub/users/dimitri/doxygen-1.8.1.1.windows.bin.zip, with doxygen.exe placed in /usr/local/bin: see [DR issue 815](https://github.com/DynamoRIO/dynamorio/issues/815)). + +On Windows, we prefer our WIX-based MSI installer over the older NSIS installer. Ensure you have at least CMake 3.3.0, WIX is installed, and WIX's `candle.exe` is on your PATH. The configure status lines should then indicate the selection of WIX instead of NSIS for the installer build. + +To instead build an NSIS-based installer, ensure you have NSIS installed with large string support so that the installer will be properly built. Configure status lines will indicate this. See [the build instructions](How-To-Build#other-tools) for more information. + +## Building Candidate Packages + +Build using package.cmake, and ensure your drmemory_package sources are up to date. On Windows: + +``` +cd ~/drmemory/build_package/ +compilerVS2010_32 +ctest -VV -S d:/drmemory_package/package.cmake,drmem_only\;build=2\;cacheappend=TOOL_VERSION_NUMBER:STRING=1.11.0\;cacheappend=DRMF_VERSION:STRING=1.0.0\;use_ninja\;cpackappend=set\(CPACK_PACKAGE_FILE_NAME\ DrMemory-Windows-1.11.0-2\)\;cacheappend=VERSION_NUMBER:STRING=6.2.0 +chmod ugo+rx *.msi +``` + +On Linux: + +``` +cd /work/drmemory/build_package/ +ctest -V -S ctest -V -S /work/drmemory_package/package.cmake,drmem_only\;build=2\;cacheappend=TOOL_VERSION_NUMBER:STRING=1.11.0\;cacheappend=DRMF_VERSION:STRING=1.0.0\;cpackappend=set\(CPACK_PACKAGE_FILE_NAME\ DrMemory-Linux-1.11.0-2\)\;cacheappend=VERSION_NUMBER:STRING=6.2.0 +``` + +ARM eabihf: +``` +cd /work/drmemory/build_package/ +ctest -D CMAKE_SYSTEM_PROCESSOR=arm -V -S /work/drmemory_package/package.cmake,drmem_only\;32_only\;build=2\;cacheappend=TOOL_VERSION_NUMBER:STRING=1.11.0\;cacheappend=DRMF_VERSION:STRING=1.0.0\;cacheappend=CMAKE_TOOLCHAIN_FILE=/work/drmemory/git/src/dynamorio/make/toolchain-arm32.cmake\;cpackappend=set\(CPACK_PACKAGE_FILE_NAME\ DrMemory-ARM-Linux-EABIHF-1.11.0-2\)\;cacheappend=VERSION_NUMBER:STRING=6.2.0 +``` + +Android: + +``` +cd /work/drmemory/build_package/ +ctest -D CMAKE_SYSTEM_PROCESSOR=arm -V -S /work/drmemory_package/package.cmake,drmem_only\;32_only\;build=2\;cacheappend=TOOL_VERSION_NUMBER:STRING=1.11.0\;cacheappend=DRMF_VERSION:STRING=1.0.0\;cacheappend=CMAKE_TOOLCHAIN_FILE=/work/drmemory/git/src/dynamorio/make/toolchain-android.cmake\;cacheappend=ANDROID_TOOLCHAIN=/work/toolchain/android-ndk-21\;cpackappend=set\(CPACK_PACKAGE_FILE_NAME\ DrMemory-ARM-Android-EABI-1.11.0-2\)\;cacheappend=VERSION_NUMBER:STRING=6.2.0 +``` + +On MacOS: + +``` +cd ~/drmemory/build_package +ctest -V -S /drmemory_package/package.cmake,drmem_only\;32_only\;build=2\;cacheappend=TOOL_VERSION_NUMBER:STRING=1.11.0\;cacheappend=DRMF_VERSION:STRING=1.0.0\;cpackappend=set\(CPACK_PACKAGE_FILE_NAME\ DrMemory-MacOS-1.11.0-2\)\;cacheappend=VERSION_NUMBER:STRING=6.2.0 +``` + +Increment the build number for each RC. + +## Tag and Post + +Tag the release point: + +``` +git tag release_1.9.0 09e3d62 +git push origin --tags +``` + +Then make a new release on github at https://github.com/DynamoRIO/drmemory/releases. Point at the precise changelist version in the online source viewer (look at prior releases for examples). + +Update download.dox + +Then commit to the top-level CMakeLists.txt (yes, *after* the tag) to update the line that sets `VERSION_NUMBER_DEFAULT`, potentially the lines that set `DynamoRIO_VERSION_REQUIRED` and `DRMF_VERSION_DEFAULT`, and the release changelist. + +## Update Documentation + +Update the online docs. +**************************************************************************** +**************************************************************************** +*/
diff --git a/drmemory/docs/projects.dox b/drmemory/docs/projects.dox new file mode 100644 index 0000000..5b8983c --- /dev/null +++ b/drmemory/docs/projects.dox
@@ -0,0 +1,398 @@ +/* ********************************************************** + * Copyright (c) 2010-2021 Google, Inc. All rights reserved. + * **********************************************************/ + +/* Dr. Memory: the memory debugger + * + * This library is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; + * version 2.1 of the License, and no later version. + + * This library is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Library General Public License for more details. + + * You should have received a copy of the GNU Lesser General Public + * License along with this library; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +/** + **************************************************************************** + **************************************************************************** +\page page_projects Project Suggestions for Dr. Memory Contributors + +# General Prerequisites for Contributors + +Dr. Memory and its underlying engine, DynamoRIO, are written in C. Knowledge of C programming is required for all of the projects listed here. Some projects that involve external pieces of software, such as a Visual Studio plugin, may additionally require C++ programming. + +Some lower-level projects need knowledge of x86/amd64 assembly code (though mainly just knowledge of the architecture, register set, and instruction set: experience writing in a particular assembler is not required), or program analysis and reverse engineering skills. Each project lists the particular skill set most suited to work on that project. + +Knowledge of operating system fundamentals and experience with runtime systems are helpful for all of the projects here. + +# Project Suggestions for Contributors + +## Dr. Memory debugger integration + +The first step of this project would involve Visual Studio IDE integration. Dr. Memory can currently be used as an External Tool in the Visual Studio +IDE, but it requires manual setup. We'd like to automate that and have Dr. Memory auto-registered as an External Tool during installation. + +The next step of this project will integrate with the Visual Studio debugger. While Dr. Memory preserves transparency of as much as the application as possible, including the application stack, when examined in a debugger the program counter points into the software code cache used to implement Dr. Memory's instrumentation and not into the original application code. This can be confusing to a user. + +This part of the project involves first connecting Dr. Memory to the debugger by creating a Visual Studio plugin. The next step involves translating the program counter from the code cache to the corresponding application address whenever it is presented to the user. In addition, at certain points during the execution, the full register state must be translated. + +The final step involves enabling a Dr. Memory-reported error to break into the debugger and allow the user to examine the application state at the point of the error. + +__Prerequisites__: The Visual Studio plugin will require C++ programming. Converting the internal tool state to the presented debugger state will require C programming and x86/amd64 knowledge. + +See also: + - [issue 800](https://github.com/DynamoRIO/drmemory/issues/800) + - [issue 1009](https://github.com/DynamoRIO/drmemory/issues/1009) + - [issue 600](https://github.com/DynamoRIO/drmemory/issues/600) + - [issue 532](https://github.com/DynamoRIO/dynamorio/issues/532) + - [issue 559](https://github.com/DynamoRIO/dynamorio/issues/559) + - [Advanced debugging tools](Projects#advanced-debugging-tools) + +Similar work could be done for gdb or any other debugger. + +## Standalone GUI for running Dr. Memory + +Currently, Dr. Memory's main interface is command-line-only. On Windows we also support drag-and-drop onto our icon, but no arguments can be passed that way. After a manual setup step, Dr. Memory can also be run from within Visual Studio (see the prior project on improving that). + +This project involves building a graphical interface for launching Dr. Memory (or any DynamoRIO-based tool) and viewing the results. For supporting general tools (such as code coverage, e.g.), the project could involve designing a client API for displaying arbitrary tool results in the GUI. + +__Prerequisites__: This project could be written entirely in C++ if desired and is one of the most high-level projects on the list, requiring the least low-level or architecture-specific knowledge. It could build on MFC or any desired graphical application framework. + +See also: + - [issue 1128](https://github.com/DynamoRIO/drmemory/issues/1128) + - [issue 1150](https://github.com/DynamoRIO/drmemory/issues/1150) + +## Dr. Heapstat + +Today we have a prototype of a heap usage and "heap staleness" tool. There +is much work to be done in flushing it out. + +"Staleness": a memory profiling tool that leverages dynamic instrumentation +to gather "staleness information" about the lifetime and access history +of heap objects, guiding memory usage improvements. + +Visualizing the resulting data is a big part of making a tool like this +successful. + +Another facet of this project could be to bridge domains in dual-language +applications. For example, bridge the Javascript and C++ domains, or +the Java and C++ domains. + +__Prerequisites__: The visualizer could be written in a high-level language, such as Java. The data-gathering tool needs C programming experience and some x86/amd64 assembly knowledge. + +## Partial native execution for hybrid tools + +Dr. Memory's light mode does not need to execute the entire process, +meaning it can be applied to only part of the execution. Dr. Memory can +also be used as the dynamic portion of a hybrid tool that uses +compiler-inserted instrumentation where possible. Performance can be +improved for these situations by natively executing the portion of the +application not being monitored by Dr. Memory + +__Prerequisites__: C programming and x86/amd64 knowledge. + +See also: + - [issue 978](https://github.com/DynamoRIO/dynamorio/issues/978) + +## Port Dr. Memory to MacOS + +Add support for MacOS to both the DynamoRIO platform and the Dr. Memory +tool built on top of it. As DynamoRIO is a low-level system that operates +at the system call layer this involves solving challenging problems unique +to MacOS. + +__Prerequisites__: C programming and MacOS or general UNIX operating system-level knowledge. + +See also: + - [issue 58](https://github.com/DynamoRIO/dynamorio/issues/58) + +## Port Dr. Memory to ARM + +Port DynamoRIO and Dr. Memory to the ARM architecture. + +__Prerequisites__: C programming and ARM architectural knowledge or possibly general computer architecture knowledge. + +## Provide further information on the origins of uninitialized read bugs + +Dr. Memory must wait for a "meaningful" read before reporting an +uninitialized read error in order to avoid false positives. But this can +make it harder to track backward to the source of the bug. We would like +to optionally track extra information on the origins of these bugs. + +__Prerequisites__: C programming and x86/amd64 assembly knowledge. + +See also: + - [issue 170](https://github.com/DynamoRIO/drmemory/issues/170) + +## Extend Dr. Memory's Windows system call database + +Analyze Windows system calls in order to remove false positives from +Dr. Memory's error reports. + +__Prerequisites__: Program analysis and reverse engineering skills; Windows operating system knowledge. + +See also: + - [issue 98](https://github.com/DynamoRIO/drmemory/issues/98) + - [issue 424](https://github.com/DynamoRIO/drmemory/issues/424) + - [issue 437](https://github.com/DynamoRIO/drmemory/issues/437) + - [issue 1093](https://github.com/DynamoRIO/drmemory/issues/1093) + - [issue 1094](https://github.com/DynamoRIO/drmemory/issues/1094) + - [issue 1095](https://github.com/DynamoRIO/drmemory/issues/1095) + +## Analyze and eliminate Windows leak false positives + +On Windows, Dr. Memory reports a number of leaks involving system libraries. We're not sure of the cause of these leaks being reported and which ones are true leaks versus false positives due to non-standard pointer treatment in the library code. + +__Prerequisites__: Program analysis and reverse engineering skills; Windows operating system knowledge. + +## Extend Dr. Memory's Linux system call database + +Add Linux system call information in order to remove false positives from +Dr. Memory's error reports. + +__Prerequisites__: Linux operating system knowledge. + +See also: + - [issue 92](https://github.com/DynamoRIO/drmemory/issues/92) + - [issue 1019](https://github.com/DynamoRIO/drmemory/issues/1019) + - [issue 1106](https://github.com/DynamoRIO/drmemory/issues/1106) + +## Add post-processing and multi-run-aggregation features to Dr. Memory + +We would like to be able to re-symbolize, re-suppress, or combine results +from prior runs under Dr. Memory. + +__Prerequisites__: C programming. + +See also: + - [issue 446](https://github.com/DynamoRIO/drmemory/issues/446) + - [issue 614](https://github.com/DynamoRIO/drmemory/issues/614) + +## Attach/detach on Linux or Windows + +Some modes of Dr. Memory do not need to observe the entire program +execution. Attaching after program startup can improve performance for +repeated start/stop testing of large applications. + +__Prerequisites__: C programming, either Linux or Windows operating system knowledge, and x86/amd64 assembly knowledge. + +See also: + - [issue 37](https://github.com/DynamoRIO/dynamorio/issues/37) + - [issue 38](https://github.com/DynamoRIO/dynamorio/issues/38) + - [issue 95](https://github.com/DynamoRIO/dynamorio/issues/95) + - [issue 725](https://github.com/DynamoRIO/dynamorio/issues/95) + +## First-instruction injection + +Currently, DynamoRIO does not take over until some system library +initialization is complete. This feature covers taking over at the very +first instruction, on both Linux and Windows. This would simplify many +aspects of Dr. Memory. + +__Prerequisites__: C programming, either Linux or Windows operating system knowledge, and x86/amd64 assembly knowledge. + +See also: + - [issue 234](https://github.com/DynamoRIO/dynamorio/issues/234) + +## Cross-architecture process following + +Currently, Dr. Memory is unable to monitor a child process of a different +bitwidth (32 vs 64) from the parent process. + +__Prerequisites__: C programming, Windows operating system knowledge, and x86/amd64 assembly knowledge. + +See also: + - [issue 49](https://github.com/DynamoRIO/dynamorio/issues/49) + +## Annotation infrastructure + +Annotations in the application would provide many benefits to Dr. Memory. + +__Prerequisites__: C programming. + +See also: + - [issue 283](https://github.com/DynamoRIO/drmemory/issues/283) + - [issue 41](https://github.com/DynamoRIO/drmemory/issues/41) + - [issue 572](https://github.com/DynamoRIO/drmemory/issues/572) + - [issue 573](https://github.com/DynamoRIO/drmemory/issues/573) + +## Persistent cache + +Persistent code caches can improve startup performance for Dr. Memory when +repeatedly running large applications. Some work has been done in this +area but it is not complete. + +__Prerequisites__: C programming, x86/amd64 knowledge. + +See also: + - [issue 769](https://github.com/DynamoRIO/drmemory/issues/769) + - [issue 867](https://github.com/DynamoRIO/drmemory/issues/867) + - [issue 40](https://github.com/DynamoRIO/dynamorio/issues/40) + +## System call tracing tool + +Dr. Strace is our prototype system call tracing tool for Windows. We're +missing all of the non-pointer arguments for all graphical system calls and +many recent system calls, however, and the tool is in rudimentary state: it +does not display field details of most complex parameter types. + +__Prerequisites__: C programming, Windows operating system knowledge. + +## Build better performance analysis tools for analyzing Dr. Memory itself + +We need better tools to analyze performance of Dr. Memory and other +DynamoRIO-based tools. + +__Prerequisites__: C programming, computer architecture knowledge. + +See also: + - [issue 140](https://github.com/DynamoRIO/dynamorio/issues/140) + - [Profiling DynamoRIO and Clients](https://dynamorio.org/page_profiling.html) + +## Better cache consistency handling + +Dynamically generated code is a challenge for Dr. Memory and DynamoRIO to +handle. We would like to improve our cache consistency handling by using a dual page map scheme that arranges for writable code to be backed by a file that is mapped twice, one read-only and one writable. + +__Prerequisites__: C programming, x86/amd64 knowledge. + +See also: + - [issue 1114](https://github.com/DynamoRIO/dynamorio/issues/1114) + +## Run 32-bit applications in 64-bit mode to improve Dr. Memory performance + +Overview: We would like to experiment with Dr. Memory switching to 64-bit mode when running a 32-bit application on a 64-bit kernel. We could use the extra registers as scratch space to reduce spills and improve performance. Dr. Memory could also use the extra address space for more efficient memory shadowing. + +Current status: Currently, DynamoRIO has basic support for running mixed-mode (i.e., mixing 32-bit and 64-bit code), though some corner cases are not covered. This is a special case of mixed-mode, though, where the application has rigid boundaries of its code transitions, which eliminates many corner cases. + +The next step is to scale it up and shift to Dr. Memory. We need to handle callbacks and other transitions through the WOW64 layer, and ensure we preserve r12-r15, which are assumed to be untouched on every re-entry to WOW64 from 32-bit. + +We would start with Dr. Memory’s pattern mode, as that does support 64-bit applications today. + + - Goal: improve performance of Dr. Memory on 32-bit applications. + - Platforms: Windows. (32-bit is not very relevant on Linux.) + - Milestones: + 1. 32-to-64 DynamoRIO running Windows Hello,World + 1. 32-to-64 DynamoRIO running calc.exe + 1. Finalize 32-to-64 client API: resolve issues such as whether clients must handle 32-bit instrlists or can assume all-64 (forcing us to translate things like BCD). + 1. 32-to-64 Dr. Memory pattern mode running calc.exe + 1. 32-to-64 Dr. Memory pattern mode running Chromium browser_tests + 1. 32-to-64 Dr. Memory shadow full mode running Chromium browser_tests + +__Prerequisites__: C programming, x86/amd64 knowledge. + +See also: + - [issue 751](https://github.com/DynamoRIO/dynamorio/issues/751) + +## Dynamic register stealing and coordination framework + +Improve Dr. Memory instrumentation by building a framework for dynamic +register usage coordination for multi-component dynamic tools. + +__Prerequisites__: C programming, x86/amd64 knowledge, ideally experience with a compiler framework such as gcc or LLVM. + +See also: + - [issue 511](https://github.com/DynamoRIO/dynamorio/issues/511) + +## Add a buffer filling API to DynamoRIO + +The design could be based on the +[PiPA academic tool](http://dynamorio.org/pubs/PiPA-pipelined-profiling-cgo08.pdf). + +__Prerequisites__: C programming, x86/amd64 knowledge. + +See also: + - [issue 513](https://github.com/DynamoRIO/dynamorio/issues/513) + +## Advanced debugging tools + +This project involves extending existing debuggers with dynamic instrumentation to support novel capabilities in a performant manner. + +First, we would implement a communication and extension protocol for the chosen debugger: gdb's remote stub API, or a Visual Studio plugin (see also +[the Dr. Memory debugger integration project](Projects#dr-memory-debugger-integration) above). + +Second, we would implement state translation to present the application state rather than the software code cache state used by DynamoRIO to implement dynamic instrumentation. (See also [issue 532](https://github.com/DynamoRIO/dynamorio/issues/532) and +[issue 559](https://github.com/DynamoRIO/dynamorio/issues/559)). + +Finally, once we have a baseline debugger integration to build off of, a wide variety of debugger improvements can be implemented using the power of dynamic instrumentation: + + - Thread-specific breakpoints (or watchpoints). On applications with many threads, a thread-specific breakpoint in a traditional debugger is inefficient. With dynamic instrumentation we can create a thread-private software code cache for a target thread and implement a breakpoint with zero overhead on the other threads. + - More powerful and scalable watchpoints. Using dynamic instrumentation and shadow memory we can support literally millions of watchpoints, while a traditional debugger is limited to the hardware registers, beyond which it typically goes into single-step mode. + - Apply arbitrary customized tools to selected code sequences while debugging. For example, an uninitialized read detector, or a memory overflow detector, could be invoked while debugging from the debugger and applied to a sequence of code. A memory trace could be gathered, an instruction trace, etc., all while debugging. + - Faster conditional breakpoints. A traditional debugger interrupts the program every time to check whether a breakpoint's condition has been met. Dynamic instrumentation can inline the condition and only break out when it is met. + - Reverse execution + - Dynamic slicing + +__Prerequisites__: C programming, x86/amd64 knowledge. + +See also: + + - [Dr. Memory debugger integration](Projects#dr-memory-debugger-integration) + - Academic paper: ["How to do a million watchpoints: Eļ¬cient Debugging using Dynamic Instrumentation"](http://dynamorio.org/pubs/zhao-million-watchpoints-CC08.pdf) + - Academic paper: ["PinADX: an interface for customizable debugging with dynamic instrumentation"](http://dl.acm.org/citation.cfm?id=2259032) + +## Probe mode + +Complete the DynamoRIO Probe API for lighter-weight tools that only need to +insert probes/callouts/hooks at certain points during execution + +__Prerequisites__: C programming, x86/amd64 knowledge. + +See also: + - [issue 1281](https://github.com/DynamoRIO/dynamorio/issues/1281) + +## Add DynamoRIO IPC support + +We would like to augment DynamoRIO's IPC API support, as using native API's +can result in no alertable syscalls or other problems. We'd like to add +shared memory, pipes, and semaphores in a Linux + Windows cross-platform +manner. + +__Prerequisites__: C programming, either Linux or Windows operating system knowledge, x86/amd64 knowledge. + +See also: + - [issue 498](https://github.com/DynamoRIO/dynamorio/issues/498) + +## DynamoRIO performance optimizations + +Hashtable and indirect branch lookup optimizations that may improve +DynamoRIO's performance. + +__Prerequisites__: C programming, x86/amd64 knowledge. + +See also: + - [issue 31](https://github.com/DynamoRIO/dynamorio/issues/31) + - [issue 32](https://github.com/DynamoRIO/dynamorio/issues/32) + - [issue 33](https://github.com/DynamoRIO/dynamorio/issues/33) + +## Create a new tool + +Create a dynamic tool using the DynamoRIO tool platform to use as a sample. +Particular tools could include a fuzzer, a profiler (basic block, edge, +function, parallelized profiling, etc.), some kind of reverse engineering +tool, etc. + +__Prerequisites__: C or C++ programming, ideally x86/amd64 knowledge. + +## Create a tool library + +Create a library for dynamic tools to use. Particular needs include +building a control flow graph, building a call graph, maintaining a shadow +stack, or data dependence analysis. + +__Prerequisites__: C or C++ programming. + +## Other, smaller features + +Search the issue tracker for the ["help wanted"](https://github.com/DynamoRIO/drmemory/issues?utf8=%E2%9C%93&q=is%3Aissue%20is%3Aopen%20label%3A%22help%20wanted%22) and ["good first issue"](https://github.com/DynamoRIO/drmemory/issues?utf8=%E2%9C%93&q=is%3Aissue%20is%3Aopen%20label%3A%22good%20first%20issue%22%20) labels. + +**************************************************************************** +**************************************************************************** +*/
diff --git a/drmemory/docs/test.dox b/drmemory/docs/test.dox new file mode 100644 index 0000000..314745e --- /dev/null +++ b/drmemory/docs/test.dox
@@ -0,0 +1,180 @@ +/* ********************************************************** + * Copyright (c) 2010-2021 Google, Inc. All rights reserved. + * **********************************************************/ + +/* Dr. Memory: the memory debugger + * + * This library is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; + * version 2.1 of the License, and no later version. + + * This library is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Library General Public License for more details. + + * You should have received a copy of the GNU Lesser General Public + * License along with this library; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +/** + **************************************************************************** + **************************************************************************** +\page page_test Test Suite + +# Automated Test Machines + +We use Github Actions with a setup that is similar to that [described for DynamoRIO[(https://dynamorio.org/page_test_suite.html). + +# Test Suite + +The test suite for Dr. Memory resides in the tests/ directory. +The tests are applications that we execute under control of Dr. Memory. + +## Building and Running Tests + +The tests are built by default along with the rest of Dr. Memory. + +CTest is used to run the tests. After building, you can use the `test` +make target to run them: + +``` + make -j6 + make test +``` + +You can also invoke `ctest` directly, which gives greater control over +which subsets of the tests are run. Use `ctest -N` to see the list of +tests and which number is assigned to each. Then you can use `ctest -I +x,y` to run all tests from number x to number y, and the -V parameter to +display the command line and the output of the test. For example: + +``` + ctest -V -I 49,49 +``` + +You can also specify tests using inclusion and exclusion regular +expressions. For example: + +``` + ctest -R 'suppress|realloc' +``` + +will run all tests with the string "suppress" or the string "realloc" in their +names, while + +``` + ctest -E alloc +``` + +will run all tests except those with alloc in their names. + +If you are using CTest version 2.8 or later, you can run tests in +parallel by passing -jN to `ctest` on the command line: + +``` + ctest -j5 +``` + +As an alternative to using -V to display the test command line and output +during executing, CTest stores that information (even when not run with -V) +in the `Testing/Temporary/LastTest.log` file in the build directory. + +## Test Output + +Each test produces two types of output: stdout and the Dr. Memory results +file. Both are checked against expected output, stdout versus a .out file +and the results file versus a .res file. The checking is not a regular +diff and simply looks for each line. We may want to change this to a +stronger diff with regular expression support in the future. + +## Pre-Commit Test Suite + +The tests/runsuite.cmake script is used to execute a series of builds and +test runs. Running `make test` in a single build directory is not +sufficient to test all of the configurations that we support. + +The runsuite.cmake script is meant to be executed from an empty directory. +It creates a subdirectory for each build in the suite. Use `ctest -S` to +execute the script. Here is an example: + +``` + mkdir ../build_suite + cd ../build_suite + ctest -S ../drmemory/tests/runsuite.cmake +``` + +You can pass -V to ctest to see the results as the test suite runs. Note +that it is normal to have ctest output strings such as "No tests were +found!!!" as we have builds for which we run no tests (particularly in the +short suite). It is also expected to have an error at the end of the +suite (if run with -V): + +``` +CMake Error: Some required settings in the configuration file were missing: +CTEST_SOURCE_DIRECTORY = E:/derek/drmemory/src/suite/.. +CTEST_BINARY_DIRECTORY = (Null) +CTEST_COMMAND = c:/PROGRA~2/CMAKE2~1.6/bin/ctest.exe +``` + +This warning is an artifact of how CTest assumes we've set things up and +can be ignored. + +At the end of the suite a file called results.txt will be created and +displayed. It shows configure failures, build failures, and test failures. + +Here are sample results from running the test suite: + +``` +### ============================================ + +RESULTS + +drheapstat-dbg-32: all 19 tests passed +drheapstat-rel-32: build successful; no tests for this build +drmemory-dbg-32: all 23 tests passed +drmemory-rel-32: build successful; no tests for this build +final package: build successful; no tests for this build + +Error in read script: /home/bruening/work/build/drmem_suite/src/tests/runsuite.cmake +``` + +The "Error in read script" is expected: ignore it. + +Here is an example of running the test suite on Windows from the cmd shell +on a new checkout, with the -V flag for more verbose output: + +``` +cd "c:\Program Files (x86)\Microsoft Visual Studio *" +VC\vcvarsall.bat + +set DDKROOT=C:/derek/ddk +set FLEXROOT=C:/derek/flex_sdk_4.1 + +cd \derek\drmemory +git clone https://github.com/DynamoRIO/drmemory.git +cd drmemory +make/git/devsetup.sh + +cd \derek\drmemory\build_suite + +"c:\Program Files (x86)\CMake 2.8\bin\ctest.exe" -V -S c:/derek/drmemory/src/tests/runsuite.cmake +``` + +## Embedded Versus Separate DynamoRIO + +DynamoRIO is included inside the Dr. Memory source tree as an svn:externals property. We support both building that embedded DR along with pointing to an already-built separate DR export tree. Whenever a new feature from DynamoRIO is used within Dr. Memory, be sure to update the svn:externals following the instructions at UpdatingDR. In addition, be sure to run the pre-commit test suite using the embedded DR, or be very careful if using an external DR that it matches the svn:externals version. + +## Cross-Compilation and Android Testing + +The test suite includes cross-compilation tests to ensure that the ARM and Android builds are not broken. If a cross-compiler is not found on the PATH, these builds will fail, but they are considered optional, so the whole suite will not be considered a failure. However, we recommend installing the cross-compilers and placing them on your PATH for more thorough testing. + +If you have an Android device set up for access through the `adb shell` utility, the Android build is capable of automatically copying binaries to the device and running tests. If both the Android cross-compiler and `adb` are on your PATH, and `adb status` indicates an attached device, the tests will be run. + + + +**************************************************************************** +**************************************************************************** +*/
diff --git a/make/git/git_review.sh b/make/git/git_review.sh index dab7e99..8422473 100755 --- a/make/git/git_review.sh +++ b/make/git/git_review.sh
@@ -1,7 +1,7 @@ #!/bin/bash # ********************************************************** -# Copyright (c) 2014-2017 Google, Inc. All rights reserved. +# Copyright (c) 2014-2021 Google, Inc. All rights reserved. # ********************************************************** # Redistribution and use in source and binary forms, with or without @@ -40,7 +40,7 @@ # Check whether a number. [ "${prefix:1}" -eq "${prefix:1}" ] 2>/dev/null); then echo "ERROR: please use the branch naming conventions documented at " - echo "https://github.com/DynamoRIO/drmemory/wiki/Workflow#branch-naming-conventions" + echo "https://dynamorio.org/page_workflow.html#sec_branch_naming" exit 1 fi