| /* ****************************************************************************** |
| * Copyright (c) 2010-2021 Google, Inc. All rights reserved. |
| * ******************************************************************************/ |
| |
| /* |
| * Redistribution and use in source and binary forms, with or without |
| * modification, are permitted provided that the following conditions are met: |
| * |
| * * Redistributions of source code must retain the above copyright notice, |
| * this list of conditions and the following disclaimer. |
| * |
| * * Redistributions in binary form must reproduce the above copyright notice, |
| * this list of conditions and the following disclaimer in the documentation |
| * and/or other materials provided with the distribution. |
| * |
| * * Neither the name of Google, Inc. nor the names of its contributors may be |
| * used to endorse or promote products derived from this software without |
| * specific prior written permission. |
| * |
| * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" |
| * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE |
| * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE |
| * ARE DISCLAIMED. IN NO EVENT SHALL VMWARE, INC. OR CONTRIBUTORS BE LIABLE |
| * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL |
| * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR |
| * SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER |
| * CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT |
| * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY |
| * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH |
| * DAMAGE. |
| */ |
| |
| /** |
| **************************************************************************** |
| \page page_profiling Profiling DynamoRIO and Clients |
| |
| # Linux |
| |
| ## DynamoRIO PC self-sampling |
| |
| The client can use dr_set_itimer() for programmatic PC self-sampling, with dr_where_am_i() providing information on where the sample was taken. This provides general categorization of where time is being spent in the overall instrumentation system, with potential to drill down further offline based on the PC. |
| |
| For PC sampling via DR's `-prof_pcs` runtime option instead, that is available internally in varying degrees on different platforms but is not polished enough and is missing some pieces (see the bottom of this page). |
| |
| ## External sampling tools |
| |
| Perf and oprofile are the two prominent sampling profilers on Linux today. Perf is newer and has a nicer interface, but it requires patching and building from source in order to get symbols for DR. oprofile is typically available on older distros, but it's not available on Ubuntu Precise, it seems to cause system lockups, and we're not sure we trust the results. |
| |
| Before doing any micro-optimization based on the profile, make sure to disable CPU frequency scaling before taking measurements: |
| ``` |
| for N in /sys/devices/system/cpu/cpu[; do echo performance | sudo tee $N/cpufreq/scaling_governor ; done |
| ``` |
| |
| ### oprofile |
| |
| To install oprofile, type: |
| ``` |
| # Or other distro command |
| sudo apt-get install oprofile |
| # We don't need to profile the kernel |
| sudo /usr/bin/opcontrol --no-vmlinux |
| ``` |
| |
| To make sudo opcontrol work w/o a password, type `sudo visudo` and add one line to the `/etc/sudoers` file: |
| ``` |
| your_username ALL=NOPASSWD: /usr/bin/opcontrol |
| ``` |
| |
| To run oprofile, you can use a script like the following to start and stop it around the command you wish to run: |
| ``` |
| sudo opcontrol --shutdown || true # Shutdown existing oprofile daemon, if any |
| sudo opcontrol --reset # Throw away previously collected data |
| sudo opcontrol -c 0 # Replace 0 with N if you need callgraph |
| sudo opcontrol --start |
| your_command_here |
| sudo opcontrol --stop |
| sudo opcontrol --dump # Dumps data into local file |
| |
| # Now get the report: |
| opreport -t 1 -l object_file_to_get_stats_for |
| # or to get the callgraph (don't forget to -c N above!) |
| opreport -c -l object_file_to_get_stats_for |
| ``` |
| |
| Example report output: |
| ``` |
| [rnk@wittenberg src](0-9])$ opreport -t 2 -l ../../dynamorio/build/install/lib64/release/libdynamorio.so.3.2 |
| ... |
| samples % symbol name |
| 4971 7.1208 insert_exit_stub_other_flags |
| 2107 3.0182 mutex_lock |
| 2082 2.9824 decode_sizeof |
| 1774 2.5412 build_bb_ilist |
| 1695 2.4280 encoding_possible_pass |
| 1653 2.3679 fragment_lookup_fine_and_coarse |
| 1647 2.3593 instr_encode_common |
| 1502 2.1516 dispatch |
| 1399 2.0040 hashtable_fragment_lookup |
| ``` |
| |
| ### perf |
| |
| Perf currently does not handle symbols in DSOs that have a preferred base, and they only recently added support for following .gnu_debuglink. Since profiling without symbols isn't very useful, the following instructions are for building perf from source with a patch I wrote to fix the problem. |
| |
| The patch to get good symbols with perf is available here: |
| https://github.com/rnk/linux/compare/perf-p_vaddr.diff |
| |
| You can clone the entire branch, or you can apply the patch to some other copy of the Linux kernel source. Either way, cd into tools/perf and run 'make' to build just perf. It will warn you about each library or header that it can't find, and you can install the appropriate package. |
| |
| Running 'make install' as a normal user will install to $HOME/bin and $HOME/libexec. |
| |
| To do a run and get a report, it's quite simple: |
| ``` |
| perf record your_command # Stores result in cwd/perf.data |
| perf report |
| ``` |
| |
| Example output: |
| ``` |
| [src](rnk@wittenberg)$ perf report | head |
| # Overhead Command Shared Object Symbol |
| # ........ .............. ................... ........................................................................... |
| # |
| 17.68% DumpRenderTree perf-14440.map [0x0000000071c59213 |
| 5.14% DumpRenderTree libdynamorio.so.3.2 [.](.]) insert_exit_stub_other_flags |
| 4.51% DumpRenderTree libdynamorio.so.3.2 [hashtable_fragment_lookup.isra.31 |
| 2.27% DumpRenderTree libdynamorio.so.3.2 [.](.]) mutex_lock |
| 1.94% DumpRenderTree libdynamorio.so.3.2 [build_bb_ilist |
| 1.92% DumpRenderTree libdynamorio.so.3.2 [.](.]) encoding_possible_pass |
| ``` |
| |
| The perf-NNN.map DSO corresponds to DR's code cache. As you can see from above, at the time of writing, stub updating is a hotspot. You can focus in on just DR by passing "-d libdynamorio.3.2". |
| |
| To get a combined source and asm annotation, you can use "perf annotate -s insert_exit_stub_other_flags". Example output: |
| ``` |
| ... |
| ▒ |
| │ byte * ▒ |
| │ insert_relative_jump(byte *pc, cache_pc target, bool hot_patch) ▒ |
| │ { ▒ |
| │ ASSERT(pc != NULL); ▒ |
| │ **pc = JMP_OPCODE; ▒ |
| │ pc++; ◆ |
| 0.24 │ dc: lea 0x1(%rax),%rdx ▒ |
| │ ▒ |
| │ byte ** ▒ |
| │ insert_relative_jump(byte *pc, cache_pc target, bool hot_patch) ▒ |
| │ { ▒ |
| │ ASSERT(pc != NULL); ▒ |
| │ **pc = JMP_OPCODE; ▒ |
| 0.16 │ movb $0xe9,(%rax) ▒ |
| │ byte ** ▒ |
| │ insert_relative_target(byte **pc, cache_pc target, bool hot_patch) ▒ |
| │ { ▒ |
| │ /** insert 4-byte pc-relative offset from the beginning of the next instruction ▒ |
| │ **/ ▒ |
| │ int value = (int)(ptr_int_t)(target - pc - 4); ▒ |
| 0.16 │ sub %rdx,%r14 ▒ |
| │ sub $0x4,%r14d ▒ |
| │ IF_X64(ASSERT(CHECK_TRUNCATE_TYPE_int(target - pc - 4))); ▒ |
| │ ATOMIC_4BYTE_WRITE(pc, value, hot_patch); ▒ |
| │ xchg %r14d,(%rdx) ▒ |
| │ **((ptr_uint_t **)pc) = (ptr_uint_t)l; pc += sizeof(l); ▒ |
| │ #ifdef X64 ▒ |
| │ } ▒ |
| │ #endif ▒ |
| │ /** jmp to exit target */ ▒ |
| │ pc = insert_relative_jump(pc, exit_target, NOT_HOT_PATCHABLE); ▒ |
| 97.40 │ add $0x5,%rax |
| ``` |
| |
| 97% of the samples in this function were on "add $0x5, %rax", which is misleading. The expensive instruction is more likely the "xchgl %r14d, (%rdx)" before it, which instruction we use to atomically update the code cache. In this particular case, we happen to be emitting the full exit stub, so it's unlikely that this needs to be an atomic update. |
| |
| # Windows |
| |
| We've used Code Analyst successfully. |
| |
| TODO, more detail. |
| |
| # Cross-platform -prof_pcs |
| |
| There are many open issues for cleaning this up, such as [issue 140](https://github.com/DynamoRIO/dynamorio/issues#issue/140), [issue 359](https://github.com/DynamoRIO/dynamorio/issues#issue/359), [issue 767](https://github.com/DynamoRIO/dynamorio/issues#issue/767). On Linux the dr_set_itimer() solution above provides programmatic support. |
| |
| |
| **************************************************************************** |
| */ |
