| Building and using LTTng an enabled server |
| ========================================== |
| |
| LTTng is a big topic. Consult the online documentation for the |
| details of running the tools. The development is being done using Fedora |
| using their packages, version 2.3.0. |
| |
| Install the following RPMs on Fedora or, if on RHEL, use the EPEL repo. |
| There are other lttng related packages but these are the relevant ones |
| for this level. |
| |
| lttng-tools |
| lttng-ust-devel |
| lttng-ust |
| babeltrace |
| |
| Build with -DUSE_LTTNG=ON to enable it. All of the tracepoints |
| have an '#ifdev USE_LTTNG" around them so the default of 'OFF' |
| creates no build or runtime dependencies on lttng. |
| |
| The build creates and installs an additional shared object. The |
| libganesha_trace.so object which contains all the defined tracepoint |
| code and the code in the application that uses it are constructed such |
| that the tracepoint code does not require the shared object to be |
| loaded. This means that the application can be built with LTTng code |
| but it can be run on a system that has no LTTng support. Tracepoints are |
| enabled once the module is loaded. See below for details. |
| |
| I use the following script to run: |
| |
| #!/bin/sh |
| |
| DIR=/usr/local/lib64/ganesha |
| |
| lttng create |
| lttng enable-event -u -a |
| lttng start |
| |
| LD_PRELOAD=$DIR/libganesha_trace.so /usr/local/bin/ganesha.nfsd ${*} |
| |
| In this case, I am preloading libganesha_trace.so which turns on tracing before |
| before the server starts. If you do not preload, ganesha runs as if tracing is |
| not there and the only overhead is the predicted missed branch to the tracer. |
| LTTng supports the loading of the tracing module into a running application by |
| loading (dlopen) the modul This would be useful for production environments |
| but that feature is on the TODO list. |
| |
| There are never enough tracepoints. Like log messages, they get added from time |
| to time as needed. There are two paths for adding tracepoints. |
| |
| * Sets of tracepoints are categorized in 'components'. These should conform |
| to the same general categories as logging components. All the tracepoints |
| in a component should be functionally related in some way. |
| |
| * Finer grained tracing adds new tracepoints to an existing category. |
| |
| Creating a New Component |
| ------------------------ |
| Two files must be created and a number of other files must be edited to create |
| a new tracepoint component. |
| |
| The tracepoint itself is created as a new include (.h) file in |
| src/include/ganesha_lttng. This is the bulk of the work. Note that the |
| file has a specific form. It is best to copy and edit an existing file so as |
| to get all the necessary definitions in the right places. There can be any |
| number of tracepoint definitions but each one *must* have a TRACEPOINT_EVENT |
| and TRACEPOINT_LOGLEVEL defined (after it). |
| |
| The tracepoint include file has a companion file in src/tracing that includes |
| the header after defining TRACEPOINT_CREATE_PROBES. Add this file to the |
| sources list in tracing/CMakeLists.txt so that it gets built into the tracing |
| module. |
| |
| Every source file that will use these tracepoints must also include the specific |
| include file(s) for the tracepoint component(s). Note that components are |
| defined separately. Both the #include and the tracepoints themselves should be |
| wrapped with an "#ifdef USE_LTTNG". |
| |
| The CMakeLists.txt file of a source sub-directory should be edited to add a |
| conditional 'include_directories' command so that LTTNG_INCLUDE_DIR will added |
| to the includes path. |
| |
| The last step in adding a component is to add an '#include' of the new header |
| in MainNFSD/nfs_main.c. This bit is LTTng magic to set up the dynamic linkage. |
| Every component header file is listed here ONLY ONCE. This includes any |
| tracepoints that may be added to nfs_main.c. |
| |
| Adding New Tracepoints |
| ---------------------- |
| Adding a new tracepoint to an existing component is really simple. First, |
| add the TRACEPOINT_EVENT and its TRACEPOINT_LOGLEVEL to the component's |
| header file. This is all that is necessary to create the tracepoint. The |
| next step is to add the new tracepoint into the code. There are a few extra |
| bits to remember in adding the tracepoint. |
| |
| * If the tracepoint is added to a file in a subdirectory that had no |
| tracepoints before, add the 'include_directories' directive to the |
| CMakeLists.txt file. |
| |
| * If the tracepoint category is new to the source file, add a '#include' |
| for it. |
| |
| * Wrap the #include and all tracepoints with USE_LTTNG. |
| |
| Notes on Using Tracepoints |
| ========================== |
| All this trace point organization is for the "enable-event" command above. |
| The "-a" turns them all on. If all you wanted was logging, you would change |
| the "-a" to "ganesha_logger:log". See the LTTng docs for the details. |
| |
| Tracepoint logs are placed in your $HOME/lttng-traces directory. Note that |
| if you are running as 'root' which is necessary for running nfs-ganesha, this |
| directory is actually /root/lttng-traces, not your regular $HOME. The lttng |
| commands will report the subdirectory where the traces are placed. You can |
| also specify a directory name if you like. If the lttng command reports |
| 'auto-20140804-102010' as the trace directory, you can find those files in |
| $HOME/lttng-traces/auto-20140804-102010. |
| |
| The 'babeltrace' tool is used to process and display traces. There are others |
| but this is the simplest and most general tool. To do a simple dump of |
| the trace from above, use: |
| |
| $ babeltrace $HOME/lttng-traces/auto-20140804-102010 |
| |
| This will dump a trace in text form. See the man page for all the options. |
| There are a number of other tools that can also munch traces. Traces |
| are in a common format that many tools can read and process/display them. |