Copyright 2008, Google Inc.

All rights reserved.

Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:

* Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.

* Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.

* Neither the name of Google Inc. nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

XRay: A Hierarchical Profiler for Native Client

Objective: To provide Native Client applications the ability to quickly generate hierarchical profiles, with cycle timing information and optional user supplied annotations. XRay requires minimal code changes and once added to a project, should easily facilitate a rapid edit-compile-run-profile cycle. At runtime, generating capture reports using XRay should require less than one second of cpu time -- even for large multi-frame, multi-megabyte captures.

Note: The .xhtml version of this document is derived from the OpenOffice .odt file and provided as a convenience. Edits should be performed on the .odt file and re-exported to the .xhtml version.

Introduction: Understanding the performance of large bodies of code is a difficult task without the use of a profiler. XRay is a profiling library designed to be user customizable via extending either the source code directly, or running user scripts on the output report. It relies on the gcc compiler's option -finstrument-functions, which implicitly adds calls to an entry and exit function for every compilied function.

Setup: A small amount of work is required to integrate XRay into an application. It is recommended to setup a seperate build target for profiling. (See the xray build target in native_client/common/Makefile.mk) This target should enable appropriate compiler optimizations, profiling options, preprocessor definitions, include the XRay runtime code, and linker mapfile generation.

Preprocessor defines: Two preprocessor defines are needed for XRay. One is to enable the runtime profiler itself, which will consume extra memory and impact performance slightly. The other option controls whether or not user supplied annotations should be compiled into code.

Compiler optimizations: A profiled build should enable optimizations to make it perform as close to release code as possible. For gcc and g++, a reasonable starting point is -O2 Floating point heavy applications may want to also enable use of SSE instructions by specifying -mfpmath=sse -msse

Profiling options: An XRay profiled build should enable the gcc code generation option -finstrument-functions and -falign-functions=32. These options add the required implicit calls to __cyg_profile_func_enter and __cyg_profile_func_exit (provided for you by the XRay library) XRay requires all functions to be aligned to at least 16 bytes (32 bytes are recommended) in order to effectively pack the function address into the tracebuffer during capture. For Native Client, this is a non-issue, since functions are aligned by at least this much anyway.

The XRay Runtime Library: Include the files xray.c stringpool.c hashtable.c & symtable.c, in your build. These files are small and, unless space is an extreme requirement, can be included in all builds.

Enable Linker Options and Post Build Steps: XRay will require a mapfile to match function addresses to function names. The linker option -Map myapp.map will output a name mangled mapfile. Most people find mangled names difficult to read, so a post-build step to de-mangle names via the GNU's c++filt utility is recommended. It is also recommended to output static names to the mapfile, or else the trace output will be unable to list names for static functions. Do this by #defining STATIC to nothing for profile builds, but to be static for other builds, and replacing all use of static in your code with STATIC.

#ifdef (XRAY)

#define STATIC

#else

#define STATIC static

#endif

Runtime: A small amount of runtime code will need to be added to the user application. a) Initialization of XRay, which will load the mapfile, initialize the symbol table, and initialize the tracebuffer. b) The area of code to profile will need to make calls to the XRay library telling it to start and stop profile captures into the tracebuffer. A profile can consist of one or more 'frames.' c) When capturing has completed, a text based report file is written out. This report is generated with an explicit call to XRaySaveReport(), so it can be driven from an external event (keypress, unusually long frame, after a certain amount of elapsed time, etc.) And finally d) shutdown XRay to release resources.

Important: When used with Native Client, make sure to launch your profiled application using sel_ldr's -d command line argument, which will enable debug mode and allow file I/O. Otherwise, XRay won't be able to read the symbol table and/or save the trace report.

Overhead: An empty function with XRay profiling will require about 100 cycles to add profiling information to the trace buffer. With optimizations enabled, a profile build will have average performance somewhere between a debug and a release build. Optional annotations will add 1000 cycles per annotation (roughly eqv. to a sprintf and a strcpy for each annotation.) Normally annotations are disabled and compile to a no-operation. The XRAY_ANNOTATE preprocessor define will allow include annotations.

Inline Functions: XRay profiles inline functions. This also means that XRay overhead is added to them. This can be disastrous for tiny inlined functions such as get/set class member functions, which would normally reduce down to one or two cycles of move instructions. With the XRay trace buffer overhead, these functions will balloon to 100 cycles. An explicit XRAY_NO_INSTRUMENT modifier can be added to a function declaration to exclude it. The time it consumes will still be accounted for in the profiled parent functions. Future versions of nacl-gcc and nacl-g++ may include -finstrument-functions-except-inlines, which will avoid the requirement to explicitly mark inline functions XRAY_NO_INSTRUMENT.

Limitations: XRay has a few practical limitations. At the moment, a trace entry requires 16 bytes. 8 bits are for representing call tree depth, so XRay won't report functions deeper than 255 calls. 24 bits are for representing the function address (which has to be aligned to at least 16 bytes), so XRay can only profile up to 28 bits of code address space – which should be enough for most Native Client applications. 64 bits of the trace buffer entry are used for recording RDTSC 'ticks' which is the RDTSC delta. Even with 64 bits this counter can periodically overflow; XRay makes no attempt to detect and account for this situation. Annotations are limited to 1024 characters per trace entry. XRay cannot (currently) profile code with exceptions or setjmp/longjmp usage. It requires x86 cpus to support the RDTSC instruction. While XRay can correctly operate in a multi-threaded application, it can only profile a single thread at a time. There are also some limitations with RDTSC – it reports cycles globally, so other threads can interrupt a profile and their time will show up in functions of the thread being profiled. On multi-processor systems RDTSC might not be exactly synchronized across cores. In practice, limitations with RDTSC can be worked around with processor affinity and repeated captures. Slow code usually overshadows these limitations by orders of magnitude.

Deep recursion distortion: currently XRay makes no attempt to make any guesstimates as to the amount of overhead it imposes on profiled code. In practice it might be around 100 cycles per function. This isn't bad when profiling near the root of the call tree, but can start to distort timings in deeper call trees. This is especially evident in deep recursion of cheap functions, where the profile overhead is compounded, adding significant distortion to the execution time. If the code is tail recursive, consider re-writing it to eliminate the recursion – your trace outputs will be more concise and easier to analyze. Other types of recursion may be more difficult to eliminate. Limiting the stack depth can also improve the timing of deep recursion, but does not eliminate the overhead altogether – XRay still has to track and check the depth value. Future versions of XRay may attempt to recursively subtract its own overhead, but this is still only a guess, as modern out-of-order execution processors make predicting the actual overhead almost impossible – it will be unique to the surrounding code and how well it slots in with the instructions added by XRay. XRay could self-profile a best minimal case at initialization and subtract that across the board during report generation, but if we are too aggressive, some functions will have negative cycle counts.

Pruning The Trace: There are currently three ways to prune a large trace capture. First, most non-recursive code won't have very deep call trees – maybe 16 deep at the most. But when recursion does enter the picture, the call tree depth can surge up to 255 entries deep. Most of the time this deep recursion just pollutes the trace with an unnecessary amount of information. So one option in XRay is to stop capture when a call tree exceeds a user specified depth. In most cases, it is sufficient to do a capture run with the limit set to 255, identify where recursion occurs, and then set the depth limiter to a reasonable amount depending on when the recursion tends to occur. A reasonable guess for most code would be a call depth between 16 and 32 entries. Again, the time spent in deep call tress is still reported in the profiled parent functions. This is a pruning option that occurs at trace capture time.

The second option for pruning the trace output occurs when traces are dumped to a file. XRay will automatically normalize the amount of time spent in a function as a percent of the total frame. Most of the time, functions that only require a tiny percent of the total frame are... uninteresting – there are bigger fish to be frying. So it is possible to instruct XRay to explicitly omit all functions (and their children) from the trace output if their percent of the captured frame is less than n percent, where n is specified by the user. Again, the time spent in these omitted functions is still reflected (and reported) in the parent functions. If a function is taking a suspicious amount of time, this filter can be temporarily lifted to reveal who is taking this time. This is a pruning option that occurs at report time.

The third option has already been alluded to earlier – any function can be removed from the trace by adding XRAY_NO_INSTRUMENT to its declaration. This will omit profiling for that function, but still include profiling for any children (assuming they aren't filtered themselves) It is recommended that short inline 'leaf' functions be marked as XRAY_NO_INSTRUMENT. (Usually you don't want to profile an inline member get function that boils down to one or two cycles of move instructions. A couple of exceptions to this is when you need the function to appear in the trace report but don't care about the cycle penalty imposed by profiling, or the parent function is showing unexpectedly slow cycle times and you suspect these small inline functions might actually have poor cache performance.) This is a pruning option that occurs at compile time.

By mixing the above three pruning methods -- compile time, capture time, and report time -- large bodies of code can be reduced down to manageable trace outputs.

Adding User Annotations: XRay features an optional, but useful, API for explicitly annotating the call tree. XRayAnnotate(const char *str, ...) is identical to printf in the arguments it takes, but instead of displaying the string on stdout, the string is copied into the trace buffer as part of the capture. This annotation is attached to the function in which the call to XRayAnnotate appeared in. Later, if a trace report is generated, the captured string is output adjacent to the function in the report.

For example:

int foo(int x) {

int r;

XRayAnnotate(“Value of x: %d”, x);

r = x * x * x;

XRayAnnotate(“ r: %d”, r);

return r;

}

void bar() {

foo(3);

foo(2);

foo(1);

}

In the trace report generated by XRay, the annotation will reveal the value of x and r for every call to function foo().

(continued)

Address Ticks Percent Function <optional annotation>

----------------------------------------------------------------

...

0x00010700 3300 0.9 bar()

0x00010310 1002 0.3 foo() Value of x: 3 r: 27

0x00010310 1004 0.3 foo() Value of x: 2 r: 8

0x00010310 1001 0.3 foo() Value of x: 1 r: 1

...

Always On Profiling: XRay has reasonably minimal performance overhead. In most cases, it can be left in an “always capture” mode. The trace buffer is a large ring buffer, and constantly overwrites itself. It is important to select the appropriately sized trace buffer for your application -- too small a trace buffer will result in incomplete profiles. Too large a trace buffer, and you will find yourself having to surf too much profile data. With a correctly selected trace buffer size at initialization, many frames of a manageable amount of profile information will always be available in the ring buffer.

An important feature of XRay is taking advantage of always on profiling. Some profilers are dormant until the trigger occurs, at which point they perform a capture, but at much reduced execution speed of the application. These profilers make it very hard to capture rarely occurring hiccups in performance – because you have to guess ahead of time when to trigger the capture! XRay can be configured to always capture and if a hiccup occurs, a temporal trigger can dump the profile results after-the-fact, making it very easy to capture rare once-in-a-blue-moon performance stutters & heartbeats.

Analyzing Trace Ouput: First, it is recommended to capture a few trace reports using aggressive filtering. Set the report filtering at two percent or higher. Set the call tree depth to 4 calls. Apply XRAY_NO_INSTRUMENT to the top few inline functions that are otherwise polluting the trace report. This should yield a trace report that is only a few pages, even for huge bodies of code. Spend some time looking over this high level “birds-eye” view. Some surprises might be quickly evident.

Next, burrow down further into the top level parent functions that seem to be taking longer than you expect. Take a moment to look at the text format of the trace capture output. Consider writing grep, awk, perl or python scripts to help you analyze larger trace captures.

Observations: A large software project usually contains a mix of engineers with different skill sets and performance backgrounds. Usually, surprises in performance come from unexpected areas. The usual suspects: garbage collection spikes, poorly written code in scripting languages, unexpected use of algorithms (especially overloaded [] in C++, where one would assume O(1) lookup, but the underlying implementation uses a linear... or worse... lookup on the index), well intentioned data structures that have poor cache performance, code that uses strings instead of enumerations, data structures that use too many cache in-coherent pointers instead of cache coherent arrays, code that unexpectedly breaks the “ways” of the cache, pathological branch prediction, numerical code that looks trivial but ends up exercising software floating point emulation or rounding modes, use of atof() when parsing a large number of floats, etc...

Portability: All parts of XRay, at the moment, are written in Native Client. It requires use of the RDTSC instruction, and file I/O to dump the trace capture, which can be enabled by using the “-d” debug flag on sel_ldr when launching NaCl applications. It can be used equally well on Windows, Mac and Linux.

Conclusion: XRay in its current state is about 20 pages of C code. It is intended to be a minimal -- but highly useful and easy to understand -- hierarchical profiler. The annotation ability also provides some limited debugging capability.

Future work: support for more reporting options, binary dump of trace capture for deeper off-line report / analysis, real-time graphing via shared memory ring buffer and concurrent profiling graph utility. Runtime self-modifying code to disable/enable __cyg_profile_func_enter() / __cyg_profile_func_exit() function call overhead when not profiling, ability to profile multiple threads concurrently.

Example Trace Report on Quake, using heavily pruned output. Functions requiring less than 0.5% of the frame are pruned. The frame table of contents and the first two frames of rendering are shown here. Following this example trace report is a short walk through demonstrating how to add XRay to a code base. The report is in two parts, the first part is an overview of all the frames. You can then use the label field to find the appropriate frame further down in the second half of the profile report.

Frame# Total Ticks Capture size Annotations Label

----------------------------------------------------------------------

0 38578023 18471 1 @@@frame0

1 38609433 18403 1 @@@frame1

2 38278584 18946 1 @@@frame2

3 37807992 18817 1 @@@frame3

4 38440449 19085 1 @@@frame4

5 39003597 19055 1 @@@frame5

6 41945193 19437 1 @@@frame6

7 39021318 19132 1 @@@frame7

8 38953836 19288 1 @@@frame8

9 40055157 19583 1 @@@frame9

10 41722308 21473 1 @@@frame10

11 40208364 21394 1 @@@frame11

12 40096296 21462 1 @@@frame12

13 40166820 21880 1 @@@frame13

14 39802320 22132 1 @@@frame14

15 44599851 23978 1 @@@frame15

16 42858477 23524 1 @@@frame16

17 42737040 22140 1 @@@frame17

18 41868396 21986 1 @@@frame18

19 42065118 22375 1 @@@frame19

20 41432913 21610 1 @@@frame20

21 41432085 21366 1 @@@frame21

22 41422329 21438 1 @@@frame22

23 42658218 22965 1 @@@frame23

24 42124986 23416 1 @@@frame24

25 40656915 23409 1 @@@frame25

26 40898682 24175 1 @@@frame26

27 40306761 22569 1 @@@frame27

28 39704139 22061 1 @@@frame28

29 40285233 23210 1 @@@frame29

30 40479561 22830 1 @@@frame30

31 39923307 22013 1 @@@frame31

32 40395078 22168 1 @@@frame32

33 40016079 22648 1 @@@frame33

34 40001832 22770 1 @@@frame34

35 39875715 22758 1 @@@frame35

36 40033116 22952 1 @@@frame36

37 40063905 23049 1 @@@frame37

38 40234455 23068 1 @@@frame38

39 40101408 23211 1 @@@frame39

40 40385430 24121 1 @@@frame40

41 49233465 27678 1 @@@frame41

42 48870225 27874 1 @@@frame42

43 49595139 27578 1 @@@frame43

44 49410378 28399 1 @@@frame44

45 48723327 28485 1 @@@frame45

46 48720195 28765 1 @@@frame46

47 48202893 28018 1 @@@frame47

48 48704220 27988 1 @@@frame48

49 48594735 27848 1 @@@frame49

50 48975957 27802 1 @@@frame50

51 49999113 28864 1 @@@frame51

52 50078736 30096 1 @@@frame52

53 52864776 36456 1 @@@frame53

54 54470799 38659 1 @@@frame54

55 53051958 39805 1 @@@frame55

56 53089164 41015 1 @@@frame56

57 53421876 41180 1 @@@frame57

58 53214642 41225 1 @@@frame58

----------------------------------------------------------------------

XRay: 59 frame(s) 1468103 total capture(s)

======================================================================

label @@@frame0

Address Ticks Percent Function <optional annotation>

----------------------------------------------------------------------

0x0002FAA0 38565216 100.0 Host_Frame

0x0002F500 38564388 100.0 _Host_Frame

0x000759A0 38335896 99.4 SCR_UpdateScreen

0x0008CC80 23560650 61.1 V_RenderView

0x00067920 23533893 61.0 R_RenderView

0x00067100 23531589 61.0 R_RenderView_

0x000664C0 18393984 47.7 R_EdgeDrawing

0x000604C0 258417 0.7 R_RenderWorld

0x0005FEA0 257841 0.7 R_RecursiveWorldNode

0x0005FEA0 255933 0.7 R_RecursiveWorldNode

0x0005FEA0 254799 0.7 R_RecursiveWorldNode

0x0005FEA0 254088 0.7 R_RecursiveWorldNode

0x0005FEA0 253143 0.7 R_RecursiveWorldNode

0x0005FEA0 250380 0.6 R_RecursiveWorldNode

0x0005FEA0 248418 0.6 R_RecursiveWorldNode

0x000643C0 18058671 46.8 R_ScanEdges

0x00025800 9790209 25.4 D_DrawSurfaces

0x000290C0 1767573 4.6 D_DrawSpans8

0x00028EE0 343350 0.9 D_DrawZSpans

0x000290C0 640764 1.7 D_DrawSpans8

0x000290C0 3491316 9.1 D_DrawSpans8

0x00028EE0 616662 1.6 D_DrawZSpans

0x000290C0 329067 0.9 D_DrawSpans8

0x000290C0 213138 0.6 D_DrawSpans8

0x000290C0 331992 0.9 D_DrawSpans8

0x000290C0 991287 2.6 D_DrawSpans8

0x00025800 5613084 14.5 D_DrawSurfaces

0x000290C0 1132128 2.9 D_DrawSpans8

0x00028EE0 221553 0.6 D_DrawZSpans

0x000290C0 497043 1.3 D_DrawSpans8

0x000290C0 590841 1.5 D_DrawSpans8

0x000290C0 1223667 3.2 D_DrawSpans8

0x00028EE0 227484 0.6 D_DrawZSpans

0x000290C0 380529 1.0 D_DrawSpans8

0x000290C0 886743 2.3 D_DrawSpans8

0x00066BE0 4103073 10.6 R_DrawEntitiesOnList

0x0005FB20 1559475 4.0 R_AliasDrawModel

0x0005F800 1553427 4.0 R_AliasPreparePoints

0x0005FB20 2117061 5.5 R_AliasDrawModel

0x0005F800 2113443 5.5 R_AliasPreparePoints

0x0005FB20 256095 0.7 R_AliasDrawModel

0x0005EDC0 251388 0.7 R_AliasPrepareUnclippedPoints

0x00028E20 243297 0.6 D_PolysetDraw

0x00028B80 243072 0.6 D_DrawNonSubdiv

0x00066840 919269 2.4 R_DrawViewModel

0x0005FB20 907380 2.4 R_AliasDrawModel

0x0005F800 902331 2.3 R_AliasPreparePoints

0x00089CA0 14754924 38.2 VID_Update

0x00089A80 14752476 38.2 (null) // static funciton

======================================================================

label @@@frame1

Address Ticks Percent Function <optional annotation>

----------------------------------------------------------------------

0x0002FAA0 38596356 100.0 Host_Frame

0x0002F500 38595402 100.0 _Host_Frame

0x000759A0 38373732 99.4 SCR_UpdateScreen

0x0008CC80 23582070 61.1 V_RenderView

0x00067920 23553576 61.0 R_RenderView

0x00067100 23551164 61.0 R_RenderView_

0x000664C0 18382464 47.6 R_EdgeDrawing

0x000604C0 264087 0.7 R_RenderWorld

0x0005FEA0 263484 0.7 R_RecursiveWorldNode

0x0005FEA0 261027 0.7 R_RecursiveWorldNode

0x0005FEA0 259641 0.7 R_RecursiveWorldNode

0x0005FEA0 258867 0.7 R_RecursiveWorldNode

0x0005FEA0 258111 0.7 R_RecursiveWorldNode

0x0005FEA0 255168 0.7 R_RecursiveWorldNode

0x0005FEA0 253575 0.7 R_RecursiveWorldNode

0x000643C0 18039600 46.7 R_ScanEdges

0x00025800 9837378 25.5 D_DrawSurfaces

0x000290C0 1650078 4.3 D_DrawSpans8

0x00028EE0 325638 0.8 D_DrawZSpans

0x000290C0 641736 1.7 D_DrawSpans8

0x000290C0 3468933 9.0 D_DrawSpans8

0x00028EE0 613539 1.6 D_DrawZSpans

0x000290C0 349830 0.9 D_DrawSpans8

0x000290C0 265491 0.7 D_DrawSpans8

0x000290C0 333585 0.9 D_DrawSpans8

0x000290C0 1089666 2.8 D_DrawSpans8

0x00028EE0 212247 0.5 D_DrawZSpans

0x00025800 5571315 14.4 D_DrawSurfaces

0x000290C0 1059129 2.7 D_DrawSpans8

0x00028EE0 215793 0.6 D_DrawZSpans

0x000290C0 510471 1.3 D_DrawSpans8

0x000290C0 571500 1.5 D_DrawSpans8

0x000290C0 1177155 3.0 D_DrawSpans8

0x00028EE0 213192 0.6 D_DrawZSpans

0x000290C0 433746 1.1 D_DrawSpans8

0x000290C0 920241 2.4 D_DrawSpans8

0x00066BE0 4149441 10.7 R_DrawEntitiesOnList

0x0005FB20 1561581 4.0 R_AliasDrawModel

0x0005F800 1555011 4.0 R_AliasPreparePoints

0x0005FB20 2168361 5.6 R_AliasDrawModel

0x0005F800 2164608 5.6 R_AliasPreparePoints

0x0005FB20 252918 0.7 R_AliasDrawModel

0x0005EDC0 248139 0.6 R_AliasPrepareUnclippedPoints

0x00028E20 240201 0.6 D_PolysetDraw

0x00028B80 239967 0.6 D_DrawNonSubdiv

0x00066840 897588 2.3 R_DrawViewModel

0x0005FB20 886122 2.3 R_AliasDrawModel

0x0005F800 881136 2.3 R_AliasPreparePoints

0x00089CA0 14769072 38.3 VID_Update

0x00089A80 14767911 38.2 (null) // static function

======================================================================

Example usage: This section will explain the required steps to add XRay to an existing code base. All source files that need to make explicit calls to XRay functions (including XRayAnnotate) will need to #include “xray.h”

Step a) First, in your startup code, make a call to XRayInit() Using Quake as an example, a good place the call XRayInit() is in sys_nacl.c, in function main(), right before it enters the application's main loop.

// Step a

// Initialize XRay and load symbols from mapfile

XRayInit(4, 4000000, 100, “naclquake.map”)

The first argument is the maximum stack depth XRay should record. XRay can record stack depths of up to 255 entries. By selecting smaller values, it is possible to prune call trees, especially if they contain deep recursion.

The second argument is the number of entries you want to use for the trace buffer. Each entry requires 16 bytes. Small projects can probably get away with 100,000 entries. Large projects can use much bigger buffers, as much as Native Client will allow. Quake, for example, can use a buffer size of 4,000,000 entries. Although most modern machines come equipped with plenty of physical memory, take care not to exceed it or profile results will be meaningless due to excessive paging.

The third argument is the frame count. This is an upper ceiling on how many frames you want to capture. In Quake, a frame is a single video frame of animation. Quake runs about 60 frames per second. A frame doesn't have to be tied to video, it could be more abstract, like a database transaction. In XRay, XRayStartFrame() / XRayEndFrame() pair define a frame.

The last argument is the name of the mapfile generated by nacl-ld obtained by using the linker option -Map myapp.map Or, if your project uses nacl-gcc / nacl-g++ to link, use -Wl,-Map,myapp.map If you find c++ name mangling to be inconvenient, use the GNU utility c++filt to demangle all the symbols in the mapfile. See the xray build target options in the makefile native_client/common/Makefile.mk

Step b) Wrap the high level function you want to profile. In Quake this is the function Host_Frame(time) in sys_nacl.c

// Step b

XRayStartFrame();

Host_Frame(time); // profile Quake main loop

XRayEndFrame();

// Step c

// elsewhere in code, '-' keypress sets g_profile to true

if (g_profile) {

XRaySaveReport(“quake.profile”, 0.5f);

g_profile = false;

}

Step c) Figure out how you want to be able to trigger dumping a profile report. One method is to bind an unused key. When this key is pressed, set a global variable indicating that the user has requested a profile dump. This needs to be deferred, because dumping the profile cannot occur from inside an XRayStartFrame() / XRayEndFrame(). For example, profiles could be triggered when the user presses the minus (-) key on the numeric keypad if that key is normally unused by the application. You will need to find where the application responds to keyboard events and add code indicating that the user has requested a profile dump. If this event loop is called while the profiler is active (ie the code is between XRayStartFrame() and XRayEndFrame()) then you will need to set a global variable to indicate that after the frame completes, a profile report should be saved. Do not try to save profile reports between XRayStartFrame() and XRayEndFrame().

In code outside the XrayStartFrame() / XrayEndFrame() pair, if the user has requested a profile report, call XRaySaveReport(const char *filename, float percent_cutoff) This will open the named file and write out a text based profile report, with some pruning depending on how aggressive the percent_cutoff value is. As a starting point, it is reasonable to prune all functions that consume less than 1% of a frame – these small functions (most likely leaves on the calltree) usually aren't where big optimization gains will be made when profiling large, complex bodies of code.

// Step d

XRayShutdown();

Step d) Shutdown XRay in your application shutdown to release resources.

You will probably need to tweak a few of the options that prune the profile to get useful output. Start by aggressively pruning to get a good bird's eye view of where time is being spent in the top level functions. If an area of the application takes longer than you expect, especially as a percent of the other operations, relax the pruning constraints and take more profiles. It shouldn't take long to drill down and figure out why code takes an unexpected amount of time.

Once you are familiar with this profiler, build-run-profile-edit sessions should go very quickly. You can also use grep, awk, perl, or python on the generated reports, or modify XRay itself to extend its capabilities.

Appendix A - XRay Reference

void XRayInit(int stack_size,

int buffer_size,

int frame_count,

const char *filename)

inputs:

stack_size maximum call tree depth. The profiler will not

add functions to the trace buffer whose depth in

the call stack exceed this value.

buffer_size size of the trace buffer, in units of trace entries.

A trace entry is 16 bytes, and holds either a single

function call, or a string fragment from an

annotation. Small programs can use 100,000 entries.

Large programs can use 4,000,000 or more.

frame_count number of frames to capture before wrapping around.

This is the maximum number of frames to record at

any given time. The actual number recorded can be

less (the user only took 5 frames of capture, or the

buffer_size isn't large enough to accommodate all

frames.) If you only need a single profile, use

a value of 1.

filename filename of nacl-gcc generated mapfile. Without

this file, the trace report won't have function

names.

void XRayShutdown()

action: shuts down and releases memory used by XRay.

void XRayStartFrame()

action: starts recording of next frame into trace buffer.

Only the thread that calls XRayStartFrame() will be

recorded. This call is not re-entrant and cannot be

nested.

void XRayEndFrame()

action: stops recording into trace buffer. Only the thread

that originally called XRayStartFrame() should call

XRayEndFrame(). This function must be called to

close the frame before opening the next frame or

saving a profile report.

void XRaySetAnnotationFilter(uint32_t filter)

input:

filter Sets the annotation filter bit mask. By default,

all annotation channels are enabled. This is a

mechanism that can be used to filter annotations in

the trace output. The filter can be different

developers, different areas of the code, or any

arbitrary filter – it is entirely up to the user.

void XRayAnnotate(const char *str, ...)

input:

str... A printf style string with optional arguments. The

output string is appended to the trace buffer. In

the trace report, the string appears on the same

line the function who called it. This is an easy

way to add simple debug information to the trace

report.

Keep in mind that each call consumes performance,

about 1000 RDTSC ticks. If XRAY_ANNOTATE isn't

defined, XRayAnnotate() calls are reduced to null

code.

void XRayAnnotateFiltered(int mask, const char *str, ...)

input:

mask A mask value that is tested against the current

filter set by XraySetAnnotationFilter. If the mask

test fails, the annotation string isn't appended to

the trace and the execution time of this function is

significantly less.

str... printf style argument(s), same as XRayAnnotate()

void XRaySaveReport(const char *filename, float cutoff)

input:

filename output filename to save text profile report.

cutoff functions below this cutoff, expressed as a percent,

will not be output to the trace report. This is a

handy way to filter out small, insignificant

functions. Usually filtering out functions that

take less than 1% of a frame will produce a nicely

pruned report for fast bird's eye view analysis.

When you want to generate the entire call tree

report, set this value to 0.0

action: Saves the trace report in text form to the specified

file. This function must be called outside

XRayStartFrame() / XRayEndFrame()