perf/README - chromiumos/third_party/cairo - Git at Google

 This is cairo's performance test suite.

 One of the simplest ways to run the performance suite is:

     make perf

 which will give a report of the speed of each individual test. See
 more details on other options for running the suite below.

 Running the cairo performance suite
 -----------------------------------
 The performance suite is composed of two types of tests, micro- and
 macro-benchmarks. The micro-benchmarks are a series of hand-written,
 short, synthetic tests that measure the speed of doing a simple
 operation such as painting a surface or showing glyphs. These aim to
 give very good feedback on whether a performance related patch is
 successful without causing any performance degradations elsewhere. The
 second type of benchmark consists of replaying a cairo-trace from a
 large application during typical usage. These aim to give an overall
 feel as to whether cairo is faster for everyday use.

 Running the micro-benchmarks
 ----------------------------
 The micro-benchmarks are compiled into a single executable called
 cairo-perf-micro, which is what "make perf" executes. Some
 examples of running it:

     # Report on all tests with default number of iterations:
     ./cairo-perf-micro

     # Report on 100 iterations of all gradient tests:
     ./cairo-perf-micro -i 100 gradient

     # Generate raw results for 10 iterations into cairo.perf
     ./cairo-perf-micro -r -i 10 > cairo.perf
     # Append 10 more iterations of the paint test
     ./cairo-perf-micro -r -i 10 paint >> cairo.perf

 Raw results aren't useful for reading directly, but are quite useful
 when using cairo-perf-diff to compare separate runs (see more
 below). The advantage of using the raw mode is that test runs can be
 generated incrementally and appended to existing reports.

 Running the macro-benchmarks
 ----------------------------
 The macro-benchmarks are run by a single program called
 cairo-perf-trace, which is also executed by "make perf".
 cairo-perf-trace loops over the series of traces stored beneath
 cairo-traces/. cairo-perf-trace produces the same output and takes the
 same arguments as cairo-perf-micro.  Some examples of running it:

     # Report on all tests with default number of iterations:
     ./cairo-perf-trace

     # Report on 100 iterations of all firefox tests:
     ./cairo-perf-trace -i 100 firefox

     # Generate raw results for 10 iterations into cairo.perf
     ./cairo-perf-trace -r -i 10 > cairo.perf
     # Append 10 more iterations of the poppler tests
     ./cairo-perf-trace -r -i 10 poppler >> cairo.perf

 Generating comparisons of separate runs
 ---------------------------------------
 It's often useful to generate a chart showing the comparison of two
 separate runs of the cairo performance suite, (for example, after
 applying a patch intended to improve cairo's performance). The
 cairo-perf-diff script can be used to compare two report files
 generated by cairo-perf.

 Again, by way of example:

     # Show performance changes from cairo-orig.perf to cairo-patched.perf
     ./cairo-perf-diff cairo-orig.perf cairo-patched.perf

 This will work whether the data files were generate in raw mode (with
 cairo-perf -r) or cooked, (cairo-perf without -r).

 Finally, in its most powerful mode, cairo-perf-diff accepts two git
 revisions and will do all the work of checking each revision out,
 building it, running cairo-perf for each revision, and finally
 generating the report. Obviously, this mode only works if you are
 using cairo within a git repository, (and not from a tar file). Using
 this mode is as simple as passing the git revisions to be compared to
 cairo-perf-diff:

     # Compare cairo 1.2.6 to cairo 1.4.0
     ./cairo-perf-diff 1.2.6 1.4.0

     # Measure the impact of the latest commit
     ./cairo-perf-diff HEAD~1 HEAD

 As a convenience, this common desire to measure a single commit is
 supported by passing a single revision to cairo-perf-diff, in which
 case it will compare it to the immediately preceeding commit. So for
 example:

     # Measure the impact of the latest commit
     ./cairo-perf-diff HEAD

     # Measure the impact of an arbitrary commit by SHA-1
     ./cairo-perf-diff aa883123d2af90

 Also, when passing git revisions to cairo-perf-diff like this, it will
 automatically cache results and re-use them rather than re-running
 cairo-perf over and over on the same versions. This means that if you
 ask for a report that you've generated in the past, cairo-perf-diff
 should return it immediately.

 Now, sometimes it is desirable to generate more iterations rather than
 re-using cached results. In this case, the -f flag can be used to
 force cairo-perf-diff to generate additional results in addition to
 what has been cached:

     # Measure the impact of latest commit (force more measurement)
     ./cairo-perf-diff -f

 And finally, the -f mode is most useful in conjunction with the --
 option to cairo-perf-diff which allows you to pass options to the
 underlying cairo-perf runs. This allows you to restrict the additonal
 test runs to a limited subset of the tests.

 For example, a frequently used trick is to first generate a chart with
 a very small number of iterations for all tests:

     ./cairo-perf-diff HEAD

 Then, if any of the results look suspicious, (say there's a slowdown
 reported in the text tests, but you think the text test shouldn't be
 affected), then you can force more iterations to be tested for only
 those tests:

     ./cairo-perf-diff -f HEAD -- text

 Generating comparisons of different backends
 --------------------------------------------
 An alternate question that is often asked is, "how does the speed of one
 backend compare to another?". cairo-perf-compare-backends can read files
 generated by cairo-perf and produces a comparison of the backends for every
 test.

 Again, by way of example:

     # Show relative performance of the backends
     ./cairo-perf-compare-backends cairo.perf

 This will work whether the data files were generate in raw mode (with
 cairo-perf -r) or cooked, (cairo-perf without -r).


 Creating a new performance test
 -------------------------------
 This is where we could use everybody's help. If you have encountered a
 sequence of cairo operations that are slower than you would like, then
 please provide a performance test. Writing a test is very simple, it
 requires you to write only a small C file with a couple of functions,
 one of which exercises the cairo calls of interest.

 Here is the basic structure of a performance test file:

     /* Copyright © 2006 Kind Cairo User
      *
      * ... Licensing information here ...
      * Please copy the MIT blurb as in other tests
      */

     #include "cairo-perf.h"

     static cairo_perf_ticks_t
     do_my_new_test (cairo_t *cr, int width, int height)
     {
 	cairo_perf_timer_start ();

 	/* Make the cairo calls to be measured */

 	cairo_perf_timer_stop ();

 	return cairo_perf_timer_elapsed ();
     }

     void
     my_new_test (cairo_perf_t *perf, cairo_t *cr, int width, int height)
     {
 	/* First do any setup for which the execution time should not
 	 * be measured. For example, this might include loading
 	 * images from disk, creating patterns, etc. */

 	/* Then launch the actual performance testing. */
 	cairo_perf_run (perf, "my_new_test", do_my_new_test);

 	/* Finally, perform any cleanup from the setup above. */
     }

 That's really all there is to writing a new test. The first function
 above is the one that does the real work and returns a timing
 number. The second function is the one that will be called by the
 performance test rig (see below for how to accomplish that), and
 allows for multiple performance cases to be written in one file,
 (simply call cairo_perf_run once for each case, passing the
 appropriate callback function to each).

 We go through this dance of indirectly calling your own function
 through cairo_perf_run so that cairo_perf_run can call your function
 many times and measure statistical properties over the many runs.

 Finally, to fully integrate your new test case you just need to add
 your new test to three different lists. (TODO: We should set this up
 better so that the lists are maintained automatically---computed from
 the list of files in cairo/perf, for example). Here's what needs to be
 added:

  1. Makefile.am: Add the new file name to the cairo_perf_SOURCES list

  2. cairo-perf.h: Add a new CAIRO_PERF_DECL line with the name of your
     function, (my_new_test in the example above)

  3. cairo-perf.c: Add a new row to the list at the end of the file. A
     typical entry would look like:

 	{ my_new_test, 16, 64 }

     The last two numbers are a minimum and a maximum image size at
     which your test should be exercised. If these values are the same,
     then only that size will be used. If they are different, then
     intermediate sizes will be used by doubling. So in the example
     above, three tests would be performed at sizes of 16x16, 32x32 and
     64x64.


 How to record new traces
 -----------------------
 Using cairo-trace you can record the exact sequence of graphic operations
 made by an application and replay them later. These traces can then be
 used by cairo-perf-trace to benchmark the various backends and patches.

 To record a trace:
 $ cairo-trace --no-mark-dirty --no-callers $APPLICATION [$ARGV]

 --no-mark-dirty is useful for applications that are paranoid about
 surfaces being modified by external plugins outside of their control, the
 prime example here is firefox.
 --no-callers disables the symbolic caller lookup and so speeds tracing
 (dramatically for large c++ programs) and similarly speeds up the replay
 as the files are much smaller.

 The output file will be called $APPLICATION.$PID.trace, the actual path
 written to will be displayed on the terminal.

 Alternatively you can use:
 $ cairo-trace --profile $APPLICATION [$ARGV]
 which automatically passes --no-mark-dirty and --no-callers and compresses
 the resultant trace using LZMA. To use the trace with cairo-perf-trace you
 will first need to decompress it.

 Then to use cairo-perf-trace:
 $ ./cairo-perf-trace $APPLICATION.$PID.trace

 Alternatively you can put the trace into perf/cairo-traces, or set
 CAIRO_TRACE_DIR to point to your trace directory, and the trace will be
 included in the performance tests.

 If you record an interesting trace, please consider sharing it by compressing
 it, LZMA preferred, and posting a link to cairo@cairographics.org, or by
 uploading it to git.cairographics.org/cairo-traces.


 How to run cairo-perf-diff on WINDOWS
 -------------------------------------
 This section explains the specifics of running cairo-perf-diff under
 win32 plateforms. It assumes that you have installed a UNIX-like shell
 environment such as MSYS (distributed as part of MinGW).

  1. From your Mingw32 window, be sure to have all of your MSVC environ-
     ment variables set up for proper compilation using 'make'

  2. Add the %GitBaseDir%/Git/bin path to your environment, replacing the
     %GitBaseDir% by whatever directory your Git version is installed to.

  3. Comment out the "UNSET CDPATH" line in the git-sh-setup script
     (located inside the ...Git/bin directory) by putting a "#" at the
     beginning of the line.

 you should be ready to go !

 From your mingw32 window, go to your cairo/perf directory and run the
 cairo-perf-diff script with the right arguments.

 Thanks for your contributions and have fun with cairo!
	This is cairo's performance test suite.

	One of the simplest ways to run the performance suite is:

	make perf

	which will give a report of the speed of each individual test. See
	more details on other options for running the suite below.

	Running the cairo performance suite
	-----------------------------------
	The performance suite is composed of two types of tests, micro- and
	macro-benchmarks. The micro-benchmarks are a series of hand-written,
	short, synthetic tests that measure the speed of doing a simple
	operation such as painting a surface or showing glyphs. These aim to
	give very good feedback on whether a performance related patch is
	successful without causing any performance degradations elsewhere. The
	second type of benchmark consists of replaying a cairo-trace from a
	large application during typical usage. These aim to give an overall
	feel as to whether cairo is faster for everyday use.

	Running the micro-benchmarks
	----------------------------
	The micro-benchmarks are compiled into a single executable called
	cairo-perf-micro, which is what "make perf" executes. Some
	examples of running it:

	# Report on all tests with default number of iterations:
	./cairo-perf-micro

	# Report on 100 iterations of all gradient tests:
	./cairo-perf-micro -i 100 gradient

	# Generate raw results for 10 iterations into cairo.perf
	./cairo-perf-micro -r -i 10 > cairo.perf
	# Append 10 more iterations of the paint test
	./cairo-perf-micro -r -i 10 paint >> cairo.perf

	Raw results aren't useful for reading directly, but are quite useful
	when using cairo-perf-diff to compare separate runs (see more
	below). The advantage of using the raw mode is that test runs can be
	generated incrementally and appended to existing reports.

	Running the macro-benchmarks
	----------------------------
	The macro-benchmarks are run by a single program called
	cairo-perf-trace, which is also executed by "make perf".
	cairo-perf-trace loops over the series of traces stored beneath
	cairo-traces/. cairo-perf-trace produces the same output and takes the
	same arguments as cairo-perf-micro. Some examples of running it:

	# Report on all tests with default number of iterations:
	./cairo-perf-trace

	# Report on 100 iterations of all firefox tests:
	./cairo-perf-trace -i 100 firefox

	# Generate raw results for 10 iterations into cairo.perf
	./cairo-perf-trace -r -i 10 > cairo.perf
	# Append 10 more iterations of the poppler tests
	./cairo-perf-trace -r -i 10 poppler >> cairo.perf

	Generating comparisons of separate runs
	---------------------------------------
	It's often useful to generate a chart showing the comparison of two
	separate runs of the cairo performance suite, (for example, after
	applying a patch intended to improve cairo's performance). The
	cairo-perf-diff script can be used to compare two report files
	generated by cairo-perf.

	Again, by way of example:

	# Show performance changes from cairo-orig.perf to cairo-patched.perf
	./cairo-perf-diff cairo-orig.perf cairo-patched.perf

	This will work whether the data files were generate in raw mode (with
	cairo-perf -r) or cooked, (cairo-perf without -r).

	Finally, in its most powerful mode, cairo-perf-diff accepts two git
	revisions and will do all the work of checking each revision out,
	building it, running cairo-perf for each revision, and finally
	generating the report. Obviously, this mode only works if you are
	using cairo within a git repository, (and not from a tar file). Using
	this mode is as simple as passing the git revisions to be compared to
	cairo-perf-diff:

	# Compare cairo 1.2.6 to cairo 1.4.0
	./cairo-perf-diff 1.2.6 1.4.0

	# Measure the impact of the latest commit
	./cairo-perf-diff HEAD~1 HEAD

	As a convenience, this common desire to measure a single commit is
	supported by passing a single revision to cairo-perf-diff, in which
	case it will compare it to the immediately preceeding commit. So for
	example:

	# Measure the impact of the latest commit
	./cairo-perf-diff HEAD

	# Measure the impact of an arbitrary commit by SHA-1
	./cairo-perf-diff aa883123d2af90

	Also, when passing git revisions to cairo-perf-diff like this, it will
	automatically cache results and re-use them rather than re-running
	cairo-perf over and over on the same versions. This means that if you
	ask for a report that you've generated in the past, cairo-perf-diff
	should return it immediately.

	Now, sometimes it is desirable to generate more iterations rather than
	re-using cached results. In this case, the -f flag can be used to
	force cairo-perf-diff to generate additional results in addition to
	what has been cached:

	# Measure the impact of latest commit (force more measurement)
	./cairo-perf-diff -f

	And finally, the -f mode is most useful in conjunction with the --
	option to cairo-perf-diff which allows you to pass options to the
	underlying cairo-perf runs. This allows you to restrict the additonal
	test runs to a limited subset of the tests.

	For example, a frequently used trick is to first generate a chart with
	a very small number of iterations for all tests:

	./cairo-perf-diff HEAD

	Then, if any of the results look suspicious, (say there's a slowdown
	reported in the text tests, but you think the text test shouldn't be
	affected), then you can force more iterations to be tested for only
	those tests:

	./cairo-perf-diff -f HEAD -- text

	Generating comparisons of different backends
	--------------------------------------------
	An alternate question that is often asked is, "how does the speed of one
	backend compare to another?". cairo-perf-compare-backends can read files
	generated by cairo-perf and produces a comparison of the backends for every
	test.

	Again, by way of example:

	# Show relative performance of the backends
	./cairo-perf-compare-backends cairo.perf

	This will work whether the data files were generate in raw mode (with
	cairo-perf -r) or cooked, (cairo-perf without -r).


	Creating a new performance test
	-------------------------------
	This is where we could use everybody's help. If you have encountered a
	sequence of cairo operations that are slower than you would like, then
	please provide a performance test. Writing a test is very simple, it
	requires you to write only a small C file with a couple of functions,
	one of which exercises the cairo calls of interest.

	Here is the basic structure of a performance test file:

	/* Copyright © 2006 Kind Cairo User
	*
	* ... Licensing information here ...
	* Please copy the MIT blurb as in other tests
	*/

	#include "cairo-perf.h"

	static cairo_perf_ticks_t
	do_my_new_test (cairo_t *cr, int width, int height)
	{
	cairo_perf_timer_start ();

	/* Make the cairo calls to be measured */

	cairo_perf_timer_stop ();

	return cairo_perf_timer_elapsed ();
	}

	void
	my_new_test (cairo_perf_t perf, cairo_t cr, int width, int height)
	{
	/* First do any setup for which the execution time should not
	* be measured. For example, this might include loading
	* images from disk, creating patterns, etc. */

	/* Then launch the actual performance testing. */
	cairo_perf_run (perf, "my_new_test", do_my_new_test);

	/* Finally, perform any cleanup from the setup above. */
	}

	That's really all there is to writing a new test. The first function
	above is the one that does the real work and returns a timing
	number. The second function is the one that will be called by the
	performance test rig (see below for how to accomplish that), and
	allows for multiple performance cases to be written in one file,
	(simply call cairo_perf_run once for each case, passing the
	appropriate callback function to each).

	We go through this dance of indirectly calling your own function
	through cairo_perf_run so that cairo_perf_run can call your function
	many times and measure statistical properties over the many runs.

	Finally, to fully integrate your new test case you just need to add
	your new test to three different lists. (TODO: We should set this up
	better so that the lists are maintained automatically---computed from
	the list of files in cairo/perf, for example). Here's what needs to be
	added:

	1. Makefile.am: Add the new file name to the cairo_perf_SOURCES list

	2. cairo-perf.h: Add a new CAIRO_PERF_DECL line with the name of your
	function, (my_new_test in the example above)

	3. cairo-perf.c: Add a new row to the list at the end of the file. A
	typical entry would look like:

	{ my_new_test, 16, 64 }

	The last two numbers are a minimum and a maximum image size at
	which your test should be exercised. If these values are the same,
	then only that size will be used. If they are different, then
	intermediate sizes will be used by doubling. So in the example
	above, three tests would be performed at sizes of 16x16, 32x32 and
	64x64.


	How to record new traces
	-----------------------
	Using cairo-trace you can record the exact sequence of graphic operations
	made by an application and replay them later. These traces can then be
	used by cairo-perf-trace to benchmark the various backends and patches.

	To record a trace:
	$ cairo-trace --no-mark-dirty --no-callers $APPLICATION [$ARGV]

	--no-mark-dirty is useful for applications that are paranoid about
	surfaces being modified by external plugins outside of their control, the
	prime example here is firefox.
	--no-callers disables the symbolic caller lookup and so speeds tracing
	(dramatically for large c++ programs) and similarly speeds up the replay
	as the files are much smaller.

	The output file will be called $APPLICATION.$PID.trace, the actual path
	written to will be displayed on the terminal.

	Alternatively you can use:
	$ cairo-trace --profile $APPLICATION [$ARGV]
	which automatically passes --no-mark-dirty and --no-callers and compresses
	the resultant trace using LZMA. To use the trace with cairo-perf-trace you
	will first need to decompress it.

	Then to use cairo-perf-trace:
	$ ./cairo-perf-trace $APPLICATION.$PID.trace

	Alternatively you can put the trace into perf/cairo-traces, or set
	CAIRO_TRACE_DIR to point to your trace directory, and the trace will be
	included in the performance tests.

	If you record an interesting trace, please consider sharing it by compressing
	it, LZMA preferred, and posting a link to cairo@cairographics.org, or by
	uploading it to git.cairographics.org/cairo-traces.


	How to run cairo-perf-diff on WINDOWS
	-------------------------------------
	This section explains the specifics of running cairo-perf-diff under
	win32 plateforms. It assumes that you have installed a UNIX-like shell
	environment such as MSYS (distributed as part of MinGW).

	1. From your Mingw32 window, be sure to have all of your MSVC environ-
	ment variables set up for proper compilation using 'make'

	2. Add the %GitBaseDir%/Git/bin path to your environment, replacing the
	%GitBaseDir% by whatever directory your Git version is installed to.

	3. Comment out the "UNSET CDPATH" line in the git-sh-setup script
	(located inside the ...Git/bin directory) by putting a "#" at the
	beginning of the line.

	you should be ready to go !

	From your mingw32 window, go to your cairo/perf directory and run the
	cairo-perf-diff script with the right arguments.

	Thanks for your contributions and have fun with cairo!