How to use TAU on Intrepid with PAPI
From PERI
From Sameer Shende (sameer@cs.uoregon.edu), 8/13/08
To use TAU on Intrepid with PAPI, I set:
% soft add +tau % setenv TAU_MAKEFILE /soft/apps/tau/tau_latest/bgp/lib/Makefile.tau-multiplecounters-mpi-papi-pdt
This specifies IBM compilers, with MPI level instrumentation and automatic instrumentation of source code (PDT) and support for collecting multiple PAPI counters in the same run. Other options (for say GNU with or without MPI, pthread without MPI, etc.) are:
Makefile.tau-bgptimers-multiplecounters-mpi-papi-gnu-python-pdt Makefile.tau-bgptimers-multiplecounters-mpi-papi-pdt Makefile.tau-multiplecounters-mpi-papi-gnu-pdt Makefile.tau-multiplecounters-mpi-papi-gnu-pthread-pdt Makefile.tau-multiplecounters-mpi-papi-pdt Makefile.tau-multiplecounters-mpi-papi-pdt-trace Makefile.tau-multiplecounters-papi-pdt Makefile.tau-multiplecounters-papi-pthread-pdt Makefile.tau-phase-multiplecounters-mpi-papi-compensate-pdt Makefile.tau-phase-multiplecounters-mpi-papi-pdt
Instead of using mpixlf90 as my Fortran compiler, I switch to tau_f90.sh (tau_cxx.sh/tau_cc.sh) to compile the code. To run the application, I use:
cqsub -t 12 -n 4 -e COUNTER1=GET_TIME_OF_DAY:COUNTER2=PAPI_NATIVE_PNE_BGP_PU0_FPU_MULT_1:TAU_THROTTLE=1 -p PEACEndStation -q prod-devel ./matmult
If you prefer to use qsub, please use
--env COUNTER1=GET_TIME_OF_DAY:COUNTER2=PAPI_NATIVE_PNE_BGP_PU0_FPU_MULT_1:TAU_THROTTLE=1
instead of -e ...
This measures two counters: wallclock time (GET_TIME_OF_DAY) and the native floating point multiply instructions executed on the processing unit 0 (PU0). The complete list of counters is available in PAPI's header files:
/soft/apps/V1R1M2/papi-c-3.9.0/include/linux-bgp-native-events.h
for a quick reference and the details of each event are listed on page 53 and 54 of the IBM redbook, which is accessible from http://wiki.alcf.anl.gov (a great resource!)
