Skip to content

Trace Generation

Jump to bottom
dchasap edited this page Sep 28, 2023 · 1 revision

Tracing Overview

MUSA targets hybrid MPI+OpenMP/OmpSs applications. In this section we describe how to generate and merge the necessary traces for such an application, but it's possible to only trace OpenMP/OmpSs applications, by skipping any MPI related step.

Typically MUSA requires three distinct tracing runs:

  1. Trace MPI runtime events
  2. Trace OpenMP/OmpSs runtime events
  3. Trace instructions and memory events

Step 1 and 2 can be run simultaneously, reducing the tracing runs to two, but the 3rd step needs to be run separately, otherwise the overhead of the instrumentation affects the timing of the higher level events traced in steps 1 and 2. In this guide we will show how to trace an application doing only two tracing passes.

To meet the requirements for tracing an hybrid application it is required to provide two binaries complied with mercurium. One version linked against the nanos performance library (called ${BIN}), and a second version linked against the instrumented nanos library (called "${BIN}_instr" ).

Compiling a hybrid application using mercurium as compiler, requires to specify the runtime MPI library at compilation and linkage time. As an example, the following commands;

  • Performance version
mcxx -c --ompss -I${MPI_PATH}/include -o code.o code.c 
mcxx -L${MPI_PATH}/lib -lmpi -o code code.o
  • Instrumented version
mcxx -c --ompss --instrument -I${MPI_PATH}/include -o code.o code.c 
mcxx -L${MPI_PATH}/lib -lmpi -o code_instr code.o

Both must be present in the same directory (symlinks are ok).

Upon successful installation of musa, you should be able to call launch_musa_tracer.bash. Three arguments are required, the number of MPI ranks, the binary to trace and the arguments to that binary. Moreover, there are a number of environmental variables that can be set, which specify tracing mode and other execution parameters.

1. Generate MPI and OpenMP/OmpSs trace

The following variables are required:

export MEMORY_MODE=0 # 0 for burst tracing and 1 for detailed tracing.  Default is 0.
export PARALLEL_INSTRUMENTATION=1 # set to 0 if you want to generate MPI and OpenMP/OmpSs traces in two separate runs.  Default is 1.

After setting the variables simply execute the following command:

launch_musa_tracer.bash $RANKS $BIN $ARGS

2. Generate detailed instruction and memory trace

MUSA is capable of simulating the execution of tasks and reconstruct the tasks execution for a specified number of cores. This is based on reconstructing the execution of burst per tasks (burst mode) or by simulating the concurrency on the memory hierarchy (detailed mode). For the second, it is required to obtain the memory instructions from the sequential execution. Obtaining the memory trace is expensive in terms of execution time and disk space. As such, typically the use will want to only trace a certain ROI of the application. To run launch_musa_tracer.bash in memory mode and select a ROI, the following variables need to be set

The following variables are required:

export MEMORY_MODE=1 # 0 for burst tracing and 1 for detailed tracing
export MEM_PHASES_INIT=1 # The first OmpSs phase to trace
export MEM_PHASES_NUM=1 # How many OmpSs phases to trace

The above will trace the first OmpSs phase only. After setting the variables simply execute the following command:

launch_musa_tracer.bash $RANKS $BIN $ARGS

At the end, the memory trace is merged against the performance trace.

An example SLURM script for tracing can be found in src/tasksim/examples/step1_tracing.

Note: If you are using SLURM or any other job scheduler, increase the number of ranks by one. That is for the tracing tools that runs on the background.

Clone this wiki locally