Detail at scale in performance analysis

Size: px

Start display at page:

Download "Detail at scale in performance analysis"

Jody Copeland
5 years ago
Views:

1 Detail at scale in performance analysis Jesus Labarta Director Computer Sciences Dept. BSC Outline On the title Performance analysis Scale Detail Some examples Visualizing variability Relevant information Instrumentation and sampling Jesus Labarta, EuroMPI, September

2 Performance analysis tools objective Generate nice color plots Who can I blame? Jesus Labarta, Detail@scale, EuroMPI, September Performance analysis tools objective Fly with instruments How is my application performing? Can I describe it in a simple way? Quantitatively? Is there anything I can do to improve its performance What? Preferably with minimum effort/cost Understand our systems Jesus Labarta, Detail@scale, EuroMPI, September

3 Scale and Detail: typical perception Scalability: It is all about size Space: #cores Time Detail: Granularity / #metrics Routine loop lines Metrics: time, message sizes, hardware counters, Size x Detail unmanageable. Scalability problem!!! drop detail Main practices: Data handling mechanisms (i.e. parallelize the tool) Profiles, aggregates, Jesus Labarta, Detail@scale, EuroMPI, September Performance analysis tools objective Fly with instruments How is my application performing? Can I describe it in a simple way? Quantitatively? Is there anything I can do to improve its performance What? Preferably with minimum effort/cost Understand our systems Information, not data Jesus Labarta, Detail@scale, EuroMPI, September

4 This talk Scalability is more an issue of dynamic range than absolute size Details ARE important To understand variability in space and time Microscopic causes of macroscopic effect We need to be able to handle/measure/analyze different levels of detail Some example techniques Jesus Labarta, EuroMPI, September Scalability Scalability is more an issue of dynamic range than absolute size 10 6 Is more a matter of intelligence (data processing) than force (data handling) First what functionality is useful, then how far can I go in size Many performance issues do give signs at small sizes (other suddenly appear at a given size) Jesus Labarta, Detail@scale, EuroMPI, September

CEPBA tools framework XML control Valgrind Extrae Trace handling & display Simulators Open Source (Linux and windows) http://www.bsc.es/paraver MRNET Dyninst,, PAPI.prv Time Analysis, filters.prv +.

5 CEPBA tools framework XML control Valgrind Extrae Trace handling & display Simulators Open Source (Linux and windows) MRNET Dyninst,, PAPI.prv Time Analysis, filters.prv +.pcf Paraver.cfg.trf DIMEMAS VENUS (IBM-ZRL) how2gen.xml Stats Gen.viz.cube.xls.txt Machine description Instr. Level Simulators PeekPerf Data Display Tools Jesus Labarta, EuroMPI, September The butterfly effect Sensitivity to initial conditions Huge impacts of small causes High non linearities with accumulative effects a Does the flap of a butterfly s wings in Brazil set off a tornado in Texas? Common in computer systems behavior Jesus Labarta, Detail@scale, EuroMPI, September

6 Interconnects a valley of butterflies 512 nodes, 4MB Dependence on appl. phase (comm. Pattern) 64 nodes, G=8, 4MB External contention Internal contention All2all - 32 Bubble propagation Propagation of internal contention 1μs delay in arrival 1.5 ms longer call duration Protocol /data messages interaction in adapter Jesus Labarta, Detail@scale, EuroMPI, September Examples Analyzing variability Histograms Scatter plots hardware counts: all in one Can be done at scale: Selective data emission Communication, Load balance, micro load imbalance, OS noise Sampling + instrumentation Jesus Labarta, Detail@scale, EuroMPI, September

7 Visualizing variability Jesus Labarta, EuroMPI, September Visualizing variability: Histograms Variability is out there, often more than we are aware of. (i.e. Load balance) Histograms of any metric Useful Duration IPC SPECFEM3D Instructions L2 miss ratio Courtesy Dimitri Komatitsch Jesus Labarta, EuroMPI, September

= continuous computation region between exit of an MPI call and entry to the next, instrumented routine, Scatter plot on some relevant metrics

8 Visualizing variability: Histograms Six months later. Useful Duration IPC Instructions L2 miss ratio Jesus Labarta, EuroMPI, September Visualizing variability: scatter plots Burst = continuous computation region between exit of an MPI call and entry to the next, instrumented routine, Scatter plot on some relevant metrics Instructions: idea of computational complexity, computational load imbalance, IPC: Idea of absolute performance and performance imbalance Automatically Identify clusters SPECFEM3D GROMACS WRF Jesus Labarta, Detail@scale, EuroMPI, September

Visualizing variability: scatter plots Time/space Distribution

2010 17 Detail as completeness of metrics Limited set Hardware

characterization of hardware counters for the different regions

9 Visualizing variability: scatter plots Time/space Distribution cores Jesus Labarta, EuroMPI, September Detail as completeness of metrics Limited set Hardware counters How can we have a complete/precise/accurate characterization of hardware counters for the different regions of a program? From a single run? Jesus Labarta, Detail@scale, EuroMPI, September

10 Emmiting relevant information Jesus Labarta, EuroMPI, September Emitting relevant data Detail for what is important, software counters(*) for what is not that important What is important? First order approach: Computation!!! MPI: a gas. Fills whatever space you give it. Very often not the major cause of problems Major computation bursts (i.e. > X ms) Entry and exit timestamps and hardware counters Communication phases. Software counters: # MPI calls, aggregated bytes, %time in MPI, (*) Jesús Labarta, Judit Giménez, Eloy Martínez, Pedro González, Harald Servat, Germán Llort, Xavier Aguilar: Scalability of tracing and visualization tools, PARCO 2005 Jesus Labarta, Detail@scale, EuroMPI, September

11 GADGET Case BGP 1024 processes 6,000 5,000 4,000 3,000 2,000 Speedup Useful duration % MPI time 1,000 0, processors S(P) Model density.c gravtree.c 246 hydra.c Speedup 0 transpose_mpi.c 385 pm_periodic.c # collectives Collective bytes # p2p p2p bytes p2p BW Jesus Labarta, Detail@scale, EuroMPI, September GADGET Case BGP 2048 processes 6,000 5,000 4,000 3,000 2,000 Speedup Useful duration % MPI time 1,000 0, processors S(P) Model Speedup 0 transpose_mpi.c pm_periodic.c 188 hydra.c 167 density.c gravtree.c # collectives Collective bytes # p2p p2p bytes p2p BW Jesus Labarta, Detail@scale, EuroMPI, September

GADGET Case A @ BGP 4096 processes 6,000 5,000 4,000 3,000 2,000 Speedup Useful duration % MPI time 1,000 0,000 0 2000

12 GADGET Case BGP 4096 processes 6,000 5,000 4,000 3,000 2,000 Speedup Useful duration % MPI time 1,000 0, processors S(P) Model density.c gravtree.c 246 hydra.c Speedup 0 transpose_mpi.c 385 pm_periodic.c # collectives Collective bytes # p2p p2p bytes p2p BW Jesus Labarta, Detail@scale, EuroMPI, September jugene 1 iteration 8K cores 12K cores 16K cores Jesus Labarta, Detail@scale, EuroMPI, September

PFLOTRAN @ jugene network traffic Bytes on X

dimension Bytes on 3 dimensions 400K Jesus

2010 25 PFLOTRAN @ jugene - Detailed network

Collective send bytes 110K Bytes out of node

Can we improve the way we manage and use

13 jugene network traffic Bytes on X dimension Imbalance on link/direction utilization will limit communication performance Bytes on Y dimension Bytes on Z dimension Bytes on 3 dimensions 400K Jesus Labarta, Detail@scale, EuroMPI, September jugene - Detailed network traffic Zoomed region in previous slide Collective send bytes 110K Bytes out of node 400K How much network bandwidth do we need? Can we improve the way we manage and use networks? Bandwidth <15MB/s Jesus Labarta, Detail@scale, EuroMPI, September

PFLOTRAN @ jaguar Jacobian KSPSolve Color indicates cluster ID Length indicates computation burst

Analysis of PFLOTRAN on Jaguar CScADS Workshop on Performance Tools for Petascale Computing August

Default (pin to core) 488 seconds Explicit Pin to Core ( fastest ) 463 seconds Color indicates Cycles

14 jaguar Jacobian KSPSolve Color indicates cluster ID Length indicates computation burst length Outliers as small as ~0 seconds! K. Huck et all. Analysis of PFLOTRAN on Jaguar CScADS Workshop on Performance Tools for Petascale Computing August 2-5, 2010 Jesus Labarta, Detail@scale, EuroMPI, September Jaguar: OS noise impact Default (pin to core) 488 seconds Explicit Pin to Core ( fastest ) 463 seconds Color indicates Cycles per microsecond Pin to CPU (NUMA) 455 seconds No pinning (slowest) 620 seconds (timelines not to scale) Jesus Labarta, Detail@scale, EuroMPI, September

PFLOTRAN @ Jaguar: OS noise impact zoomed view Default Color indicates Cycles per microsecond Pre-emptions have significant effect In FLOW stage Pin

..but not in the TRAN stage (timelines not to scale) Jesus Labarta, Detail@scale, EuroMPI, September 2010 29 PFLOTRAN @ Jaguar: Spare core results no

15 Jaguar: OS noise impact zoomed view Default Color indicates Cycles per microsecond Pre-emptions have significant effect In FLOW stage Pin to Core ( fastest )...but not in the TRAN stage (timelines not to scale) Jesus Labarta, Detail@scale, EuroMPI, September Jaguar: Spare core results no improvement 682 nodes, 7502 total cores 538 seconds 150 Seconds! 744 nodes, 8184 total cores 448 seconds 682 nodes, 6820 total cores 566 seconds 819 nodes, 8184 total cores (last 6 unused) 536 seconds (timelines not to scale) Jesus Labarta, Detail@scale, EuroMPI, September

16 Example PEPC tasks on Jaguar Duration of the computation bursts # of MPI collective operations Jesus Labarta, Detail@scale, EuroMPI, September jugene: 8K cores Useful durartion MPI calls Microscopic load imbalance!!!! Jesus Labarta, Detail@scale, EuroMPI, September

Variability in microscopic behavior GROMACS: Only computation phases parallelized with SMPSs SMPSs tasks and MPI calls ( ~ multispectral) Jesus Labarta,

17 Variability in microscopic behavior GROMACS: Only computation phases parallelized with SMPSs SMPSs tasks and MPI calls ( ~ multispectral) Jesus Labarta, Detail@scale, EuroMPI, September Variability in microscopic behavior Four loops/routines Sequential order Jesus Labarta, Detail@scale, EuroMPI, September

18 Instrumentation + sampling Jesus Labarta, Detail@scale, EuroMPI, September Instrumentation Events correlated to specific program activity Start/exit iterations, functions, loops, Start Iter Start Iter MPICall fb MPICall fb MFLOPS Different intervals: May be very large, may be very short Variable precision Captured data:: Hardware counters, call arguments, call path,. Accurate statistics: profiles, Jesus Labarta, Detail@scale, EuroMPI, September

19 Sampling Events uncorrelated to program activity (at least not specific) Time (or counter) overflow MPICall fb MPICall fb MFLOPS Controlled granularity: Sufficiently large to minimize overhead Guaranteed acquisition interval/precision Statistical projection %time (or metric) = f( %counts ) Assuming no correlation, sufficiently large #samples Jesus Labarta, Detail@scale, EuroMPI, September Instrumentation + sampling Both Start Iter Start Iter MPICall fb MPICall fb MFLOPS Guaranteed interval Captured data: Hardware counters (since previous probe) call path Call arguments in some probes Jesus Labarta, Detail@scale, EuroMPI, September

Large data size Jesus Labarta, Detail@scale, EuroMPI, September 2010 39 Large data size Safe sampled functions

20 Instrumentation + Sampling High sampling frequency (>> Nyquist) Guaranteed detail. Probably useful for many analyses. Large data size Jesus Labarta, Detail@scale, EuroMPI, September Instrumentation + Sampling High sampling frequency (>> Nyquist) Guaranteed detail. Probably useful for many analyses. Large data size Safe sampled functions MFLOPS at each interval Instrumented MPI calls Jesus Labarta, Detail@scale, EuroMPI, September

21 Instrumentation + Sampling High sampling frequency (>> Nyquist) Guaranteed detail. Probably useful for many analyses. Large data size Safe sampled functions MFLOPS at each interval Instrumented MPI calls Jesus Labarta, Detail@scale, EuroMPI, September Sampling frequency Trade-off: Too low no detail Too high too much overhead Challenge: Can we get lot of detail, very fine grain information : i.e. instantaneous performance metric rates With very little overhead: ie. sampling a few times per second Work by Harald Servat Jesus Labarta, Detail@scale, EuroMPI, September

22 New roles Instrumentation Reference Identify different instances of a region for which to obtain detailed time evolution of metrics Stationary behaviour assumed Target region: Iteration Routine Routine excluding MPI calls MPICall fb MPICall fb Harald Servat et all.. Detailed performance analysis using coarse grain sampling PROPER, 2009 H. Servat Folding: providing detailed performance metrics using coarse grain sampling UPC-DAC-RR Jesus Labarta, Detail@scale, EuroMPI, September New roles Sampling role relative data Guarantee granularity Provide data to increase granularity MPICall fb MPICall fb Jesus Labarta, Detail@scale, EuroMPI, September

EuroMPI, September 2010 45 Folding counters: Fitting Eliminate outliers

23 Folding counters: Projecting Cumulative count since reference Variance in duration Eliminate outliers Scale Jesus Labarta, EuroMPI, September Folding counters: Fitting Eliminate outliers Kriging interpolation Jesus Labarta, EuroMPI, September

Impact of the number of folded instances The more samples being fold, the more detailed results Longer executions Increase frequency Reach stability?

2010 47 Impact of the number of folded instances Experiments comparing few samples per second to 1000 times higher sampling frequency.

24 Impact of the number of folded instances The more samples being fold, the more detailed results Longer executions Increase frequency Reach stability? Example: NAS BT class B copy_faces showing from 10 to 200 iterations 20 samples per SGI Altix Jesus Labarta, Detail@scale, EuroMPI, September Impact of the number of folded instances Experiments comparing few samples per second to 1000 times higher sampling frequency. Not necessary to fold a very big number of instances potential application even in slowly time varying programs. Jesus Labarta, Detail@scale, EuroMPI, September

Emitted data Timelines Performance counters: Sample again fitted

specifying routines of interest Jesus Labarta, Detail@scale, EuroMPI,

Could think of emitting an analytical expression NAS BT Scalability

!!! Even if generating traces Example (Gadget2 using 128 tasks) 100 its,

25 Emitted data Timelines Performance counters: Sample again fitted function and inject synthetic events into trace Call stack Truncated by specifying routines of interest Jesus Labarta, EuroMPI, September Emitted data MIPS MFLOPS Plots, statistics Time, IPC, Could think of emitting an analytical expression NAS BT Scalability impact!!!! Even if generating traces Example (Gadget2 using 128 tasks) 100 its, 5 samples/s during 90minutes ~ 236MB Folding on samples/s ~ 64 MB ALYA SIESTA Jesus Labarta, Detail@scale, EuroMPI, September

26 PfloTran (data obtained with 5 samples/s) Jesus Labarta, Detail@scale, EuroMPI, September PEPC (data obtained with 5 samples/s) Jesus Labarta, Detail@scale, EuroMPI, September

27 Summary Jesus Labarta, EuroMPI, September Summary Performance tools are more and more needed!!!!! To tune our applications, to design our system software. To understand what really happens, how our systems really behave, Great progress is taking place Functionality Scalability: Dynamic range Detail IS important and can be obtained/handled A lot of open research I have seen things you people wouldn't believe... Roy Batty Blade Runner Seeing is believing... measuring is better Free adaptation of a Spanish saying Jesus Labarta, Detail@scale, EuroMPI, September

Performance Analysis with Vampir VIRTUAL INSTITUTE HIGH PRODUCTIVITY SUPERCOMPUTING

Performance Analysis with Vampir Outline Part I: Welcome to the Vampir Tool Suite Event Trace Visualization Vampir & VampirServer The Vampir Displays Part II: Vampir Hands-On Visualizing and analyzing