____  _             
| __ )| | ___   __ _ 
|  _ \| |/ _ \ / _` |
| |_) | | (_) | (_| |
|____/|_|\___/ \__, |
               |___/ 
     _               _            
 _/_| |__   ___ _ __| |_   _  _/_ 
/ _ \ '_ \ / _ \ '__| | | | |/ _ \
  __/ |_) |  __/ |  | | |_| |  __/
\___|_.__/ \___|_|  |_|\__,_|\___|

< Previous post | Next post >

[2025-12-09][Daily]

Does StarPU calibration impact performances?

STARPU_CALIBRATE=0 STARPU_NCUDA=0 STARPU_SCHED=dmda STARPU_TRACE_BUFFER_SIZE=2048 STARPU_GENERATE_TRACE=1 STARPU_FXT_TRACE=1 STARPU_FXT_PREFIX=./traces ./basic_cnn
# and
STARPU_CALIBRATE=1 STARPU_NCUDA=0 STARPU_SCHED=dmda STARPU_TRACE_BUFFER_SIZE=2048 STARPU_GENERATE_TRACE=1 STARPU_FXT_TRACE=1 STARPU_FXT_PREFIX=./traces ./basic_cnn

I was wondering if enabling/disabling STARPU_CALIBRATE (which controls task monitoring to compute performance models) would change a lot the scheduling. On both examples we obtain pretty similar results, over 25 batches. The parallelization is pretty consistent and honestly decent with few waiting times.

Trying with GPU+CPU, we notice parallelization issues. Our program struggles a lot to parallelize efficiently between the cores and the GPU. We notice big chunks of task (both on CPU and GPU) but then sparsely scheduled task with a lot of waiting times. We might have several problems:

• The same task granularity problem applies to the GPU, some tasks are way to tiny and we're losing considerable time due to scheduling overhead.
• Data transfers do take long time and seems to outweight (or at least reduce) the gain that we could have by using the GPU.
• convolution_2d_backward_filters have a similar runtime on CPU and GPU (~11, ~7ms respectively). This is pretty bad and should probably be tackled.

Running on GPU with bigger tasks, we notice no parallelization issue when running on GPU with a bigger dataset (using factice dataset 512x512x3). Our task are >10ms so this is great.

GPU+CPU parallelization show better results when running on bigger tasks, yet we still have some big overheads represented by the brown colors on the graph. Note that I ran this experiment both with and without starpu calibrate and we obtained similar results. This might not affect too much the experiments.

Also these experiments were tested without starpu blocking drivers to see if it would lead to better parallelization, but again here results were pretty similar.

However, we can conclude that bigger task on GPU reduces overhead just as on CPU, and also makes it easier for starpu to schedule tasks between cpu/gpu. It is still not totally perfect though.

< Previous post | Next post >