____ _ | __ )| | ___ __ _ | _ \| |/ _ \ / _` | | |_) | | (_) | (_| | |____/|_|\___/ \__, | |___/_ _ _/_| |__ ___ _ __| |_ _ _/_ / _ \ '_ \ / _ \ '__| | | | |/ _ \ __/ |_) | __/ | | | |_| | __/ \___|_.__/ \___|_| |_|\__,_|\___|
[2025-11-18][Daily]
Commands:
STARPU_SCHED=dmda STARPU_TRACE_BUFFER_SIZE=1028 STARPU_GENERATE_TRACE=1 STARPU_FXT_TRACE=1
STARPU_FXT_PREFIX=./traces ./basic_cnn
In these experiments I used big fashion dataset. However images were too big to be loaded in memory:
#sample* nx * ny * nz * sizeof(double)
44441 * 1440 * 1080 * 3 * 8 * 10^-9 ~= 1658 Gigabyte.
So I cropped them to 270x360x3.
Oh by the way, it hasn't been mentionned in the previous posts but green = executing a task, red = sleeping, so we try to maximize green :)
When running on my laptop (14 cpu) vs rammus (28 cpu + rtx 4060), we notice that things are getting better parallelized on my laptop. We notice strange things on GPU execution, while most task are effectivitly 10 times faster than CPU, some of them take as much time.
We still notice too much dependency constraints on the conv2d backward filters because it produces a checkerboard on cpu + gpu.
On the gpu only run, everything is greatly parallelized (well there's only one GPU :)) We again notice drastic task lenght differences, even on same codelets.For example, conv2d has one execution of 29milliseconds, but most of the others are around 0.03 milliseconds
On the cpu only run, parallelization is better, but we still notice drawbacks of checkerboard and waiting in between the layers.
All the previous experiments were ran with batch_size = 60. Now we'll compare different batch sizes.
With a batch size of 120, we see solid blocks when scheduling especially for the forward passes, however we still see the same issues during the backward.
We get the same behavior when reducing to batch size 30.
Work should be done to tackle the different issues in the backward pass.