____ _ | __ )| | ___ __ _ | _ \| |/ _ \ / _` | | |_) | | (_) | (_| | |____/|_|\___/ \__, | |___/_ _ _/_| |__ ___ _ __| |_ _ _/_ / _ \ '_ \ / _ \ '__| | | | |/ _ \ __/ |_) | __/ | | | |_| | __/ \___|_.__/ \___|_| |_|\__,_|\___|
[2026-10-13][Daily]
In this lab entry we explore the hypothesis that auto-adaptive task granularity could be beneficial for runtime and energy consumption on a wide range of workloads.
In the previous experiments, we noticed very early that training with small datasets (e.g. fashion-mnist, CIFAR-10...) was "overkill". What we mean is that our machine is so "powerful" that such a small workload is not efficient when parallelized at such a high granularity, i.e. to the matrix/vector scale. This behaviour be attributed to task granularity, which is too high for such a small workload, resulting in overheads. However, on small machines, it perfectly makes sense because individual tasks take longer. This mitigates any scheduling or task invocation overhead.
In fact, the problem is that when designing a parallel algorithm, the developer have to chose it's granularity. As an example the NNTile framework took a drastic choice to parallelize up to vector operations. This very high granularity is super efficient to train huge model (the order of billion parameters), but it obviously doesn't scale for moderate models. In our framework DAHL, we clearly face similar problems but at a lower scale. In the NCA article we found out that images between 10 to 200 kilopixels have poor execution proportion time on our biggest server. But this isn't the case for smaller machines.
The thing is, we will never be able to adapt to ANY combination of workload/machine. So my idea is that tasks should be able to be fused together when the granularity is too high. This way we could use simple heuristics to reduce dynamically the granularity. We could for example aim to improve the execution time proportion: StarPU gives time proportion related to workers for sleep, execution, and scheduling categories. In our case, improved execution time proportion indicates better parallelization and more efficiency.
Implementing this idea would represent huge engineering in DAHL, so for now we crafted a small hard-coded examples that only takes into account the convolution forward pass. We compare 4 different granularity modes:
Globally it gives something like this:
Granularity 1 Granularity 2 Granularity 3 Granularity 4 +-----------+ +-----------+ +-----------+ +-----------+ |Convolution| |Convolution| |Convolution| |Convolution| |Forward | |Forward | |Forward | |Forward | |Batch | |Batch | |Batch | |Batch | +-----------+ +-----------+ +-----------+ +-----------+ |Convolution| |Convolution| |Convolution| |conv2d +add| |Forward | |Forward | |Forward | +-----------+ |Sample | |Sample | |Sample | +-----------+ +-----------+ +-----------+ |Filters | |Filters | |conv2d +add| +-------+---+ +-----------+ +-----------+ |conv2d |add| |conv2d +add| +-------+---+ +-----------+
So for example at granularity 3, we do all conv2d+add for each filters in the same task. At granularity 4, we do all operations for each samples of the batch in the same task.
Now let's have a look at the obtained scheduling for each of these configurations.
Here, this granularity is the same than in DAHL and I use a dataset that I know works well on my machine. So without surprises the scheduling is pretty good, with some minor issues due to sychronization. But otherwise tasks have a decent size: ~10-50ms for convolutions, and a bit less for the task which add bias.
Now, taking a more drastic example: with my smallest dataset we can clearly see that the scheduling in inefficient, tasks ranges from ~0.08 to 0.1 ms. One might think that there isn't enough parallel task to fulfil each workers, but improving the batch size does not seem to help:
(Here we also increased the number of samples, tasks appear bigger but the zoom is just different)
So now let's try our different granularities. We will first decrease the granularity using the biggest dataset, then we'll do the same for the smaller one.
For the granularity 2 there's not much to say, only conv and add have been fused together and it clearly make sense as they act on the same buffer.
.
With granularity we clearly see tasks getting bigger: between 70 to 120ms which is close to what StarPU recommend to mitigate scheduling overhead. However, we can clearly see that wrong scheduling decisions leads to bigger "holes". Tasks are bigger, so if a worker miss the opportunity to launch a task it creating big waiting times.
Of course with granularity 4 this gives very bad results because we lose a lot of opportunities to parallelize. As a disclaimer, this result could totally be improved if we implement the whole model, because this granularity would fuse the computation of all (or parts) of the other layers.
Now let's investigate the result on the smallest dataset. So we've already seen that granularity 1 was very bad, let's have a look at granularity 2:
Here we don't notice that much differences as we only fused conv2d and add
Here it's starting to get better, as task duration are around ~0.4ms compared to ~0.07 in the granularity 1.
For the tiniest granularity so far, we obtain tasks that runs around 5 to 8ms. This is a huge improvement, however we lost the ability to parallelize along batches. In a way it makes sense, it means that for this kind of workload, and on this machine, I wouldn't have to do batch parallelism.
Nevertheless, even with these parallelization issues (that could be overcome with the full layers), we notice that granularity 4 gives better runtime than granularity 1:
andrew@senna ~/l/i/d/d/build (main) [nix] > time ./gra4 generic ../../datasets/big-fashion/26_34/ 8192 32 10
[starpu][starpu_initialize] Warning: StarPU was configured with --enable-debug (-O0), and is thus not optimized
[starpu][starpu_initialize] Warning: StarPU was configured with --enable-spinlock-check, which slows down a bit
[starpu][_starpu_init_topology] Warning: there are several kinds of CPU on this system. For now StarPU assumes all CPU are equal
[starpu][_starpu_initialize_workers_bindid] Warning: hwloc reported 14 logical CPUs for 12 cores, this is not homogeneous, will assume 1 logical CPUs per core
Epoch: 0
Epoch: 1
Epoch: 2
Epoch: 3
Epoch: 4
Epoch: 5
Epoch: 6
Epoch: 7
Epoch: 8
Epoch: 9
________________________________________________________
Executed in 16.26 secs fish external
usr time 39.42 secs 813.00 micros 39.42 secs
sys time 5.32 secs 967.00 micros 5.32 secs
andrew@senna ~/l/i/d/d/build (main) [nix] > time ./gra1 generic ../../datasets/big-fashion/26_34/ 8192 32 10
[starpu][starpu_initialize] Warning: StarPU was configured with --enable-debug (-O0), and is thus not optimized
[starpu][starpu_initialize] Warning: StarPU was configured with --enable-spinlock-check, which slows down a bit
[starpu][_starpu_init_topology] Warning: there are several kinds of CPU on this system. For now StarPU assumes all CPU are equal
[starpu][_starpu_initialize_workers_bindid] Warning: hwloc reported 14 logical CPUs for 12 cores, this is not homogeneous, will assume 1 logical CPUs per core
Epoch: 0
Epoch: 1
Epoch: 2
Epoch: 3
Epoch: 4
Epoch: 5
Epoch: 6
Epoch: 7
Epoch: 8
Epoch: 9
________________________________________________________
Executed in 57.91 secs fish external
usr time 147.92 secs 1.03 millis 147.92 secs
sys time 36.21 secs 1.20 millis 36.21 secs
For the experiments I "simulated" training on 5 epochs with 8192 samples, 32 batch size. Of course we are only doing convolutions here, not actually training a model. Here are the results:
As a reminder, gra1 is a high granularity, gra2 medium,
gra3 low, gra4 lowest possible. I have several remarks on these results:
Caveats: Here I tried on datasets between 2k to 600k pixels because I wanted to try on a larger span of images. In my previous paper I stopped at 200k. Also, we only tried on 5 epochs, which is very tiny for the smallest dataset (~5sec runtime). Trying on more small datasets, with more epochs, and more tasks (implementing the other layers) could give better results. Lastly I have no confidence interval which makes malphite / poppy results barely usable.