____ _ | __ )| | ___ __ _ | _ \| |/ _ \ / _` | | |_) | | (_) | (_| | |____/|_|\___/ \__, | |___/_ _ _/_| |__ ___ _ __| |_ _ _/_ / _ \ '_ \ / _ \ '__| | | | |/ _ \ __/ |_) | __/ | | | |_| | __/ \___|_.__/ \___|_| |_|\__,_|\___|
[2025-09-30][Daily]
STARPU_SCHED=dmda STARPU_NCPU=14 STARPU_CALIBRATE=0 STARPU_TRACE_BUFFER_SIZE=1028
STARPU_GENERATE_TRACE=1 STARPU_FXT_TRACE=1 STARPU_FXT_PREFIX=./traces ./example-mnist
From last post:
We still notice some sleep time in red, this is caused by the gather functions (block_sum_z_axis,tensor_sum_t_axis...) that reduces the batch results in order to update weights and biases. This could probably be fixed by using built-in starpu reduce capabilities, because right now I simply looping through the datas and summing "by hand" at the end of each "layer batch".
Note that all experiments here will use images of 512x512x3.
So I integrated StarPU's redux system into DAHL and indeed it works flawlessly. The accumulator tasks are getting parallelized effectively, which indeed reduces synchronism as you can see:
However we still notice that some tasks are struggling to be properly parallelized (backward pooling for example). In fact StarPU is complaining a lot that some tasks were taking longer than what was estimated by the scheduler which causes a lot of sleep time. Simply changing to a regression model helps a lot!
Also I tried to play a bit with the batch size and I noticed that when I set batch_size superior to the number of CPU the parallelization struggles a bit: At the end of some layers every tasks gets queued up to the same core? Pretty weird.
On the other hand, a batch size inferior to number of CPU struggles to fill all CPU cores, which seems pretty fair.
So all of these changes are really interesting and helps a lot parallelization, however we can clearly notice the synchronization barriers between each layers. We should find a way to prevent that. One very interesting idea would be to implement a PipeDream-like algorithm, so we could fill all cores and reduce the synchronization constraints. But I wonder if we would end up with similar results, i.e. some tasks still running on one cpu at the end of each layer, which slows down the whole training? The advantage of PipeDream is that we will be able to run multiple different layers pass at the same time, which could permit StarPU to fill the empty spots.