____ _ | __ )| | ___ __ _ | _ \| |/ _ \ / _` | | |_) | | (_) | (_| | |____/|_|\___/ \__, | |___/_ _ _/_| |__ ___ _ __| |_ _ _/_ / _ \ '_ \ / _ \ '__| | | | |/ _ \ __/ |_) | __/ | | | |_| | __/ \___|_.__/ \___|_| |_|\__,_|\___|
[2025-12-17][Daily]
Note about DDP:
Here we can't go bellow 28 batch size, because the program is launched identically, meaning that every worker will receive the same number of batch size the program has been initialized with. Instead, we should divide the batch size by the number of workers.
This also adds some drawbacks when the batch size is not divisible by the number of workers, making the comparison with DAHL harder.
Here we also notice that pytorch DDP is clearly not comparable to DAHL: it uses Single Program Multiple DATA (SPMD) which represent an enormous overhead compared to dahl. Maybe if we used CPU + GPU on a single machine it would make more sense to use DDP, but clearly, DDP would make more sense if distributing on multiple machines.
When comparing DAHL with pytorch intraop (which is the default parallelizing scheme), we notice way more comparable results:
