The real bottleneck in distributed model training isn't the GPU

Adding GPUs to a training cluster without matching interconnect bandwidth just moves the bottleneck. InfiniBand or equivalent high-bandwidth, low-latency networking is what lets multi-GPU, multi-node training actually scale near-linearly instead of plateauing.