⚠ Switch to EXCALIDRAW VIEW in the MORE OPTIONS menu of this document. ⚠ You can decompress Drawing data with the command palette: ‘Decompress current Excalidraw file’. For more info check in plugin settings under ‘Saving’
Excalidraw Data
Text Elements
Compute Stream
Communication Stream (A2A)
Stream
BASELINE(No Overlap)
F/Attention
F/Dispatch a2a x 2
F/MLP
F/Combine a2a x 1
B/Combine a2a x 1
B/MLP
B/Attention
Overlap waith W/D Split mlp
F/Attention
F/Dispatch ag + a2a
F/MLP
F/Combine a2a x 1
B/Combine a2a x 1
B/ DgradMLP
B/ WgradMLP
B/Dispatch a2a
B/Dispatch a2a x 1
B/Attention
Speed Up
permute rank-expert major
permute rank-expert major
permute rank-expert major
permute rank-expert major
Compute Stream
Communication Stream (A2A)
F/Attention
F/Dispatch ag + a2a
F/MLP
F/Combine a2a
permute rank-expert major
Overlap waith W/D Split mlp and 2 micro batch
AMB(Another Micro Batch)
B/Combine a2a
AMB
B/ DgradMLP
B/ WgradMLP
permute rank-expert major
B/Dispatch a2a
AMB
AMB
AMB
AMB
B/Attention
AMB
Speed Up
当前的 comb 来看的话 如果先调 forward 会导致 AMB 的 backward 的 mlp 放不到 forward mlp 前面