Distributed Training Part 2: Parallel Programming
- Broadcast
- Reduce & AllReduce
- Gather & AllGather
- Scatter & ReduceScatter
About 5 min
Distributed Training Part 1: Memory Usage in Model Training
- Model Training Process and Important Hyperparameter
- Memory Usage in Model Training
- Memory Optimization Suggestions
- Activation Recomputation / Gradient Checkpointing
- Gradient Accumulation
- Mixed Precision Training
About 10 min