Distributed Training Part 5: Introduction to GPU

• GPU Architecture
• How to Improve Performance with Kernels
• Fused Kernels
• Flash Attention

Distributed Training Part 4: Parallel Strategies

• Five Dimensions of Parallelization Strategies
  • batch dimension
  • hidden_state dimension
  • sequence dimension
  • model_layer dimension
  • model_expert dimension
• Optimal Training Configuration
• Tensor Parallelism（TP）
• Sequence Parallelism (SP)
• Context Parallelism (CP)
• Pipeline parallelism (PP)
• Expert Parallelism (PP)

Distributed Training Part 3: Data Parallelism

• Data Parallelism (DP)
• DP Optimization
• DP Practice
• ZeRO-1 / ZeRO-2 / ZeRO-3 (FSDP)