PyTorch Day China 2025

Video generation models have emerged as powerful tools, enabling compelling multimedia content creation across various applications. At ByteDance, we tackled the substantial challenge of efficiently scaling large-scale video generation model training to thousands of GPUs using PyTorch's robust software ecosystem.

In this talk, we will present several key innovations enabled by PyTorch to significantly enhance training efficiency:
1. Efficient Kernel Fusion with PyTorch Compile for optimized GPU computations.
2. Large-Model Training enabled by PyTorch's Checkpointing and Offloading techniques to address memory constraints.
3. Scalable Distributed Training combining Sequence Parallelism (SP) and Hybrid Sharding Data Parallelism (HSDP) for near-linear GPU scalability.
4. Load-Balancing Strategy dynamically distributing workloads for variable-length videos to eliminate performance bottlenecks.

Our session will delve into the practical methodologies, share crucial insights gained, and provide actionable best practices, empowering attendees to optimize their own large-scale model training workflows using PyTorch.