Accelerates large-scale LLM pretraining using PyTorch-native 4D parallelism and Float8 optimization.
This skill enables Claude to architect and execute high-performance LLM pretraining workflows using TorchTitan, PyTorch's official platform for distributed training. It provides standardized patterns for implementing composable 4D parallelism—Fully Sharded Data Parallel (FSDP2), Tensor Parallel (TP), Pipeline Parallel (PP), and Context Parallel (CP)—allowing for efficient scaling from 8 to over 512 GPUs. By leveraging advanced features like Float8 training on H100s, torch.compile, and distributed checkpointing, this skill helps developers and researchers build Llama 3.1, DeepSeek, or custom models with significant speedups and optimized memory efficiency.
Key Features
01Distributed checkpointing with HuggingFace interoperability
02Standardized TOML configuration for Llama 3.1 and custom models
03Multi-node scaling with SLURM and torchrun integration
04Native 4D Parallelism (FSDP2, TP, PP, and CP)
05Float8 training support for H100 GPU performance boosts
063,983 GitHub stars
Use Cases
01Implementing complex 4D parallelism strategies for long-context models
02Optimizing training throughput on H100 clusters using Float8 and torch.compile
03Pretraining large language models like Llama 3.1 405B from scratch
