Just a small update since last week’s major rework, I decided to add Data Parallel mode to Raylight as well. FSDP now splits the model weights across GPUs while still running the full workload on each one.