I'm new to all this LLM stuff and I've been trying to see if I can adapt the Hugging Face language modeling example to use MPT-7B, but it runs out of memory on the training step basically no matter what I do--in regular Colab, using Colab Pro, and using various configurations of GPUs in RunPod (I've tried pods ranging from one 40GB A100 to 4 80GB A100s).
Has anyone fine-tuned MPT-7B successfully? What system configuration did you use? Did you have to do anything special in code, like using PyTorch's DistributedDataParallel or anything in that vein?