Seeking Guidance on Evaluating Fine-Tuning of LLaMA LLM using Unsloth—Is MMLU a Good Metric?

Hi everyone!

I’ve been experimenting with fine-tuning a LLaMA-based LLM using the Unsloth framework and would love some guidance on evaluating the effectiveness of this fine-tuning process. I’ve read about various evaluation metrics, and one idea I had was to use the MMLU (Massive Multitask Language Understanding) benchmark, but I’m not sure if it’s the best choice for this context.

Here are a few specific questions I have:

1. Is MMLU appropriate for evaluating fine-tuning on models like LLaMA? The MMLU benchmark seems to cover a wide range of tasks, but I’m not sure if it's suitable for a specialized fine-tuning scenario like mine, or if it’s more appropriate for general language understanding.
2. What other metrics or benchmarks would you recommend? Are there specific evaluation datasets or tools that align better with the goals of fine-tuning a LLaMA model in a particular domain or task?
3. Are there any known best practices for evaluating Unsloth fine-tuning? Since I’m using Unsloth, I’d appreciate any advice or success stories on how others have evaluated their models post-fine-tuning with this framework.
4. What’s your process for iterative fine-tuning? Do you rely on quantitative metrics alone, or do you also incorporate qualitative testing? How do you ensure your model improves without overfitting?

Any advice, tips, or shared experiences would be greatly appreciated. Thanks in advance!