Tiny SLM LS2.5-108M-A17M-Base, Chat and Coder released.

Link to Huggingface: https://huggingface.co/collections/Dsg2/ls25

Small language models with 108M params total, 66M model (active 17M, top-2 experts) + 42M Ngram.

Trained entirely on a 5060ti, 5.7B tokens pretrain + 1B sft.

(read more detailed info and benchmarks on hugginface)