impossiblefork

Gemma 3n is said to have a per-layer embedding, which I interpret as one token embedding per layer added in somewhere (I haven't read through any reference implementation, only looked at https://ai.google.dev/gemma/docs/gemma-3n).

Embeddings end up being more than half the parameter budget, and I suppose this is to some degree simply okay, but others, for example Gloeckle et al. in https://arxiv.org/abs/2404.19737 talk about how having one extra unembedding matrix for each extra position to be predicted is unacceptable memory-wise.

My own suspicion is Gloeckle et al. are simply wrong in this assessement and that having a bunch of extra embedding/unembedding matrices is fine.