Using Gemma 3 with custom vision-backbone

Hello everyone,

I have a custom vision encoder trained to encode 3D CT scans and I want to use it's embeddings with a newer model like Gemma 3. I already have my embeddings offline saved on disk, is there a way to discard the gemma vision encoder and instead use my embeddings with a trained projector?