Generation with SDXL LoRA just gives LoRA training images trained with Kohya_ss

Hello there,

I trained a model on my face using kohya_ss via stability matrix. When I use the lora to generate images with Juggernaut I get images similar to my training images. And the rest of the prompt, what ever the prompt may be is just ignored.

I tried lowering the LoRA weight, only 0.4 LoRA weight follows the prompt but still results in morphed image and low quality.

If I go above 0.4 then LoRA training image is generated and if I go below 0.4 then LoRA is ignored.

Here are the training parameters of the LoRA:

Data set:50 images

Epochs:5 Repeats:5

"guidance_scale": 3.5,

"learning_rate": 0.0003,

"max_resolution": "1024,1024",

here is the full pastebin link to the training json

What seems to be the issue here?