How do I know how much to be doing for warm up sets?

Currently a beginner so pretty weak. For example, my seated dumbbell shoulder press is only twenty in each hand currently. When I was doing it the other day, I did 10 reps of ten pounds and then 6 reps of 15 pounds for warm up before going into my working set.

The issue is that I seemed to get fatigued faster during my working set compared to when I just jump straight into doing the 20s, but I want to warm up to avoid injuries. Is my understanding of warm up not correct or is my working set too heavy and that is why I’m getting fatigued faster?