Wind, and learning FPV?

I've been learning FPV with JB's course, and have gotten to the point where I can go around some of the easier tracks consistently without crashing. So I've started trying to transfer my new skills offline, by flying my DJI Neo 2 in Manual mode.

However, what I've found is that even the gentlest wind (10-15 kph) will throw me off, and the drone be going backwards or sideways without me even realising it.

If this a common thing that everyone has to deal with? Or is more because I'm having to use a lightweight and relatively low power drone?