Car window only works when it's been in the sun

I have a strange problem The driver side window only works if the car is sitting on the sun (when is hot), if it's in the shadow or after dark the window won't roll down or up. When I push the button I can hear a click like the motor is trying to work but that's it I'm going to disassemble the door maybe it's just a loose connection but I'm looking for other ideas too
What could be the problem?