Graphics upgrade for Alienware x51

Hey guys wondering if I van get any insight into what a possible graphics card upgrade could be. I have the i7 3770 core 330w power supply and the gtx 660. I tried a msi gtx 1070 and it wouldn't even turn on. I assume to much power? I tried a msi 1050 as well but experienced some overheating and eventually loss in performance and fps possibly due to bottlenecking.

If anyone has experience they could share with me that would be great. I would like to know what could likely be a successful upgrade with my current build. Thanks!

Edit: sorry I meant I tried the 1060 not the 1070