Quant & Confidence: Will I Ever Improve?

Hi all

First, I want to say thank you to this community in that it has been very helpful to me for the past couple of months.

I am struggling with confidence in Quant! I have been out of high school since 2011 and haven't touched basic math concepts since 2008. In April I re-learned old concepts by watching Magoosh videos and completing easy problems. However, I'm really struggling with the hard/very hard problems and it's killing my confidence to the point where I have been avoiding studying for the past two weeks.

Some background: I am NOT very good at math, and never have been. My math knowledge only goes to Calc II, and I haven't touched math (aside from calculating drug doses and dilutions) since 2012. I am applying to Neuroscience PhD programs this year and would like to score 160 on each section to get into some of the top programs. I'm extremely confident in some concepts (Data Interpretation and Stats are very familiar to me due to daily exposure) but others are really giving me trouble on the Hard/Very Hard levels (Counting and Number Properties, mostly).

I am also very slow. I need the calculator. I write down everything and have to sit and think through every concept for a minute before tackling the problem. Some of the time, I can get the problem, but I am 20-40 seconds slower than the average person (according to Magoosh). With Magoosh, out of 20 Hard & Very Hard problems, I might get 60%. With the 5Lb Manhattan Book, I'll do 20 problems at a time concerning a single concept and maybe get 70-80% correct. Right now on Magoosh my projected score is 152-157 which is generous in my opinion. When I took the 5LB book's diagnostic test, I scored in the 153-155 range. Compared to a lot of posts on here, that is abysmal.

Basically, I think I am approaching everything wrong in studying. I'm not improving, just getting frustrated. Any tips on how to improve 1) time, and 2) confidence? I haven't signed up for the test yet because I am too nervous to commit, but I have a full time, very draining and intense job, which makes studying difficult. Is it even possible for me to score 160 with my lack of intrinsic math ability?