Freaking out over first semester grades

Currently a freshman and I wasn't interested in premed but now I am but I am worried about my first semester grades, my gpa then was a 3.371 but I'm getting it up this semester. Is my worrying justified?