Difference between E Phrygian and A minor.

I am a prog lead guitarist and have finally started writing my own material and therefore started to learn what to me is "intermediate" theory.

I wrote a verse which I recognize as both A minor and E Phrygian. They are diatonic and essentially different starting points inside of C major. I understand how the modes are built.

This made me wonder, what is the difference in terms of usage? Is it the chords backing the melody which determine the key/mode? If so, what chords would be different?

Said concisely, what determines which key/mode a song is in and when can i play which mode and why would I choose one above the other?

Thank you so much in advance! :)