(Spoilers Extended) The key to Lyanna’s relationship with Rhaegar and Robert can be found in this lovely excerpt of ASOS - and man, did Robert dodged a bullet…

"And what if they do? I'd sooner be stolen by a strong man than be given t' some weakling by my father."

"You say that, but how can you know? What if you were stolen by someone you hated?"

"Then I'd push him in a stream or throw a bucket o' water on him. Anyhow, men shouldn't smell sweet like flowers."

"What's wrong with flowers?"

"Nothing, for a bee. For bed I want one o' these." Ygritte made to grab the front of his breeches.

“What if the man who stole you drank too much?" he insisted. "What if he was brutal or cruel?" He tightened his grip to make a point. "What if he was stronger than you, and liked to beat you bloody?"
"I'd cut his throat while he slept. You know nothing, Jon Snow." Ygritte twisted like an eel and wrenched away from him.”

(ASOS, Jon V)