How do you wean yourself away from pinying?

I am self studying. I realised just like in English I have a hard time following formal spoken stories without actually seeing the word. Very much like I need a book and audio to fully understand, or I need to have subtitles on to understand a movie.

I am having a hard time memorizing the characters. And if I listen to a story with only Chinese text, i have no clue what is going on. But if I listen to the same text but it has pinying, I can understand the story.