Does anyone else finds hard to imagine things read in a book?

That's something I've been struggling with my whole life, but lately I'm getting more and more annoyed with this. The problem is simple: I find really difficult to imagine the descriptions of people and places I read in books.

The thing is so annoying that lately I've been only reading non-fiction, mainly because the things described there are easier for me to relate to since I know that is "real".

Some examples: I recently read the book The City and The City by China Mieville. Although I got the idea pretty well, I couldn't imagine how the city was (hell, halfway through the book I even imagined one city was on top of the other), neither any of the characters.

Other example: In Rendezvous With Rama I couldn't imagine even the sheep, let alone the characters.

Does anyone else have these kinds of problems? It annoys you? How do you solve this?