(Spoilers Main) Why do people believe Ned's story about Jon being a bastard?

Robert's Rebellion started because Rheagar kidnapped and "raped" Lyanna Stark, this is the catalyst that everyone in Westeros seems to accept for the war.

At war's end, Ned returns to King's Landing with the body of Lyanna and a newborn baby boy. Ned and Robert reconcile over their mutual heartbreak about Lyanna, and Ned explains that the baby is his from a dead woman he had an affair with.

Given Robert's relationship with Ned, I can understand why he accepts Ned's story about Jon, but why do the other lords agree with this flimsy story?

From what we know of John Arryn and Hoster Tully, they're both very shrewd dudes who know Ned isn't the type to randomly sleep around, and it's hard to image Hoster being ok with Ned taking a bastard baby back to his pregnant daughter who could potentially usurp her own children one day.

Then there's the matter of both Tywin and Varys who never seem to question Jon's true parentage. How do two of the most strategic minds that ever played the Game of Thrones not realize that Ned returning to King's Landing with a baby and Lyanna's corpse roughly a year after she was "raped" by Rhaegar doesn't seem oddly coincidental? The Master of Spies should have no problems verifying whether Ned's rushed backstory is true or not, so was everyone just holding an idiot ball that day and Ned got lucky?