Stones of Barenziah

I've been collecting the stones, but never officially started the quest with Vex (I hate the multiple lines in my inventory). One stone is missing; I've been to all 24 locations twice. I've already completed the College of Winterhold quest, so that stone appeared & was collected.

Do you have to actually start the quest with Vex for all of them to appear?