Silk cami fit

For those who have the silk cami and have worn the cashmere sweaters- how does the fit compare across the product types? The xs sweaters fit me perfectly, but I’m worried that I should size up with the silk…