I am working on a sweater and am having some issues. Please help me figure it out.

I am making a sweater using 4 weight yarn, with 6mm hook and moss stitch. I measured a T-shirt I liked the fit of and hand to that size. Almost 6 inches down I am not sure the panel will fit my back. Any advice?