I don't want to attend another ward

My family has been bullied by a popular, "high status" family in our ward for years, so we have chosen to participate less (not take callings, attend less often, etc.) because we don't want the conflict/drama in our lives. When I explained this to my Bishop (who asked to meet with me/ I've been trying to avoid this conversation for years), he basically invited us to attend a different ward in the same stake. I don't want to do that because people in the "new" ward will see we don't live the ward boundaries, and we'll be perceived to be operating out of compliance with norms, phone trees will light up, and we'll be bullied all over again. I've basically decided to stop attending altogther because the our ward is too toxic, gossipy, etc. Any helpful advice or recommedations would be appreciated.