Bg2.0 rear sight

So yeah, mine shoots to the left too. After reading online about the factory sights, I took my sight pusher to the range with me. After about a 100 rounds to just get used to it, I shot from a bag to group it. Was gonna push the rear sight to the right a bit but honestly felt like I was forcing the issue with how hard I was cranking the pusher. Is there some set screw or something under the striker? This thing is in there tight, didn't budge one bit

How'd you move/remove yours? I'm sure folks aren't carrying these and just planning to hold inches to the side