Need advice on being able to drill properly. Concrete / mortar / fly ash brick wall.

Hi folks. I have a consistent issue where every time I try to drill a hole, especially in a tight spot where I might be getting into concrete rather than brick, my drill hole just goes crooked, despite the fact that it has already gone in at least half an inch. I’m left with wonky holes. For instance I was drilling to install curtain brackets. The left side was pretty close to the corner and being right handed, I did not have a good angle. Needed two holes, 4mm, about three ish cm apart. First hole goes in and within half an inch drifts off to the side giving me a wide opening. I’m pretty sure I didn’t get into any concrete over there, it would have been fly ash brick. I’m kinda going nuts about this. How best do I deal with this? One out of three times, I have this issue.

PS: I’m posting this pretty late in the night so I’ll reply in the morning.