Lore Question: What lore established reasons are out there for a character that cannot use magic?

Soon my group will be starting our first foray into D&D. The DM and I already have experience with PnP RP systems, and have played the old PC cRPGs (Baldurs Gate 1/2 Icewind Dale 1/2, Neverwinter Nights 1/2 and Planescape: Torment) so we aren't entirely uninitiated with the universe of D&D. However neither myself nor the DM are overly familiar with the universe as a whole. I'm rolling a Lawful Neutral rogue character, designed to be somewhat like a 'thief-taker', a lawful counter-rogue in the Forgotten Realms setting. Part of his backstory is that he has a major fascination with magic but has never had any aptitude for it. He will be using Use Magic Device heavily, but part of the characters drive is to find out exactly why he cannot use magic inherently. I've self imposed the rule that the character can never multiclass into any magic using class, similar to some sort of Pariah from 40k (this doesn't mean magic doesn't affect him of course). What sort of lore friendly reasons are out there for this? I'm aware that hey, we can just make some stuff up, but the campaign we are going to run is the start of a much larger campaign idea we have, and I would like to have some sort of universal lore reason for the character being afflicted so. It is entirely possible I will bring this character to other groups, and would like as little house rule convolutions on him as possible. This doesn't necessarily need to be a Forgotten Realms specific thing, but that is where we are starting so its relevant.

Edit: Directions to any relevant reading material would be much appreciated, there is so much background lore out there that it is very difficult to find something on this specific thing.