My partner (21M) and I (21F) are amazing 90% of the time, but the 10% is making me go insane

We started out as close friends. Would go hiking, watch movies, talk for the longest times, found comfort in each others presence. The beginning was great and organically we became lovers.

We have been lovers for around 4 years. Our first argument was had about two years ago (ish), a communication issue. It is hard for him to stay focused in a conversation with me without being distracted by silly things around him (strange labels seen around the room, images, his phone, etc).

When we were younger this was a minor issue, but as we have grown together, and our relationship has deepened, this hurts me a lot. Especially since this habit of his results in him not remembering the events I tell him about and I end up having to retell the stories. I usually would not react if he forgot a small detail, but some of these are eventful moments, or important details about me (such as my allergies).

Me getting hurt by him not remembering details about me has become a recurring problem. When I bring this up, he responds that he does remember other irrelevant things, or that he has issues remembering, and often is after work and uni. I am always met with defense which makes me think that we are not on the same wavelength.

He fills every box, loving, charming, handsome, funny, cultured, social. I know that we are young, but we are so amazing together most of the time, and he is mature in other aspects, I don’t think that I am ready to break it off.

Can I work on my patience, or the way that I bring this up? Would anyone have experience with a similar sort of one-sided communication issue and could provide some insight?