HMRC Penalty - Self Assessment

In 2023 we moved house, I updated my address with the employer and HMRC (as well as a number of other businesses that I have an account with)

For the tax year 23/34 I was expecting I would have needed to complete a Self Assessment Tax Return, for a High Income Child Benefit Charge, so I registered for a Self Assessment Tax Return ID, filling in a form online.

About 3 months ago I received a letter from HMRC advising I was being fined for not submitting a Self Assessment Tax Return, however when I applied I was told I would be contacted if I needed to submit a SATR.

The penalty notice from HMRC was sent to my old address and only because I am still in contact with the new owners did I find out about the letter, they had told me that wasn't the first letter they had received from HMRC, the other had been sent back (return to sender).

I tried appealing the penalty but HMRC denied the appeal, should I contest it? I don't think it's fair that they were sending these notices to my old address, I've included that in my appeal but they don't think it's a legitimate reason.

Any advice would be greatly appreciated.