Dating my first loves brother

Ok so the best way for me to start this is all the way from the start. My mother has a friend she grew up with since the age of 4 named Lilly. Due to how close they were, my sister and I grew up with Lilly’s 5 kids. Nigel, Eddie, Hector, Hope and Matt. When I was six I had a major crush on Nigel but was scared to tell him. By the time I was 12 I found out that Nigel felt the same towards me so we briefly dated but then broke things off after a few months. Years pass and at around the age of 18 Nigel and I start rekindling old flames. At that time in my life things were pretty bad for me. I was going through a lot at home so I wasn’t fully mentally ready to be in a relationship. So after Nigel and I physically hooked up for the first time I ghosted him because I didn’t know how to tell him I wasn’t ready for what he was looking for. Many years passed and when I was 25 my mother told me that Eddie (Nigel’s brother) was caught in a really bad house fire and that he was in critical condition. Unfortunately Eddie passed away and I was with his family when it happened. I was there for emotional support for them. It was very tough to be honest. After Eddies passing Nigel and I hooked up again but at this point him and I were both already in relationships with other people, So nothing grew out of that. He did tell me that if when we turn 40 and neither of us are married that we will marry each other and I agreed. Fast forward I’m 35 with 4 kids and not married and Nigel is 36 with two kids and married. We haven’t spoken since the year of Eddies passing. About two years ago I started catching up with Hector. Started off reminiscing about our childhood and then lead to a big bombshell. Hector expressed to me how he’s always has had feelings towards me. That took me completely by surprise since I never noticed. It felt wrong due to my history with his brother. The more Hector and I have been talking the more I like him and am interested in working on something further with him. I just want so honest feedback back. If anyone has any questions I’m willing to answer.