Does family background of a girl matters to a guy while dating ??

For context, i am a girl. I am 18 years old now and i have dated 2 guys before and both of them came from upper middle class family and reputed one. And i noticed a pattern in both of the relationships that they both loved me but couldn't see the future with me because of my family background ( i had a messy and unstable family)

They never told me anything directly and loved me till the very end of the relationships but somehow i can see still that were not Confident about how their parents, dad and mom would view me. I was wondering if it actually matters which it does but how much ??