What precautions you can take to minimise comparisons with HP, if your story has a Magic School?

Sorry if this question is too amateur or silly, I'm pretty new to a serious writing (and also sorry for clumsy phrasing — English is not my native language). I don't expect to get any specific addition to the story from people here, but more of an advice if I need to get rid of some elements that are overused in a setting of magical schools, as I don't really read much of such literature and I expect that some people here know about this sub-genre (?) more than me.

I have a situation: my story revolves around, well, not a school, but University in a magical world. That being said, its main distinction from Hogwarts is that it treats magical and non-magical disciplines equally. Overall it's less of a Magic University and more University in a world with magic. Still, the rector/president of University is a powerful and cunning magician that plays their own game, and their character is pretty important to the overarching story. Of course, some people from the magistral stuff are important too.

Another problem. While the main conflict is very different from HP, the main protagonist is still a loner student from a family with a, ahem, not a really good mother who treats him poorly, and also has a younger child, whom she spoils. Albeit he's 20, not 11, of course. He gets a very different development from Harry, but overall the initial point is kinda similar to how we see Harry at the beginning of the first book (safe for the main character not knowing that magic exists).

Is it too much? Should I try to turn the story in a different direction, or is this amount of parallels is okay for you personally?