Discriminative behavior by my project manager

I'm a non EU national working in the Netherlands on a project with a deadline, and my contract runs until that date. On my team, apart from me and a friend who's in the same situation, everyone else is an EU citizen. My project manager told another team member that some people need this job to stay in the country, so they're deliberately working slowly to get the project extended. This simply isn't true, and it can be seen from our numbers. Hearing that he said this to a colleague really upset me. I'm thinking about raising this with company management since we don't have HR. Do you think this would be seen as discriminatory behavior on his part, or would I end up being seen as the one causing problems?