“Works with” list

My “Works with” list under the Organization tab in my email contact card is fairly accurate. These are colleagues I work with regularly, and they seem to be in order of most to least collaborated.

However, I am not on anyone else’s “Works with” list, which I find surprising. Is there something in my settings that is affecting this?