Anyone else getting flammed when building manamune?

For the double tear build you're supposed to get muramana first and finish that before stacking AA. But when my teammates see the ryze building muramana they think I'm trolling. Any one else getting flammed almost every game. And how do you usually tell them off?