I'm a bit conflicted with the direction my server is going.

To give you an idea of what's going on, I originally wanted the server to be a PvE survival server, no pvp (unless consent was given from all players taking part), and no griefing. Instead, I took the route I thought would attract more players. It's a PvP server, griefing is allowed in the wilderness, and we are using towny. At this point, we have gained a good amount of players, but I just don't like the direction the server is headed. The amount of concurrent players is actually decreasing. The rest of the staff don't want the server to change, considering we've already gotten so far. So r/admincraft, I'm coming to you for advice. What should I do?