Managing shared GPU servers - looking to chat with others who deal with this

At my job I manage 2 servers, 4 GPUs each. The problem is we have more people than GPUs, especially when people use more than one.

During peak times it gets messy - coordinating who needs what, asking people to free up resources, etc. Our current solution is basically

talk to each other and try to solve the bottleneck in the moment.

I'm thinking about building something to help with this, and here's where you come in:

I'm looking for people who work with or manage shared GPU servers to understand:

- What issues do you run into?

- How do you deal with them?

Would love to chat privately to hear about your experience!