What linux tools can I use to see how efficiently I'm using GPU resources (Nvidia)

I'm looking for ways to see how much my models are using of these resources:

- Power consumptions in watts (I've heard of turbostat)

Main Processor/Bus utilization
- PCI bus bandwidth
- CPU utilization
- Computer RAM

GPU resources

1. Memory utilization
   - NVLink utilization
   - Memory bandwidth (local and shared (presumably with NVLink)
2. Core utilization
   - CUDA cores
   - Tensor cores (if available)

I am planning to run local models on a 4-GPU System, but those now-ancient models are either 2G or 4G in VRAM capacity (750Ti and 1050Ti). (In short, I know I'm going to be disappointed sharing 2GB cards using NVLink)

I'm also looking at refurbished cards, such as a Tesla (Kepler) K80 w/ 24G VRAM
5000 CUDA cores, but also no Tensor cores. The cards are less expensive, but I need a good way to evaluate what the price/performance of the card is and try some smaller LLM implementations.

My main goal is to get a collection of tools that allow these stats to be collected and saved.