RTX3090 x 2 + 3060 speed test.

Recently, i had the opportunity to try out rtx 3060, alongside my other two rtx 3090s (60gb total). Extra 12gb allowed me to load a Q5_K_M variant of a 72b model. My intention was to keep the generation speed around normal "reading speed", which was successful.

I thought this might be useful info for anyone looking to add a 3060 into their rig for extra VRAM buffer, as it's CERTAINLY better than offloading to CPU.

Here's a benchmark:

Running benchmark (Not Saved)...

Processing Prompt [BLAS] (8092 / 8092 tokens)

Generating (100 / 100 tokens)

[13:48:30] CtxLimit:8192/8192, Amt:100/100, Init:0.84s, Process:18.27s (2.3ms/T = 442.86T/s), Generate:13.21s (132.1ms/T = 7.57T/s), Total:31.48s (3.18T/s)

Benchmark Completed - v1.79.1 Results:

======

Flags: NoAVX2=False Threads=7 HighPriority=False Cublas_Args=['normal', 'mmq'] Tensor_Split=[0.398, 0.402, 0.2] BlasThreads=7 BlasBatchSize=256 FlashAttention=True KvCache=1

Timestamp: 2025-02-20 11:48:30.069486+00:00

Backend: koboldcpp_cublas.dll

Layers: 83

Model: Evathene-v1.3.i1-Q5_K_M

MaxCtx: 8192

GenAmount: 100

-----

ProcessingTime: 18.272s

ProcessingSpeed: 442.86T/s

GenerationTime: 13.207s

GenerationSpeed: 7.57T/s

TotalTime: 31.479s