The purpose of this post is twofold. To give hope to those with older video cards, and to solicit further optimizations from the larger community. Here's the script:
cat qwen.sh
#!/bin/bash
# Configuration Variables
# ---------------------
MODEL_PATH="../models/DeepSeek-R1-Distill-Qwen-14B-Q4_K_M.gguf" # Path to your GGUF model file
LLAMA_SERVER_PATH="./build/bin/llama-server" # Path to your llama-server executable
N_GPU_LAYERS=99 # Number of layers to offload to GPU (use 99 to offload as much as possible)
N_CTX=14384 # Context window size (tokens). Adjust based on VRAM and model needs.
PORT=NNNNN # Port for the llama-server API
# --- Perform