shing3232

u/shing3232





r/LocalLLaMA•

MLA + FA now only uses K-cache - 47% saving on KV-cache size (only for use with #13435 for now) by jukofyork · Pull Request #13529 · ggml-org/llama.cpp

llama_kv_cache_unified: kv_size = 163840, type_k = 'f16', type_v = 'f16', n_layer = 61, can_shift = 0, padding = 256

llama_kv_cache_unified: CUDA0 KV buffer size = 10980.00 MiB

llama_kv_cache_unified: KV self size = 10980.00 MiB, K (f16): 10980.00 MiB, V (f16): 0.00 MiB

The full context of 160k tokens now takes up less than 11GB without kquants

13435

r/StableDiffusion•

https://github.com/mit-han-lab/nunchaku/discussions/236

🚀 Performance

  • First-Block-Cache: Up to 2× speedup for 50-step inference and 1.4× for 30-step. (u/ita9naiwa )
  • 16-bit Attention: Delivers ~1.2× speedups on RTX 30-, 40-, and 50-series GPUs. (@sxtyzhangzk )

🔥 LoRA Enhancements

  • No conversion needed — plug and play. (@lmxyy )
  • Support for composing multiple LoRAs. (@lmxyy )
  • Compatibility with Fluxgym and [FLUX-tools](htt
7416