poprhythm ad0bfb8146 Enable flash attention + quantized KV cache for ollama
Observed KV cache size for qwen2.5:7b at fixed 16384 context varying
224MB-896MB between model loads with flash attention off. The larger
figure pushes total memory needs just past the GPU's free VRAM, so
some loads only offload 25/29 layers instead of 29/29 -- causing
DJ-agent pick latency to jump from ~1s to multiple minutes. This GPU
(6GB) has very little slack for this model/context combination even
after freeing obico and stable-diffusion's VRAM reservations.

Claude-Session: https://claude.ai/code/session_01L7Rwa6guD5wK8F8tWQwcJX
2026-09-13 15:30:36 +00:00
2026-02-04 02:55:08 +00:00
2025-11-09 02:46:27 +00:00
2026-01-25 15:44:51 +00:00
2025-12-11 15:07:12 +00:00
2025-10-14 16:20:13 +00:00
2026-02-23 13:43:05 +00:00
2026-01-25 14:33:27 +00:00
2025-08-06 18:55:15 +00:00
S
Description
No description provided
Readme
877 KiB
Languages
Shell 58.2%
Python 28%
JavaScript 10.5%
HTML 2.3%
DIGITAL Command Language 0.6%
Other 0.4%