poprhythm ad0bfb8146 Enable flash attention + quantized KV cache for ollama ...

Observed KV cache size for qwen2.5:7b at fixed 16384 context varying
224MB-896MB between model loads with flash attention off. The larger
figure pushes total memory needs just past the GPU's free VRAM, so
some loads only offload 25/29 layers instead of 29/29 -- causing
DJ-agent pick latency to jump from ~1s to multiple minutes. This GPU
(6GB) has very little slack for this model/context combination even
after freeing obico and stable-diffusion's VRAM reservations.

Claude-Session: https://claude.ai/code/session_01L7Rwa6guD5wK8F8tWQwcJX

2026-09-13 15:30:36 +00:00

..

.env.example

Add OIDC secret placeholders to .env.example files

2026-02-26 01:42:34 +00:00

docker-compose.yml

Enable flash attention + quantized KV cache for ollama

2026-09-13 15:30:36 +00:00