Skip to content
#

kv-cache-quantization

Star

Here are 20 public repositories matching this topic...

Language: All
Filter by language
All 20 Python 14 C++ 2 Jupyter Notebook 1 Shell 1
Sort: Most stars
Sort options
Most stars Fewest stars Most forks Fewest forks Recently updated Least recently updated

AI-Efficiency / Awesome-Model-Quantization

Star 2.5k

A curated collection of papers, benchmarks, surveys, and tools for model quantization, covering low-bit networks, LLMs, multimodal and generative models, vector and lattice quantization, and efficient deployment.

awesome awesome-list binarization awesome-lists vector-quantization binary-neural-networks efficient-inference multimodal low-precision diffusion-models model-quantization kv-cache quantization-aware-training post-training-quantization data-free-quantization large-language-models quantizations kv-cache-quantization lattice-quantization large-language-models-llms
  • Updated Oct 5, 2026

jaylfc / taOS

Sponsor
Star 556

Self-hosted AI agent OS. Your memory, chat, agents, and files stay on hardware you own, offline by default, cloud by choice. Offline AI memory (taOSmd), self-hosted multi-framework group chat, a full web desktop + app store, and auto-clustering across the consumer hardware you already have (Orange/Raspberry Pi, Mac mini, gaming PC).

raspberry-pi privacy offline-first distributed-computing self-hosted orange-pi ai-agents data-sovereignty ai-platform local-first agent-framework apple-silicon llm vllm local-llm llm-inference kv-cache-quantization rockchip-npu turboquant
  • Updated Oct 8, 2026
  • Python

Sandermage / sndr_core_engine

Star 133

SNDR Core Engine (Genesis) — vLLM runtime patch-overlay for Qwen3.6 + Gemma4 on consumer NVIDIA (Ampere sm_86, 2× A5000/3090). Qwen3.6-35B-A3B FP8 ~240 tok/s, 27B-int4 hybrid GDN+Mamba, Gemma4 26B/31B AWQ, 256K ctx. 321 patches: TurboQuant k8v4 KV, MTP/DFlash spec-decode, FULL cudagraph, hybrid GDN. vLLM pin dev424 + Control Center GUI.

memory cuda self-hosted quantization gemma rag awq llm-serving pgvector vllm local-llm llm-inference qwen speculative-decoding kv-cache-quantization tool-calling consumer-gpu rtx-4090 rtx-3090 spec-decode
  • Updated Sep 30, 2026
  • Python

liventruth / UL-SMF-Cache-Compression

Star 9

The Unified Latent-State Memory Fabric (UL-SMF) is a hardware-software co-designed memory compression fabric that solves the memory bottleneck in long-context Transformer inference. By combining FSQ with dynamic 16-dimensional latent mapping, UL-SMF compresses Key-Value (KV) cache tensors by up to 384x while maintaining >94% semantic retention.

machine-learning compression pytorch artificial-intelligence llama gemma compression-methods compression-implementations mistral vram pytorch-implementation kv-cache llm llms vllm qwen kv-cache-quantization kv-cache-compression vram-optimization vram-optimizer
  • Updated Aug 23, 2026
  • Python

Henvezz95 / VAR-Compressor

Star 5

W4A4 and INT8 KV-cache quantization for Infinity VAR models. Optimized for high-fidelity generative AI deployment on edge GPUs (e.g. NVIDIA Jetson).

computer-vision pytorch gpu-acceleration quantization model-compression nvidia-jetson inference-optimization edge-ai on-device-ml weight-quantization post-training-quantization autoregressive-models generative-ai kv-cache-quantization activation-quantization visual-autoregressive-model svdquant infinity-var
  • Updated Jun 11, 2026
  • Python

IonDen / mlx-quant-fidelity

Sponsor
Star 4

Measure quantization quality loss on Apple Silicon MLX — KL divergence, top-token flip rate and perplexity delta for KV-cache and weight quantization

python machine-learning metal diagnostics quantization model-evaluation mlx kl-divergence perplexity model-quantization kv-cache apple-silicon llm local-llm llm-inference llm-eval kv-cache-quantization mlx-lm quantization-quality quantization-benchmark
  • Updated Sep 29, 2026
  • Python

DominikBucko / qwen38-flash-next-3090

Star 4

Run Qwen3.8-Flash-Next on ONE RTX 3090 (24 GB) + 64 GB RAM: 128K context, up to 2,100 tok/s prefill, 43–51 tok/s decode. vLLM runtime with hot MoE experts on the GPU and cold experts computed on the CPU, INT8 KV cache, Docker, OpenAI-compatible API.

moe mixture-of-experts single-gpu rtx3090 int4 llm long-context vllm local-llm llm-inference qwen kv-cache-quantization openai-compatible-api consumer-gpu rtx-3090 cpu-offloading qwen38 qwen3-8-flash-next 128k-context 24gb-vram
  • Updated Sep 30, 2026
  • Python

JakeATX / llama-cpp-qwen-ampere

Star 1

Qwen3.8-27B on Ampere (RTX 3090 / 3090 Ti): TurboQuant+ llama.cpp fork with SM86 kernel and memory work, 245K context with the model's own MTP head. Model: sjakek/Qwen3.8-27B-ATX-4-XS-GGUF on Hugging Face.

cuda quantization mtp ampere llm long-context llama-cpp ggml local-llm llm-inference qwen gguf speculative-decoding kv-cache-quantization qwen3 consumer-gpu multi-token-prediction rtx-3090 turboquant rtx-3090-ti
  • Updated Sep 23, 2026
  • C++

Add this topic to your repo

To associate your repository with the kv-cache-quantization topic, visit your repo's landing page and select "manage topics."

Learn more