fp8

A library for accelerating Transformer models on NVIDIA GPUs, including using 8-bit and 4-bit floating point (FP8 and FP4) precision on Hopper, Ada and Blackwell GPUs, to provide better performance with lower memory utilization in both training and inference.

cuDNN Frontend is NVIDIA's modern, open-source entry point to the cuDNN library and a growing collection of high-performance open-source kernels.

Microsoft Automatic Mixed Precision Library

An innovative library for efficient LLM inference via low-bit quantization

Kernel Library for Large Headdim Attention (64~1024, BF16/FP8/FP4), 1.5x~15x↑ vs PyTorch SDPA.

Flux diffusion model implementation using quantized fp8 matmul & remaining layers use faster half precision accumulate, which is ~2x faster on consumer devices.

Optimized SGLang runtime for Qwen3.8-27B FP8 with DFlash2 and Qwen3.8 Flash-Next NVFP4 with FR-Spec on one NVIDIA RTX PRO 6000 Blackwell 96 GB GPU (SM120): 524K context, HiCache and NIXL.

Qwen3.8-Flash-Next on 2× RTX 3090: with 128 GB RAM up to 4,191 tok/s prefill · 111.5 tok/s decode (131K prompt), full 256K window at 2,865 tok/s; with 64 GB RAM 3,410 tok/s prefill · 84 tok/s decode. New: opt-in uncensored mode (runtime abliteration, no new weights).

Measured SM121 compatibility recipe for MiniMax H3 FL2VA on one NVIDIA DGX Spark with vLLM-Omni and online FP8.

Serving Qwen3.8-27B-FP8 on a single DGX Spark (GB10): 7.88 to 58.5 tok/s single-stream from decode strategy alone, weights untouched. Speculative decoding and prefix caching benchmarked, plus DFlash 2 — the only Qwen3.8-27B build that can serve it under vLLM.

Pure Go machine learning framework. Train, run, and serve ML models with go build. Zero CGo.

Lynn 原生 LLM 推理引擎 · W4A8/NVFP4 量化 · 自写 CUDA/Triton kernel · MoE · 投机解码 | Lynn-native LLM inference engine for NVIDIA Blackwell

FP8 Metal compute kernels for Apple Silicon MPS — fixing what PyTorch doesn't support yet. FLUX/SD3.5/ComfyUI on Mac.

Benchmarks and notes for running modern LLMs with vLLM on 8x Tesla V100-32GB in 2026.

LLM inference engine for AMD RDNA4 — Rust + Vulkan compute shaders, gguf & native FP8.

JAX Scalify: end-to-end scaled arithmetics

Minimal SGLang patch for Qwen3.8-27B + DFlash2 on dual RTX 3080: TP-sharded Draft fc and static per-head FP8 KV.

Automated Triton w8a8 block FP8 kernel tuning tool for vLLM. Auto-detects model architecture, supports Qwen3-Coder-30B-A3B-Instruct-FP8/DeepSeek-V3/custom models, multi-GPU parallel tuning, and generates optimized kernel configs for quantization.

Qwen3.8 Flash Next AWQ INT4 4x3090 v1: 262K, Vision, FP8 QSA KV and CUDA Graphs on 4x RTX 3090

Source-code level analysis of LLM RL training infra: async RL, weight sync, FP8, MoE routing | LLM RL 训练基础设施源码级分析