DeepGEMM
SHA-256DeepSeek's high-performance GPU kernel library for FP8/FP4/BF16 GEMMs and fused MoE, matching CUTLASS-level performance with a much simpler, JIT-compiled codebase.
Smart Download
Download Download Version
v2.1.1.post3 · 8.5 MB
A clean, JIT-compiled GPU kernel library that runs LLM matrix math—FP8/FP4 GEMMs and fused MoE—at peak speed on NVIDIA Hopper and Blackwell GPUs.
Core Features
- FP8, FP4 and BF16 GEMM kernels reaching up to 1550 TFLOPS on H800
- Mega MoE fuses expert dispatch, both linear layers, SwiGLU and combine into one kernel with NVLink communication overlap
- DeepJIT runtime compilation means zero CUDA build at install time
- Deliberately small, readable codebase that doubles as a learning resource for Tensor Core optimization
- Grouped GEMMs in contiguous and masked layouts plus MQA logits kernels for the DeepSeek V3.2 lightning indexer
What It Can't Do
- •Only works on NVIDIA SM90 (H100/H800) and SM100 (B200) GPUs—consumer RTX cards and older datacenter GPUs like the A100 are unsupported. This is a low-level developer library, not an end-user application: there is no GUI and no turnkey model runner. Input transposition and FP8 casting must be handled by your own preceding kernels. It requires a Linux environment, CUDA 12.9+, PyTorch 2.3+ and a C++20 compiler with <format> support, so setup is non-trivial. Mega MoE additionally needs multi-process launch with symmetric memory, which adds deployment complexity.
Use Cases
- Accelerating FP8/FP4 training and inference for large language models such as DeepSeek V3
- Serving MoE models with fused expert computation overlapped with NVLink communication
- Studying NVIDIA Tensor Core, TMA and JIT kernel optimization techniques from production-grade code
Detailed Introduction
DeepGEMM is a unified tensor core kernel library from DeepSeek that brings together the core computation primitives of modern LLMs—FP8, FP4 and BF16 GEMMs, fused MoE with overlapped NVLink communication (Mega MoE), MQA scoring for the lightning indexer, and HyperConnection—into one clean CUDA codebase. All kernels compile at runtime via DeepJIT, so installation requires zero CUDA compilation. It borrows ideas from CUTLASS and CuTe but deliberately avoids heavy template machinery, keeping only a small set of core kernels. That makes it both a production-grade library—reaching up to 1550 TFLOPS on H800—and one of the most readable resources for learning NVIDIA GPU kernel optimization. It supports SM90 (H100/H800) and SM100 (B200) GPUs, grouped GEMMs in contiguous and masked layouts for MoE training and decoding, and utilities for TMA alignment, PDL and deterministic execution. Requires CUDA 12.9+, PyTorch 2.3+ and a C++20 compiler. MIT licensed.
Troubleshooting & FAQ (2)
TroubleshootingWhy does DeepGEMM's test_mqa_logits fail with bitwise mismatches on SM120 Blackwell GPUs, and is the FP8 MQA logits kernel output safe to use?
The failure has two independent causes on SM120 (e.g., RTX PRO 6000 Blackwell, deep_gemm 2.6.1 nv_dev): (1) The FP8 self-consistency assert fails due to nondeterministic *out-of-window* writes — in compressed mode the fp8_fp4_mqa_logits kernel stores garbage into output cells beyond each row's valid [ks, ke) window, so the test's full-buffer bitwise comparison mismatches. Deeper isolation (30 repeat launches, shapes up to 4096x8192, H=64 D=128) showed the valid-region output is bit-identical and NaN-free, so window-respecting consumers are unaffected: only read logits within each row's [ks_i, ke_i) range and treat the test failure as a test artifact, not a data race on valid data. (2) The MXFP4 path with H=64, D=128 has a genuine accuracy failure (diff ~0.62 vs 0.02 tolerance) with no fix yet — avoid the FP4 indexer path on SM120 until a per-arch synchronization fix lands, analogous to the earlier SM90 (#229) and SM100 (#285, commit 96fbced) MQA sync fixes.
TroubleshootingHow to fix "AttributeError: module 'deep_ep' has no attribute 'ElasticBuffer'" when running the mega_moe kernel?
This error occurs because ElasticBuffer is a new API introduced in DeepEP v2 — it does not exist in DeepEP v1. To fix it, upgrade your DeepEP installation to v2 (see https://github.com/deepseek-ai/DeepEP/pull/605). After updating DeepEP, the deep_ep.ElasticBuffer attribute will be available and the mega_moe kernel should run without this error.
Tags
Getting Started
Install the software
Double-click the downloaded installer and follow the prompts
Make sure you have an SM90 or SM100 NVIDIA GPU (H100, H800 or B200) with CUDA 12.9+ and PyTorch 2.3+ installed
Clone the repository with git clone --recursive so the CUTLASS submodule is included
Run ./install.sh from the project directory to build and install the package
- Make sure you have an SM90 or SM100 NVIDIA GPU (H100, H800 or B200) with CUDA 12.9+ and PyTorch 2.3+ installed
- Clone the repository with git clone --recursive so the CUTLASS submodule is included
- Run ./install.sh from the project directory to build and install the package
SHA-256 checksum verified
Checksum extracted from GitHub official Release page
SHA256 Checksum
5b34ed544d56312b257320343fc98ea9bc704f512d38cfcca35f649ef209ded5This checksum is extracted from the GitHub Release page. Verify file integrity after download.
All SHA-256 checksums on this platform are extracted from the project's official GitHub Release page, without any modification. You can independently verify them on the GitHub Releases page.
Open Source Transparency
View GitHub SourceUninstall Info
Run pip uninstall deep_gemm to remove the Python package, then delete the cloned source directory. Optionally clear the runtime JIT compilation cache in your user cache folder for a complete cleanup.
No Extra Dependencies
Ready to use after download. No additional runtime required.
Having issues? Check the FAQ below
2 FAQs
Similar Projects
ollama
Ollama lets you download, run, and manage large language models locally. One command, multiple platforms, endless possibilities.
llama.cpp
High-performance LLM inference engine in C/C++ with minimal dependencies, supporting quantized models (1.5–8 bit) and diverse hardware (Apple Silicon, CUDA, Vulkan, etc.).
mempalace
Open-source local-first AI memory system with verbatim storage and semantic search — zero API calls, no keys, top benchmark scores.