tokenspeed
Frameworklightseekorg/tokenspeed
Speed-of-light LLM inference engine for production agentic workloads.
Overview
TokenSpeed is an open-source LLM inference engine targeting TensorRT-LLM-level performance with vLLM-level usability for agentic workloads. It separates a C++ finite-state-machine control plane from a Python execution plane, enforcing safe KV cache and request lifecycle management at compile time. Its pluggable kernel system includes one of the fastest MLA implementations on Blackwell GPUs.
README Preview
\n \n\n\nTokenSpeed is a speed-of-light LLM inference engine designed for **agentic workloads**, with TensorRT-LLM-level performance and vLLM-level usability. The goal is to be the most performant inference engine for production agentic workloads.\n\nCore components:\n\n- **Scheduler**: C++ control plane and Python execution plane. Request\n lifecycle, KV cache ownership, and overlap timing are encoded as a\n finite-state machine, with safe KV resource reuse enforced by the type system at compile time.\n- **Kernels**: pluggable, layered kernel system with a portable public API and\n a centralized registry including one of the fastest **MLA**\n (Multi-head Latent Attention) implementations on Blackwell for agentic workloads.\n- **Entrypoint**: SMG-integrated AsyncLLM for low-overhead CPU-side request\n handling.\n\n## Differentiation\n\nTokenSpeed takes a fundamentally different architectural approach from existing LLM inference engines. The **PyTorch Foundation highlighted these architectural differentiators** when [welcoming TokenSpeed](https://pytorch.org/blog/pytorch-ecosystem-landscape-q3-update/) to the PyTorch Ecosystem Landscape:\n\n> TokenSpeed is an open source LLM inference engine and the first to separate the control plane from the execution plane. The control plane is implemented in C++ as a finite-state machine, using the type system to enforce safe resource management, including request lifecycles and KV cache state, at compile time rather than runtime. The execution plane is implemented in Python, enabling fast iteration and lowering the cognitive load for researchers and engineers. This architecture combines strong correctness guarantees in the core scheduling system with the development velocity of a high-level execution layer.\n>\n> TokenSpeed also treats kernels as a first-class, modular subsystem, separating them from the core engine through a portable public API, centralized registry and selection model, and an extensible plugin mechanism
FAQ (2)
TroubleshootingWhy does a single request with n>1 return identical completions when using the triton sampling backend?
Known bug: prepare_parallel_sampling_replica in parallel_sampling.py shallow-copies the tokenized request and only swaps the rid, so all n replicas share the same SamplingParams and seed. The triton sampler derives its noise from tl.randint(seed, offset) (ops/sampling/triton/gumbel.py), so every replica draws the same token at each step — while the request still returns HTTP 200 with correct usage accounting, making it a silent failure. Workarounds: (1) Send N separate requests with n: 1 instead of one request with n: N; per-request sampling produces distinct completions (confirmed at temperatures 0.7 and 1.2), at the cost of losing prefill sharing and batching. (2) On NVIDIA GPUs, use the default --sampling-backend flashinfer, whose Philox RNG includes the batch row (curand_init(philox_seed, bx, ...)), so replicas diverge naturally. Also rule out a lookalike cause: the backend defaults to greedy on non-NVIDIA platforms, which ignores temperature/top_p/top_k/seed while still returning 200. Verify the active backend with curl http://<host>:<control-port>/get_server_info | grep sampling_backend. This matters for GRPO-style RL rollouts, where n>1 identical completions produce zero-variance advantages and no training signal.
TroubleshootingWhy do some benchmark steps run ~36% slower with random ~1 second stalls when serving on AMD gfx950 at high concurrency?
The stalls are caused by JIT kernel compilation happening mid-serving instead of at startup. Two sources: (1) The scheduler aggregates prompts into prefill batches up to 8192 tokens, but the default prefill graph capture only covers up to 2048 tokens, so large prefill batches hit uncaptured paths and compile kernels while serving. (2) Attention kernels have many compile-time specializations that are not exercised during warmup. Fix: set the server-side prefill graph capture ceiling to match the scheduler's maximum prefill batch size (e.g., 8192 tokens) so compilation happens at capture time, not during requests. Additionally, run more representative warmup requests that cover the attention specializations used in production. Diagnosis tip: the signature is a fixed ~0.9s stall visible in the tail of TTFT (p90/p99) with median TTFT and TPOT unchanged, and all requests still completing successfully.