ds4

Overview

DwarfStar is a self-contained native inference engine optimized for DeepSeek V4 Flash and PRO, also supporting GLM 5.2. It runs on Metal, CUDA, and ROCm backends, with SSD streaming and multi-GPU support, plus tools for GGUF generation, imatrix, quality testing, and a coding agent.

README Preview

\n  \n\n\n**DwarfStar** is a small native inference engine optimized first for\n**DeepSeek V4 Flash**. It also supports **GLM 5.2** and, on very high-memory\nmachines, **DeepSeek V4 PRO**. It is self-contained and deliberately narrow,\nnot a general GGUF runner. Model loading, prompt rendering, tool calls, KV\nstate, the HTTP server, and the coding agent are built and tested together.\nThe repository also includes tools and data for GGUF, imatrix, quality, and speed.\n\nSupported backends:\n\n* **Metal**, the primary target, on Macs with 96 GB or more. Smaller machines\n  can use SSD streaming.\n* **NVIDIA CUDA**, including multi-GPU systems and DGX Spark.\n* **ROCm** on Strix Halo systems such as the Framework Desktop.\n\nThis project would not exist without **llama.cpp and GGML**, make sure to read\nthe acknowledgements section, a big thank you to Georgi Gerganov and all the\nother contributors.\n\nModel support is intentionally opportunistic. The project follows the best open\nweights for useful local machine sizes, especially 128 GB laptops and 512 GB\nworkstations. A model may be removed when a better replacement arrives.\n\n# So, what can I do with this software?\n\n* You can run a very capable models in your consumer hardware, a MacBook, a DGX Spark, or a Strix Halo for example. Even if you have not enough RAM, with SSD streaming, you can run it at a decent speed.\n* Using the CUDA multi-GPU support and with ds4-server micro batching of decoding and generation, you can turn a server with old-ish CUDA cards (Ada Lovelace architecture), no longer supported for new models by vLLM, into a multi-user LLM server for your company. We tested this setup with 8xL40S NVIDIA cards and multiple sessions with very good results. 120 t/s aggreated generation, 2000 t/s prefill.\n* Using two MacBook M5 Max / M3 Ultra RDMA, you can run 4 bit DeepSeek Flash or GLM 5.2 with tensor parallelism.\n* You can also use pipeline paralellism to glue together multiple systems to sum their