개요
DwarfStar는 DeepSeek V4 Flash/PRO 및 GLM 5.2에 최적화된 자체 포함형 네이티브 추론 엔진으로, Metal, CUDA, ROCm 백엔드를 지원하며 SSD 스트리밍 로딩과 멀티 GPU 병렬 처리 기능을 갖추고 있습니다. 또한 GGUF 생성, imatrix, 품질 테스트 및 코딩 에이전트 등의 도구를 제공합니다.
README 미리보기
\n \n\n\n**DwarfStar** is a small native inference engine optimized first for\n**DeepSeek V4 Flash**. It also supports **GLM 5.2** and, on very high-memory\nmachines, **DeepSeek V4 PRO**. It is self-contained and deliberately narrow,\nnot a general GGUF runner. Model loading, prompt rendering, tool calls, KV\nstate, the HTTP server, and the coding agent are built and tested together.\nThe repository also includes tools and data for GGUF, imatrix, quality, and speed.\n\nSupported backends:\n\n* **Metal**, the primary target, on Macs with 96 GB or more. Smaller machines\n can use SSD streaming.\n* **NVIDIA CUDA**, including multi-GPU systems and DGX Spark.\n* **ROCm** on Strix Halo systems such as the Framework Desktop.\n\nThis project would not exist without **llama.cpp and GGML**, make sure to read\nthe acknowledgements section, a big thank you to Georgi Gerganov and all the\nother contributors.\n\nModel support is intentionally opportunistic. The project follows the best open\nweights for useful local machine sizes, especially 128 GB laptops and 512 GB\nworkstations. A model may be removed when a better replacement arrives.\n\n# So, what can I do with this software?\n\n* You can run a very capable models in your consumer hardware, a MacBook, a DGX Spark, or a Strix Halo for example. Even if you have not enough RAM, with SSD streaming, you can run it at a decent speed.\n* Using the CUDA multi-GPU support and with ds4-server micro batching of decoding and generation, you can turn a server with old-ish CUDA cards (Ada Lovelace architecture), no longer supported for new models by vLLM, into a multi-user LLM server for your company. We tested this setup with 8xL40S NVIDIA cards and multiple sessions with very good results. 120 t/s aggreated generation, 2000 t/s prefill.\n* Using two MacBook M5 Max / M3 Ultra RDMA, you can run 4 bit DeepSeek Flash or GLM 5.2 with tensor parallelism.\n* You can also use pipeline paralellism to glue together multiple systems to sum their