9B single-GPU prefill is 1.5–3× llama.cpp at every length up to ~4.6 K tokens (2026-05-02 default flip, commit ca30368). With split-K FlashAttention on by default (commit f2e52b8), 9B 18 K prefill jumps to 1.22× llama.cpp and 27B 3-GPU 18 K reaches parity (0.99×). Generation continues to win by +30–50%. Multi-GPU prefill is now pipelined (per-GPU compute / D2H / H2D streams + double-buffered hidden chunks): 9B dual-GPU 18 K hits 1.32× llama.cpp and 27B 3-GPU 18 K hits 1.45× — the long-context multi-GPU gap is closed. See Honest Benchmarks.

A custom CUDA inference engine for Qwen3.5 / Qwen3.6 hybrid (GDN + Attention) models, written from scratch and tuned for NVIDIA mining cards (CMP 100-210, ex-mining V100) — 16 GB HBM2, sm_70, PCIe Gen1 x1, no P2P. Not a fork — every kernel is written for these constraints.

📖 한국어 README → README.ko.md

A note on authorship: I'm not a native English speaker, so this README was written in Korean and translated with an LLM. The code was written by Claude (Anthropic) across many sessions; the direction, architecture decisions, debugging, hardware diagnosis, and benchmarks are mine.

• Serves Qwen3.5 / Qwen3.6 dense hybrid models in 27B and 9B sizes (GGUF Q8_0). Qwen3-MoE / Qwen3.x-A3B GGUFs are also auto-detected (per-layer ffn_gate_inp router) and run through a grouped-expert FFN — but that MoE path is experimental and not yet validated against real MoE weights, so treat it as untested, not a supported configuration.
• v2-MTP inline GGUF support (2026-05-27): nextn head packed as blk.N.nextn.* in the GGUF is auto-detected and used as the spec drafter — no external mtp_head_*.bin needed. Accept rate measured ~83 % on Qwopus3.6-27B-v2-MTP in our testing (vs ~73 % for the legacy external head).
• Vision input via Qwen3-VL mmproj (ViT + M-RoPE + spatial reshape).
• OpenAI-compatible HTTP API (/v1/chat/completions, /v1/models, streaming, tool calls).
• Embeddings + reranking from the same binary (--mode embed / --mode rerank): Qwen3-Embedding-4B (last-token pool, L2-normalized) and Qwen3-Reranker-4B (cross-encoder via the cls.output classifier head, instruction-aware). The chat server reverse-proxies /v1/embeddings and /v1/rerank to these sidecars.
• Qwen3 thinking control: /no_think (and /think) directives in user or system messages, plus vLLM-style extra_body.chat_template_kwargs.enable_thinking — all resolve to the same force_think switch in the chat template.
• Continuous batching across N concurrent slots, with per-slot prefix caching.
• MTP draft speculative decoding (K=1) — works. DFlash block-diffusion speculative decoding is lossless on the engine side (block-diffusion draft + chain tree-verify; verify uses greedy argmax at temp≤0 and samples from the target distribution at temp>0, so the output distribution matches the target either way). Out of the box a stock drafter mismatches a distilled model (accept length ≈ 1–3, little speedup); with a distribution-matched drafter accept length reaches ~4.5–5 and throughput is projected in the mid-30s to upper-40s t/s on 27B Q8_0 / 3-GPU. A pipelined-fold path (default-on; DFLASH_FOLD=0 to disable) folds the bonus token into the verify batch for +15–22% — quality-equivalent (verified: identical compiled-C++ coding pass-rate) but not bit-identical (see Limitations).
• 3-bit KV cache (MTP_TQ) — Walsh-Hadamard rotation + Lloyd-Max scalar quant. Same family of idea as llama.cpp #21038, but 3-bit Lloyd-Max instead of 4-bit RTN.
• Fused gate+up GEMM (MLP_GATEUP_FUSED_KERNEL=1, default off): two Q8_0 weights share one Q8_1 input tile in SMEM, +5–10 % prefill on v2 models.
• Multi-GPU layer-parallel split with pinned-host activation bridge (no P2P required).

Mining cards (CMP 100-210, ex-mining V100) are dirt-cheap on the secondhand market and have HBM2 + 16 GB of VRAM that nobody is buying back, but NVIDIA cripples them in software:

• Tensor Cores throttled — HMMA latency stretched 64× (8 → 512 cycles), hard cap ~5 TFLOP via cuBLAS WMMA.
• PCIe Gen1 x1 only, no P2P, no NVLink.
• CUPTI blocked — no vendor profiler, no torch.profiler.
• All of this is enforced below the driver — PMU/SEC2 secure boot on the die, at a privilege level user code can't reach. No software unlock exists for these Volta cards; the newer GA100 sibling (CMP 170HX) was cracked in software in 2026, but through a GSP-era boot primitive Volta doesn't have, so it doesn't port.

So vLLM, llama.cpp's default cuBLAS path, FlashAttention, bitsandbytes — anything that goes through cuBLAS Tensor Cores runs at 1/64 speed or fails outright.

qengine works around it by:

• Routing the LLM's GEMMs through DP4A (int8) at ~17 TFLOP — the path that is not throttled on CMP. (HFMA2 fp16 SIMD is also unthrottled at ~24 TFLOP; in this engine it's used by the vision encoder, while the LLM weights run Q8_0 over DP4A.)
• A hand-written Q8_0 GEMM tile path for prefill.
• A FlashAttention-style fused attention layout that, for max_seq > 32 K, avoids the VRAM-heavy strict per-score buffer (its own custom kernel, not cuBLAS).
• Pinned-host activation bridge between GPUs (since P2P is unavailable).

It's not faster than llama.cpp at everything. See the honest benchmarks below.

All measurements on a CMP 100-210 host, same Q8_0 GGUFs (Qwopus3.5-9B-v3.5, Qwopus3.6-27B-v1-preview), batch 1, single inflight request, FA on, layer split, split-K FA on by default. Both engines built for sm_70 with the int8 (MMQ / DP4A) path. qengine numbers are server-side prefill wall (excludes SSE handshake) for bench_curl.sh-style real chat-completion prompts; llama.cpp numbers are llama-bench at matching prompt sizes. Bigger is better; bold = winner. Single-GPU 9B measured 2026-05-02; 27B 3-GPU and split-K 9B 18 K measured 2026-05-03. llama.cpp build 8462.

qengine: 1.62–2.99× on prompts up to ~4.6 K, 1.22× at 18 K (split-K FA, 1.46× over the pre-split-K build). Generation +51% on the comparable short-context point (70.4 vs 46.6 t/s).

Cross-GPU prefill pipelining lands 2026-05-03 (default on; PREFILL_NO_PIPELINE=1 to opt out): per-GPU compute / D2H / H2D streams + double-buffered host transfer + double-buffered per-GPU hidden chunks overlap chunk i's cross-GPU activation transfer with chunk i+1's downstream compute. 9B dual-GPU 18 K jumps from 259 → 720 t/s (2.78×) and now wins llama.cpp at every length (1.32× even at 18 K). Sampled tokens are bit-equivalent to the sequential path — verified against the per-token greedy argmax up to 18 K.

qengine: 2.86× / 2.07× / 1.84× / 1.45× at 297 / 1.16 K / 4.62 K / 18 K — pipelining lifts the long-context number from parity (139, split-K only) to a clean 1.45× win. Generation +53% at 297 ctx (27.1 vs 17.7 t/s).

• 9B single-GPU prefill: qengine wins at every length now, 1.22–2.99×. The chat-app sweet spot.
• 9B dual-GPU prefill: qengine now wins at every length (1.32× at 18 K) thanks to cross-GPU pipelining.
• 27B 3-GPU prefill: qengine wins everywhere, 1.45–2.86×. The parity gap at 18 K is gone.
• Generation throughput: qengine wins by ~30–50% on both 9B and 27B. This is what users feel as the chat being responsive.

Honest take: most of the surface-level features overlap. The list below is what actually differs in practice. Not measured head-to-head where not stated — corrections / PRs welcome.

• Generation throughput at sm_70 + CMP — measured +30–50% over llama.cpp on this exact hardware. See benchmarks above.
• OpenAI Chat Completions API built into the engine binary — streaming, image_url, tool/function calls, no separate server process. llama.cpp has llama-server which covers most of this.
• MTP_TQ uses 3-bit Lloyd-Max + WHT — llama.cpp's #21038 already lands rotation + standard scalar quant types (q4_0 etc.). Ours is 3-bit (vs 4-bit), which gives a slightly higher compression ratio on KV. Whether this beats q4_0 RTN on perplexity is not yet verified head-to-head.
• Continuous batching with per-slot prefix snapshots — not unique conceptually. The integration with our scheduler is tight; whether it actually beats llama.cpp's batched server is not yet measured.
• Qwen3-VL multimodal — we have it. So does llama.cpp via tools/mtmd/models/qwen3vl.cpp. Not an advantage.
• DFlash block-diffusion speculative decoding — functional and lossless on the engine side (block-diffusion drafter + chain tree-verify, with custom dp4a small-batch GEMV and an SMEM-resident GDN chain-scan for the verify). The catch is the drafter: a stock drafter trained on vanilla Qwen3.5 mismatches a distilled model's output distribution (low accept rate), so you need one trained/fine-tuned on your model's outputs. With a matched drafter, accept length ≈ 4.5–5 and the pipelined-fold path (default) adds another +15–22%.

Designed for / regularly tested on:

• 4× NVIDIA CMP 100-210 (Volta GV100, 16 GB HBM2, sm_70, PCIe Gen1 x1, no P2P)
• Total 64 GB VRAM, ~8 GB system RAM (yes, eight)

Should also work on (sm_70 / sm_72 / sm_75):

• V100 16/32 GB (much less throttled than CMP — should be faster)
• Titan V, Quadro GV100
• T4, RTX 20-series (sm_75) — untested, kernels target sm_70 paths

Will not work on:

• sm_60 or earlier (no DP4A)
• AMD / Apple Silicon

If you have a modern GPU (RTX 30/40/50, A100, H100), you should use vLLM or SGLang instead. They are far more optimized for those targets and have actual test coverage.

Requires CUDA 12.x, GCC 11+, CMake 3.18+.

git clone https://github.com/Haru-neo/qengine.git
cd qengine
mkdir build && cd build
cmake ..
make -j$(nproc)

Recommended config — 3 GPUs (12 % faster prefill than 4 since transfer is one hop shorter; the v2 Q8_0 28 GB model fits in 3×16 GB with the rebalanced split) + fused gate+up + sparse-attention profile:

CUDA_VISIBLE_DEVICES=0,1,2 \
MTP_TQ=1 MLP_GATEUP_FUSED=1 MLP_GATEUP_FUSED_KERNEL=1 \
MINF_SPARSE_ATTN=1 MINF_BUDGET=0.10 \
MINF_PROFILE_PATH=./profiles/27B_block_sparse.bin \
./build/qwen-engine \
  /path/to/Qwopus3.6-27B-v2-MTP-Q8_0.gguf \
  --serve 8000 --max-seq 262144 --slots 1 \
  --vision-mmproj /path/to/Qwopus3.6-27B-v2-mmproj.gguf

For v1 models (external MTP head): drop MLP_GATEUP_FUSED_KERNEL=1 (the fused kernel only matches the v2 numerics in our testing) and place mtp_head_<hidden>.bin under /home/paru/mtp_work/.

QWEN_SLOTS=4 ./build/qwen-engine \
  /path/to/Qwen3.5-9B-Q8_0.gguf \
  --serve 8001 --max-seq 32768

CUDA_VISIBLE_DEVICES=0,1 ./build/qwen-engine ... --serve 8001

The same binary serves Qwen3 dense 4B embedding and reranker models via --mode embed and --mode rerank. Both 4B Q8_0 GGUFs fit on one GPU (~4 GB each), so a spare CMP card can host both sidecars. The chat server reverse-proxies /v1/embeddings and /v1/rerank to the sidecars so clients only need to know about the chat port:

# embed sidecar (port 8001, GPU 3)
CUDA_VISIBLE_DEVICES=3 ./build/qwen-engine \
  /path/to/Qwen3-Embedding-4B-Q8_0.gguf --serve 8001 --mode embed

# rerank sidecar (port 8002, GPU 3 — shared with embed).
# Stagger: wait for the embed sidecar to finish loading before launching
# this one. Bringing up two CUDA contexts on the SAME GPU simultaneously
# intermittently corrupts the second one on CMP 100-210.
CUDA_VISIBLE_DEVICES=3 ./build/qwen-engine \
  /path/to/Qwen3-Reranker-4B-Q8_0.gguf --serve 8002 --mode rerank

# chat with proxy (clients hit only :8000)
./build/qwen-engine /path/to/27B-Q8_0.gguf --serve 8000 \
  --proxy-embed  127.0.0.1:8001 \
  --proxy-rerank 127.0.0.1:8002 \
  ...

A turnkey launcher that brings up all three (with the embed→rerank stagger baked in) is at scripts/launch_all.sh.

OpenAI-compatible embedding request:

curl http://localhost:8000/v1/embeddings -H "Content-Type: application/json" \
  -d '{"model":"qwen-embed","input":["the cat sat","a feline rested"]}'
# → {"object":"list","data":[{"embedding":[…2560 floats…], …}, …]}

Rerank request (returns documents sorted by relevance):

curl http://localhost:8000/v1/rerank -H "Content-Type: application/json" \
  -d '{"query":"capital of Korea",
       "documents":["Seoul is the capital of South Korea.",
                    "Paris is in France.",
                    "Beijing is in China."]}'
# → {"results":[{"index":0,"relevance_score":0.997,"document":"Seoul …"}, …]}

The reranker is a true cross-encoder: it feeds each query+document pair through the model under the official Qwen3-Reranker template (<Instruct>/<Query>/<Document>) and reads the cls.output classifier head — relevance_score = softmax([yes_logit, no_logit])[yes]. This is the same path llama.cpp's --reranking uses, and it's far more discriminative than reading raw yes/no token logits off the LM head.

Optional instruction field (Qwen3-Reranker is instruction-aware — a clear task description sharpens judgments, especially for non-English docs):

curl http://localhost:8000/v1/rerank -H "Content-Type: application/json" \
  -d '{"instruction":"Given a question, find the document that answers it",
       "query":"How do I build a voice assistant?",
       "documents":["Combine an LLM with speech-to-text and TTS.",
                    "JARVIS is a character from Iron Man."]}'

Building the 4B reranker GGUF: most community Qwen3-Reranker-4B GGUFs drop the cls.output.weight classifier head during conversion, which breaks this path (the model just emits "Okay"). Convert from the official Qwen/Qwen3-Reranker-4B safetensors with a recent convert_hf_to_gguf.py (it auto-detects the reranker and synthesizes the cls head), then quantize with --tensor-type "cls.output=f16" so the tiny head keeps full precision while the rest goes Q8_0.

Non-ASCII note: clients that serialize JSON with ensure_ascii=True (Python requests does by default) send Korean/CJK as \uXXXX escapes. The server decodes these to UTF-8 in all string-extraction paths — earlier builds only decoded the query, so non-ASCII documents scored as gibberish. Fixed; no client-side workaround needed.

The engine looks for the MTP nextn head in two places, in priority order:

1. Inline GGUF (v2 models) — if the last block (e.g. blk.64.nextn.eh_proj.weight) carries nextn tensors, they're used directly from the already-GPU-resident weights. Matches the model's training distribution exactly; no external file needed. Measured accept rate ~83 % on Qwopus3.6-27B-v2-MTP.
2. External binary (v1 fallback) — mtp_head_<hidden>.bin, auto-loaded from the hardcoded path /home/paru/mtp_work/mtp_head_<hidden>.bin (falling back to /home/paru/mtp_work/mtp_head.bin). Used only when the GGUF has no inline nextn (e.g. legacy v1 GGUFs). This path is not configurable via a CLI flag.

If neither is found, the engine runs plain greedy / continuous-batched.

All three trigger forms work; later wins on conflict:

• Literal /no_think or /think anywhere in user OR system messages (the directive text is stripped from the prompt before encoding so the model doesn't see it).
• vLLM-compatible extra_body.chat_template_kwargs.enable_thinking: false.
• Top-level chat_template_kwargs.enable_thinking: false (some clients).

Internally these all set the same force_think argument on the chat template (-1 = insert a closed <think>\n\n</think>\n\n block so the model emits no reasoning, 0 = let the model decide, 1 = prefill <think>\n).

OpenAI-style response_format ({"type":"json_object"} / {"type":"json_schema"}) is implemented via a state-machine grammar that masks tokens which would break JSON validity. It is off by default — the strict path (grammar mask + skip-think + spec/MTP disabled) drops throughput roughly 10×, so enable it explicitly with MINF_ENABLE_JSON_MODE=1. Note: json_schema is currently treated the same as json_object — only structural JSON validity is enforced, the schema itself is not.

Reasoning comes back in reasoning_content, not in content. Non-streaming replies carry a message.reasoning_content field (omitted when the model didn't think); streaming sends delta.reasoning_content chunks until the block closes, then delta.content. No <think> tag ever reaches the wire, and the two paths produce byte-identical text. A reply cut off at max_tokens mid-reasoning still returns what it had, with finish_reason: "length".

{"choices":[{"message":{"role":"assistant",
  "reasoning_content":"The user greeted me, so...",
  "content":"Hello!"},"finish_reason":"stop"}]}

This is the vLLM / SGLang / DeepSeek field name, so reasoning-aware clients work unchanged. tools/test_think_split.cpp is a standalone unit test for the split (g++ -std=c++17 -I../src -I/usr/local/cuda/include).

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen",
    "messages": [{"role":"user","content":"hello"}],
    "max_tokens": 256
  }'

Besides the OpenAI routes (/v1/chat/completions, /v1/models, and /v1/embeddings + /v1/rerank when the sidecars are proxied) there are three operator endpoints:

Vision (27B with mmproj):