Running open "System One" typed-decision models on a Snapdragon X Elite laptop (Hexagon NPU, Adreno GPU, Oryon CPU), and comparing them with TypeSafe's Jev API.

Typed-decision models answer structured questions about a piece of text with probabilities: pick one option, rate on a scale, or yes/no. The ones covered here are:

• Laya: runs on the NPU and the Adreno GPU.
• Decider: runs fast on the CPU through llama.cpp.
• imajev: runs on the CPU through PyTorch.
• Winnow-12B (Gemma 4 12B): runs entirely on the Hexagon NPU at 4-bit, about 2 s per request, through ONNX Runtime's QNN provider with no test-signing. It also runs on the CPU through its llama.cpp server. The build, the traps and the numbers are in docs/WINNOW-NPU.md; the NPU-ready model files are on Hugging Face: tielmane/Winnow-12B-NPU-LPBQ-X-Elite.
• decider-12b (Mapika, Gemma 4 12B): built for the Hexagon NPU with the same pipeline, about 2.2 s per request; JevBench standard 0.986, level with Jev. NPU files: tielmane/decider-12b-NPU-LPBQ-X-Elite; code in decider-npu/.
• Laya, fine-tuned into a dedicated classifier with Claude Opus labels: on a real helpdesk-ticket job it went from 63.5% to 88.5% (Jev 83.5%) and runs on the NPU at ~17 ms per ticket. Recipe and code: docs/LAYA-FINETUNE.md.

This repository is a dated snapshot, not a maintained fork. Each model lives in its upstream repository, and the fixes that belong there are proposed there (docs/UPSTREAM.md). What this repo keeps is the glue: setup scripts, Jev-compatible servers, proof scripts, and a benchmark harness. Versions, model revisions and hashes are pinned, so the scripts stay reproducible even as upstream moves on.

Median time for one request with 3–4 short questions (about 100–130 tokens):

Winnow-12B (requests of about 200–550 tokens, one or two questions):

JevBench public items (231; accuracy easy / standard / hard): Jev 1.000 / 0.986 / 0.712 · decider-2b 1.000 / 0.889 / 0.568 · imajev-2b 1.000 / 0.917 / 0.568 · Laya 0.958 / 0.694 / 0.351.

• The 12B models on the NPU: only 50 of the 111 hard items fit in 576 tokens, so they are scored on those 50.
  • decider-12b: 1.000 / 0.986 / 0.700.
  • Winnow-12B: 1.000 / 0.972 / 0.680.
  • The others on the same 50: Jev 0.760, decider-2b 0.680, CPU Winnow Q8_0 0.800.
• Details and caveats are in FINDINGS.

Public tasks with human labels (~3,900 items):

decider-12b picks the right answer as often as Winnow. Its yes/no confidences, though, are almost always near 0 or 1: Mapika's config sets a yes/no temperature of 0.05. Ranking metrics (AUROC) therefore come out lower. Re-scored at a temperature of 1.5, its yes/no AUROC matches Winnow's, with the same accuracy, and its average becomes 0.765. FINDINGS section 7 has the details.

A private helpdesk-ticket job (200 held-out real tickets; two questions: bucket out of 12, work type out of 6; only aggregates are published, details in docs/LAYA-FINETUNE.md):

The reference labels for those 200 tickets are a blind two-labeller Claude consensus, so the fine-tuned Laya (trained on Claude Opus labels) may share some of their reading of the tickets.

Accuracy, the benchmark method, and every Windows ARM64 problem we hit are in docs/FINDINGS.md.

laya/bin/      WSL launchers: winrun (run a Windows venv script), serve (start/stop Jev-compatible servers), run_suite
laya/win/      Windows-side Python: NPU/GPU servers, proofs, encoder export and QNN graph fixes, throttling opt-out
decider/       Windows ARM64 setup, CPU-only llama.cpp build, smoke tests, ONNX export + QNN diagnosis (bisect, chain)
imajev/        Windows ARM64 setup, torchvision-from-source build, smoke test
decider-npu/   decider-12b on the Hexagon NPU: Decider-format prompts/head on top of the winnow-npu pipeline, server, checks
laya-finetune/ Laya -> dedicated classifier: Opus labelling workflow, fine-tuning, NPU export and runner
winnow-npu/    Winnow-12B on the Hexagon NPU: GGUF -> ONNX chunks, calibration, LPBQ int4, packing, QNN compile, Jev-compatible
               server; tools/ (checks and probes); cpu/ (Winnow's llama.cpp server on the CPU: build, patch, launcher)
harness/       jevcmp.py (runner for any /v1/systemone endpoint), metrics and reports, benchmark item builders
docs/          FINDINGS.md, UPSTREAM.md, WINNOW-NPU.md and LAYA-FINETUNE.md (guides), fair-benchmark plan

• Windows side: each model uses its own working folder under %USERPROFILE% (local-laya, local-decider, local-imajev), each containing envs\, models\, results\ and scripts\.
• WSL side, before using laya/bin/*:

  export WIN_HOME=/mnt/c/Users/<you>
  export WINUSER=<you>

• uv: expected on the Windows PATH.
• Jev key: the harness reads TYPESAFE_API_KEY from the environment only. Never commit it.

Laya on the NPU

1. Run laya/win/setup_env.ps1 -Name npu.
2. Build the NPU contexts:

   laya/bin/winrun npu -m laya_snapdragon --models <models\npu> build --seq 128 256 512

3. Prove it runs on the NPU: laya/bin/winrun npu npu_proof.py --seq 128.
4. Serve it: laya/bin/serve start npu.

Decider (fast CPU path)

1. Run decider/setup_cpu.ps1.
2. Run decider/build_llamacpp.cmd. It needs the Visual Studio Build Tools "C++ Clang tools for Windows" component, and builds with every GPU backend off.
3. Download the GGUF at the pinned revision.
4. Run decider/smoke_gguf.py.

imajev

1. Run imajev/setup_cpu.ps1.
2. Run imajev/build_torchvision.cmd.
3. Download the pinned base model and adapter.
4. Run imajev/smoke.py.

Comparisons

1. Build items with harness/fairbench/* (public human-labelled data) or use harness/items/*.jsonl.
2. Run laya/bin/run_suite <items> <run>.
3. Produce the report with harness/report.py or harness/fair_report.py.

• Never run llama.cpp's Adreno OpenCL GPU backend on this setup. It coincided with two blue screens (bugcheck 0xD1) during testing. The build here compiles every GPU backend out. See FINDINGS.
• Servers bind to 127.0.0.1 and have no authentication. Don't expose them.
• Check free RAM before loading a model. The smoke tests refuse to start below a set threshold.

harness/items/suite-v1.jsonl is a synthetic suite written by Claude models; nothing in it is real customer data. For publication, its invented email addresses, URLs, IP addresses and phone numbers were rewritten to reserved example forms (*.example domains, the 203.0.113.0/24 documentation range, 5-filled numbers) after the benchmark runs. A few items therefore differ slightly from the text the reported numbers were measured on. The public-benchmark items are not included; harness/fairbench/ rebuilds them from their original sources.

Apache-2.0 for the code in this repository. Models and datasets keep their own licences; this repo contains none of their weights or data, only scripts that fetch them at pinned revisions.