Sub-Net-Public/llm-benchmark
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
Sub-Net model benchmarks — reproducible shootouts between self-hosted models (LLMs, VLMs, classifiers).
25 commits 1 branch 0 tags 6.8 MiB
  • Python 90.1%
  • TeX 5.8%
  • Shell 3.8%
  • Makefile 0.3%
Find a file
Exact
Florian Berthold e533a7368d laya: cost model and question-shape guidance in the report 
Adds the measured cost model (one sequence per question, so latency tracks
state_tokens x questions: 37 ms for 1q/150-token state up to 548 ms for 5q/900
tokens), the design consequence (one choice with N options costs one row, N noul
questions cost N), the idle-clock ramp (25-50 ms on the first call after a quiet
minute) and the truncation behaviour as a wiring rule: keep the state inside the
checkpoint context (512 english / 1024 typed-decisions) because laya cuts the tail
and the API now surfaces it in warnings.
2026-09-19 21:31:24 +02:00
argus-vlm make: build report PDFs on A4 (Sub-Net standard) — rebuild all reports 2026-09-16 20:30:05 +02:00
flash-next make: build report PDFs on A4 (Sub-Net standard) — rebuild all reports 2026-09-16 20:30:05 +02:00
laya laya: cost model and question-shape guidance in the report 2026-09-19 21:31:24 +02:00
ornith make: build report PDFs on A4 (Sub-Net standard) — rebuild all reports 2026-09-16 20:30:05 +02:00
plant-id make: build report PDFs on A4 (Sub-Net standard) — rebuild all reports 2026-09-16 20:30:05 +02:00
reasoning-effort make: build report PDFs on A4 (Sub-Net standard) — rebuild all reports 2026-09-16 20:30:05 +02:00
sglang-dflash2 make: build report PDFs on A4 (Sub-Net standard) — rebuild all reports 2026-09-16 20:30:05 +02:00
surya-ocr make: build report PDFs on A4 (Sub-Net standard) — rebuild all reports 2026-09-16 20:30:05 +02:00
templates LaTeX report pipeline: shared Sub-Net brand template + Makefile 2026-08-20 15:01:28 +02:00
typesafe-jev laya: correct the framing section — a shadowed import ran the Jev harness 2026-09-19 21:04:57 +02:00
vllm make: build report PDFs on A4 (Sub-Net standard) — rebuild all reports 2026-09-16 20:30:05 +02:00
.gitignore typesafe-jev: reuse the HTTP connection; correct the published latency by ~313 ms 2026-09-16 21:59:37 +02:00
LICENSE Initial commit 2026-08-20 12:43:57 +00:00
Makefile make: build report PDFs on A4 (Sub-Net standard) — rebuild all reports 2026-09-16 20:30:05 +02:00
README.md laya: fp16 is a trap on Pascal - 2.8x cost, fixed; latency 212 -> 100 ms p50 2026-09-19 21:16:20 +02:00

README.md

llm-benchmark

Sub-Net model benchmarks — reproducible shootouts between self-hosted models (LLMs, VLMs, classifiers). Each subdirectory is one self-contained benchmark: harness, raw results, and a report.

Benchmark Question Result
plant-id/ Dedicated plant classifier vs. general VLMs (retirement decision) classifier wins clearly — 34 % vs 26 % top-1 (n = 250)
vllm/ vLLM serving tuning on Blackwell (MTP vs DFlash vs ngram spec decode, concurrency, scale-up) MTP k=3 (35B) / k=2 (27B) confirmed; harnesses included
argus-vlm/ moondream3.1 vs Qwen3.8-27B as Argus tier-1 VLM on real detection frames moondream stays (caption 92 % P / 100 % R, 5x faster); its detect needs a cross-check (41 % P)
surya-ocr/ Dedicated Surya OCR service vs. Qwen3.8-27B vision on real DMS documents revised 2026-08-22: Surya keeps a German-accuracy edge (umlauts/numerics), Qwen is 10x faster + scan-robust — retirement is a tradeoff, not parity
reasoning-effort/ Qwen3.8 chat template: keep default reasoning_effort=xhigh or override to medium? keep xhigh — medium gains zero quality, costs +57 % tokens / +85 % latency
ornith/ Ornith-1.5-35B-A3B (release day) vs Lightning 30B + Qwen3.8-27B for the agent-model role fastest ever measured (284 tok/s, no spec decode), quality tie — pilot before any swap
sglang-dflash2/ SuffixDecoding vs trained drafters; SGLang + DFlash2 vs vLLM for the 27B slot suffix rejected (wins only warm exact-repetition); SGLang now prod 27B server (+23 % single-stream, ~1.2k tok/s 8-way)
flash-next/ Qwen3.8-Flash-Next (125B-A6B) on one RTX PRO 6000: llama.cpp GGUF vs SM120 SGLang fork (NVFP4 + NEXTN) SGLang fork wins (2.2x decode, real concurrency, 524K ctx) — now the gpu02 daily driver, 27B retired
typesafe-jev/ TypeSafe Jev (system-1 decision API: typed Choice/Score/Noul) vs hardcoded NPC heuristics in the NFA game server temperament + life-routine viable as an event-driven overlay (p50 553 ms, 0/46 flips, calibrated confidence); defender pursuit stays heuristic — question framing collapses it to a blanket "hold". Not adopted: hosted API, not self-hostable
laya/ Same 46-scenario Jev battery re-run against the self-hosted Laya decision API (421M encoder, jev.at/api) hosting solved — 100 ms p50 (the package's fp16 default costs 2.8x on Pascal; fp32 fixes it) against Jev's 251 ms from the same host the same night, 0 output tokens, bit-exact determinism — but 25/46 vs Jev's 31-32/46 on the hand-labelled verdicts; confidence is an entropy measure, not a probability

All benchmarks run against self-hosted models only; test data and credentials are pulled from the environment, never committed.

Adding a benchmark

  1. Create a subdirectory with the harness, raw results and a report.md.
  2. Build the branded PDF from the repo root: make <name>/report.pdf (pandoc + pdflatex with templates/subnet-report-header.tex).

Debian build deps: pandoc texlive-latex-base texlive-latex-recommended texlive-fonts-recommended lmodern cm-super-minimal.