Arkaya Labs
RESEARCH · NEWEST FIRST

The record,
kept honestly.

Everything we build is measured against the engines people already use, on machines people own, and published with the losses left in.

What we makeThe ledger
WHAT WE MAKE

Three pieces, one process.

An engine, an agent and a tuner, built together, measured together, and run on one card.

Inference engine

vivace

The engine. Open models on the GPU you have, serving many conversations at once from one card, with a finished conversation parked so the next turn starts where it left off. Open source under Apache 2.0.

Measured results
Hardware
NVIDIA RTX 30, 40 and 50 series (CUDA). Apple M5 (Metal).
Models
GGUF. Llama 3, Qwen 2.5 and 3, Qwen 3.5 hybrid, Mistral. Q4_K_M and Q8_0.
Serves
/v1/chat/completions and /v1/messages on localhost. Works with Claude Code, Qwen Code and OpenCode.
Faults
A GPU fault exits with code 3; vivace supervise restarts it and chat --resume picks the transcript back up.
Correctness
Top-1 agreement with llama.cpp on the same file at or above 99 %, with a bounded KL, in four modes.
Personal agent

Iris

The agent. It lives on your machine and knows the difference between reading and spending: it quotes its sources by file and line, keeps its memory in plain text you can open, and asks before anything leaves the computer.

Tool-choice accuracy
Permissions
Four classes: Read, WriteLocal, WriteExternal, Spend. Spending always asks; after untrusted input, every write asks.
Citations
From your documents, file and line. From the web, only quotes the saved page still shows.
Memory
Plain text files you can open, edit and delete, every write logged with its reason.
MCP
Runs as a server for other tools, and never offers them spending or writes outside this computer.
Skills
ask_docs, web_answer, read_papers, research.
Diagnosis and tuning

strided and gait

The tuner. A diagnosis whose every rule carries a citation, and a trial loop that changes one thing at a time, writes its prediction down first, and keeps the change only when the measurement proves it.

Inside one process
Rules
Each carries a citation: Roofline, PagedAttention, Sarathi-Serve, DistServe, Little's law. A rule without one will not load.
Confidence
Every finding has a number. Ambiguous evidence gives no diagnosis at all.
Trial
INTAKE → PREFLIGHT → BASELINE → APPLY → VERIFY → REVIEW → PACKET, with the prediction written before the change.
Keep rule
Kept only if the objective improves with the 95 % interval clear of zero; otherwise reverted through its declared inverse.
Authority
Performance knobs may change on their own. Anything that could change an answer needs a human.
THE ARCHITECTURE

Inside one process.

The engine, the diagnosis and the tuner share one process on the card. Choose an adjustment to see which layers it reaches.

GPURTX · Apple silicon KernelsCUDA · Metal · HIP KV cachePaged · parked SchedulerBatching · prefill WorkloadRequests · sessions

One process. One control loop.

vivace serves the model. strided reads every engine step from an in-process ring and diagnoses what limits the workload; gait tests one setting at a time. No sidecar, no IPC, no HTTP.

Changes land between steps, without a restart.

RESEARCH NOTES

What we measured.

How it was done, and what it cost. The newest notes first.

Latest

23 Oct 2026 · Engine

vivace against vLLM on one card

A data-centre engine on a gaming GPU: where its AWQ path wins on throughput, where GGUF loading does not, and why the comparison is not like for like.

Behind on throughput, the gap closing as concurrency rises; ahead on first token.

17 Oct 2026 · Engine

Many chats, one card

One consumer GPU serving sixteen conversations, measured against llama.cpp and Ollama on the same files, with the parity check first.

With eight or more chats the shared step wins; alone, llama.cpp still does.

Earlier

16 Oct 2026 · Agent

Choosing the tool, or answering

Three routers for the same decision: the model alone, a logprob gate, a hidden-state probe. Wrong-tool rate over 200 prompts on two models.

The router is only as good as the model under it: ten times fewer wrong tools with Qwen3-8B.

10 Oct 2026 · Engine

One machine's numbers: Metal on an M5

First token and tokens per second on a base M5 with 24 GB, published as one machine's numbers, not as targets.

First token 1.3 to 1.4× sooner; decode speed equal with one conversation.

Oct 2026 · Tuner

What gait kept, and what it gave back

A tuner that keeps nothing by default: the first trial, a guardrail that reverts, and the open bug where a kept value was not.

One keep on the CPU reference, none on a GPU yet. The 5070 Ti session is next.

EVERY PUBLISHED NUMBER

The ledger.

Against llama.cpp, Ollama and vLLM on the same model files, on machines people actually own. Losses stay in, and a number is never edited once it goes up.

vivace, the engine

Difference from the engine named on each row. Right of centre is better for vivace.

vivace aheadvivace behindrange across concurrency
Time to first token, 52-token promptvs llama.cpp · Apple M5, 24 GB · 9 Oct 2026
114 vs 160 ms
Aggregate tokens/s, 16 conversationsvs llama.cpp · Apple M5, 24 GB · 9 Oct 2026
+13.8 %
Aggregate tokens/s, 4 conversationsvs llama.cpp · Apple M5, 24 GB · 9 Oct 2026
−3.8 %
Aggregate tokens/s, 8 and 16 chats, Llama 3.1 8Bvs llama.cpp · RTX 5070 Ti · 17 Oct 2026
+12.7 to +15.1 %
Time per output token, p99, 8 and 16 chatsRTX 5070 Ti · 17 Oct 2026
42–50 % lower
Time to first token, p99, 8 and 16 chatsRTX 5070 Ti · 17 Oct 2026
16–46 % higher
Single chat, time to first tokenvs Ollama 0.40.1 · RTX 5070 Ti · 17 Oct 2026
2.0–2.5× slower
Aggregate tokens/s, 8 and 16 chatsvs Ollama 0.40.1 · RTX 5070 Ti · 17 Oct 2026
+20 to +49 %
Aggregate tokens/s, 4, 8 and 16 chatsvs vLLM, AWQ weights · RTX 5070 Ti · 23 Oct 2026
−5.9 to −16.8 %
Time to first token, p50vs vLLM, AWQ weights · RTX 5070 Ti · 23 Oct 2026
up to 31 % lower

Iris, the agent

Wrong tool chosen over 200 prompts. Shorter is better.

Qwen3-8BApple M5 · 16 Oct 2026
2.5 %
Llama 3.1 8BApple M5 · 16 Oct 2026
24 %

gait, the tuner

A change is kept only when the measurement proves it.

prefill_chunk_tokens 512 → 256Oct 2026
KeptCPU referenceReverted every timeRTX 3050Not run yetRTX 5070 Ti
Show the numbers as a table
DateMeasuredAgainstResultMachine
9 Oct 2026Time to first token, 52-token promptllama.cpp114 vs 160 msApple M5, 24 GB
9 Oct 2026Aggregate tokens/s, 16 conversationsllama.cpp+13.8 %Apple M5, 24 GB
9 Oct 2026Aggregate tokens/s, 4 conversationsllama.cpp−3.8 %Apple M5, 24 GB
17 Oct 2026Aggregate tokens/s, 8 and 16 chats, Llama 3.1 8Bllama.cpp+12.7 to +15.1 %RTX 5070 Ti
17 Oct 2026Time per output token, p99, 8 and 16 chats—42–50 % lowerRTX 5070 Ti
17 Oct 2026Time to first token, p99, 8 and 16 chats—16–46 % higherRTX 5070 Ti
17 Oct 2026Single chat, time to first tokenOllama 0.40.12.0–2.5× slowerRTX 5070 Ti
17 Oct 2026Aggregate tokens/s, 8 and 16 chatsOllama 0.40.1+20 to +49 %RTX 5070 Ti
23 Oct 2026Aggregate tokens/s, 4, 8 and 16 chatsvLLM, AWQ weights−5.9 to −16.8 %RTX 5070 Ti
23 Oct 2026Time to first token, p50vLLM, AWQ weightsup to 31 % lowerRTX 5070 Ti
16 Oct 2026Wrong tool chosen, 200 prompts, Qwen3-8B—2.5 %Apple M5
16 Oct 2026Wrong tool chosen, 200 prompts, Llama 3.1 8B—24 %Apple M5
Oct 2026prefill_chunk_tokens 512 → 256—Kept: CPU reference; Reverted every time: RTX 3050; Not run yet: RTX 5070 TiCPU reference, RTX 3050

The slow first token under load is the thing we are fixing first. Hover or focus a row for its full figures.

A painting of the sanctuary of Apollo at Delphi beneath the cliffs of Parnassus, olive groves running down to the Gulf of Corinth, in late daylight