Bonsai 2 27B: what 1.76 bits per weight buys you over a Q4 GGUF

A 27B model in 5.9 GB

PrismML shipped Ternary Bonsai 2 27B on 17 September 2026. It is a compression of Qwen3.8 27B, a 27.8-billion-parameter multimodal model, down to ternary weights: every weight is −1, 0 or +1, with FP16 group-wise scaling on top. That works out to 1.76 effective bits per weight and a 5.9 GB file, and the 262K-token context window survives the trip. The licence is Apache 2.0.

The reason it matters is not the ternary trick itself. It is that a 27B-class model now fits on an 8 GB GPU or a 16 GB laptop with room left for a long context, and PrismML claims it does so while keeping 98.2% of the full-precision benchmark score. If that number holds up, it changes what you can put inside an on-device or air-gapped product. The rest of this piece is about whether it holds up, what it actually costs, and how to try it yourself in an afternoon.

Running it locally

The short path is the Bonsai-demo repo. It bundles PrismML's fork of llama.cpp, MLX support for Apple Silicon, prebuilt binaries, and a ./setup.sh that downloads the weights and builds everything. It targets macOS on Metal, Linux and Windows on CUDA, Vulkan or ROCm, and plain CPU, and it is also Apache 2.0.

git clone https://github.com/PrismML-Eng/Bonsai-demo
cd Bonsai-demo
./setup.sh

Once it finishes you get a local chat server on port 8080, with vision and tool-calling wired in, and optional Open WebUI integration if you want a front end.

If you would rather drive the binary directly, the GGUF repo on Hugging Face ships three files: PTQ1_0 (1-bit) at 5.95 GB, PQ2_0 (2-bit) at 7.21 GB, and an F16 reference at 53.8 GB. The model card's sample invocation is:

./bin/llama-cli -m Ternary-Bonsai-2-27B-PQ2_0.gguf -ngl 99 -fa on -c 32768

Three flags to note. -ngl 99 offloads every layer to the GPU. -fa on turns on flash attention. -c 32768 sets a 32K context, which is a sensible default: the model supports 262K, but the community reports later in this piece suggest you should not assume quality is flat all the way out.

The one line on the model card you must not skip: "Stock llama.cpp will not run these files." The ternary kernels exist only in PrismML's fork. That is the first cost, and it is not a small one, so I will come back to it.

There is also a WebGPU browser demo on Hugging Face Spaces if you want to see the kernels run without installing anything.

The numbers next to a Q4 quant

Here is the comparison that actually matters for someone choosing a local model. The obvious alternative to Bonsai 2 is not full-precision Qwen3.8 27B, which almost nobody runs on a laptop. It is a good 4-bit GGUF of the same model.

PrismML's own model card gives you that comparison in thinking mode. Bonsai 2 27B averages 84.78 in 5.9 GB. The Unsloth UD-Q4_K_XL quant averages 85.18 in 17.6 GB, which is 98.7% of FP16. An IQ2_XXS quant, the usual way to get a 27B under 10 GB, averages 72.59 in 9.4 GB, or 84.1%.

So the trade is a 3x memory saving for roughly half a point on PrismML's own suite. The more telling row is IQ2_XXS: it is the existing sub-2-bit option, it is larger than Bonsai 2, and it loses more than twelve points. That is the gap ternary training closes. Post-hoc quantisation to two bits wrecks a model; a model trained to be ternary does not fall apart.

The broader suite backs that up with some texture. Across PrismML's 20 benchmarks covering reasoning, math, coding, instruction following, vision and agentic tool use, the ternary model scores 83.9 aggregate against 85.4 for full-precision Qwen3.8 27B, the 98.2% figure in the headline. Per category, the biggest drops are Vision (78.59 vs 81.64) and Knowledge & Reasoning (83.95 vs 86.66). Instruction Following is actually slightly higher (82.66 vs 81.25). Note that the aggregate hides a three-point vision drop; if your product is vision-heavy, that is the number to look at, not 98.2%.

For context on how fast this moved: the first Ternary Bonsai 27B, released in March, retained 95% of the aggregate. Six months later it is over 98%. The model was trained on Google v5 TPUs.

Speed and power

PrismML reports up to 143 tokens per second on an RTX 5090, 46.8 tokens per second on an Apple M5 Max through MLX, and 0.714 mWh per token on an RTX 4090, which it says is 40% more energy-efficient than a full-precision 8B model. That last one is the number to remember for battery-powered deployments: a 27B ternary model cheaper per token than an 8B model at full precision.

The Hacker News thread, at 373 points and 115 comments when I read it, is where the vendor numbers meet real hardware. Users report roughly 20 tokens per second on an M5 Pro MacBook out of the box, rising to about 44 after tuning. An M1 Pro gets about 14. An RTX 6000 Pro Blackwell gets about 120 tokens per second while using 7.9 GB of VRAM. A Pixel 9 Pro crashed on the WebGPU demo. A PrismML engineer posting as francisjp joined the thread to explain a Metal fix in the fork, which is a good sign for responsiveness and a reminder that the fork is still being patched in public.

The 7.9 GB VRAM figure is the practical one. That is the PQ2_0 file plus a working context on a card with 8 GB, which describes a very large number of laptops and desktops that could not previously run anything in the 27B class.

What it costs

The 98.2% figure is real on the suite it was measured on. The costs live outside that suite.

The fork. You cannot load these files in stock llama.cpp, Ollama, LM Studio or anything else built on upstream. You build PrismML's fork, you track PrismML's fork, and if upstream llama.cpp ships something you need, you wait for the fork to merge it. For a personal setup that is an afternoon. For a shipped product it is a dependency on a single vendor's branch of a fast-moving C++ codebase, and you should price that in. The HN criticism was blunt about the friction, and I think it is fair.

Long context. The advertised 262K window is inherited from Qwen3.8 27B. Several HN commenters report that quality degrades at long context in ways the benchmark retention number does not capture. The suite PrismML publishes does not break out results by context length, so there is no vendor number to weigh against those reports. If your use case is long documents, run your own eval before you trust the window.

Agentic work. The same thread reports weaker agentic performance than hosted models. That is partly an unfair comparison, a 5.9 GB local model against a frontier API, but it is the comparison your users will make. The suite includes agentic tool use and shows retention, so the drop from full-precision Qwen is small; the drop from what people are used to is not.

Vision. The weakest category in PrismML's own table, down three points. Fine for a chat model that can read a screenshot; check carefully before it becomes the backbone of anything that depends on image understanding.

The company itself is worth a paragraph because it tells you how long the fork is likely to be maintained. TechCrunch reports that PrismML raised a $22.25 million seed round from Khosla Ventures, Cerberus Capital and Caltech, that the first Bonsai has been downloaded over 11 million times with another 2.6 million for the smaller variants, and that the founder and CEO is Caltech professor Babak Hassibi, with UC Berkeley's Ion Stoica advising. Hassibi told TechCrunch that "the next models that we will release, hopefully in the next couple of months, will be in the several-hundred-billion-parameter range." Stoica's framing is the ambitious version: "You are going to have intelligence at your fingertips, and it's going to be free because it's going to run on the device you already bought."

What to do about it

If you build anything that runs a model on the user's machine, or in an environment with no outbound network, Bonsai 2 27B is worth a real evaluation this week, not a bookmark. Concretely:

1. Clone Bonsai-demo, run ./setup.sh, and start with the PQ2_0 file at -c 32768. Confirm the tokens-per-second on your actual target hardware, not on PrismML's RTX 5090.
2. Put your own eval set through it and through the Unsloth UD-Q4_K_XL quant of Qwen3.8 27B side by side. PrismML's half-point gap is on their suite; your prompts are the ones that matter.
3. If you use long context, test at the lengths you will actually ship. The 262K number is inherited, and the community reports say do not assume it is free.
4. If it wins, decide now how you will track the llama.cpp fork. Pin a commit, mirror the repo, and write down who owns the upgrade. The kernels are the moat and the maintenance burden at the same time.
5. Watch for upstream. The moment stock llama.cpp or MLX runs these files natively, most of the cost column disappears and the 3x memory saving is just there for the taking.

The first Bonsai was a proof that ternary training could get close. This one gets close enough that the question flips: not whether you can afford the quality loss, but whether you can afford the fork. For a lot of on-device products, the answer to both is now yes.