Kimi K3.1 Pro: bake-off before you trust benchmarks

What shipped, and why it is loud

Moonshot AI released Kimi K3.1 Pro on Friday 23 October, and its announcement calls it "our most capable 3T-class open-weight model to date". It has a 1-million-token context window. The API model IDs are kimi-k3.1-pro and kimi-k3.1-pro-highspeed, the second running 2-3x faster.

Moonshot also says "Full model weights will be released by Nov 4, 2026". The announcement page does not state the license. As of Sunday, no K3.1 weights were on Hugging Face, according to Notebookcheck. So today this is an API model with an open-weight promise attached.

The Hacker News thread reached 1,057 points and 650 comments, per an AI digest. Two arguments are running at once. One is how close a Chinese lab now sits to the frontier. The other is whether a lab can trim a model's capabilities before release when the weights cannot be recalled afterwards.

I think both are best answered the same way: with your own measurements, not the launch post.

The numbers are Moonshot's own

On Moonshot-reported Terminal-Bench 4.0, K3.1 Pro scores 65.7%, Claude Opus 5.5 scores 66.4% and GPT-6 Astra scores 58.2%. That is a near tie with the best closed model on a coding-agent benchmark.

Treat it as a claim. All of those figures are vendor-run and unverified. The one independent reference in the brief is the Artificial Analysis index, which places K3.1 Pro at 55 against 58 for Opus 5.5. That is a real gap, and it points the same direction as Moonshot's own admission that it still trails closed frontier models, which it says are public for at least three months.

The hallucination claim needs the same care. Moonshot reports that claim-level hallucination on its own hard test, run with tools, fell from 23.4% for K3 to 8.3%. Unite.ai's coverage notes that Moonshot says the test is built to probe error-prone cases, not to reflect typical error rates. Do not read 8.3% as "this model is wrong 8% of the time". Read it as "it improved a lot on a test designed to catch it out".

The price is the real news

API pricing is $2 input and $10 output per million tokens, against $3 and $15 for K3, according to OrcaRouter. That is a third off. Highspeed is $4 and $20. Cache hits cost $0.20, and there is no long-context surcharge across the 1,048,576-token window.

That last point matters if you feed whole repositories into context. Many providers charge more past a threshold, which quietly changes the economics of agent loops. Here the per-token rate stays flat.

There is a catch on the cost side, and it comes from the docs as summarised by Unite.ai:

Thinking is mandatory. Disabling it returns HTTP 400.
reasoning_effort accepts low, high or max, and max is the default.
A mistyped model ID in a third-party tool falls back to K2.8 Preview without thinking.

A default of maximum-effort thinking means output tokens, the expensive side at $10, can balloon on easy tasks. A cheaper rate card does not guarantee a cheaper bill. The fallback behaviour is worse in a different way: a typo in your config silently swaps in an older model, and your results get quietly worse without any error.

How to run the bake-off

A model that is close to Opus 5.5 on someone else's numbers at roughly two thirds of K3's price deserves a test on your own code. Keep it boring and controlled.

Pick 20 to 40 real tasks from your repo history. Closed issues with a known good fix work well. Include a few that touch large files, since the context window is the selling point.
Use the same harness for every model. Same agent loop, same tools, same prompts, same timeouts. Change only the model ID.
Fix reasoning_effort and record it. Run K3.1 Pro at max (the default) and at high, and note it in the results. If you compare one model at default effort against another at a lower setting, you are measuring the setting.
Log tokens, not only pass rate. Capture input, output and cached tokens per task, then compute cost per solved task. That is the figure that decides whether a third off the rate card survives mandatory thinking.
Assert the model ID at startup. Have your harness print the resolved model name from the response and fail if it is not kimi-k3.1-pro. This guards against the silent K2.8 Preview fallback.
Run each task more than once. Agent runs vary. Report pass rate over repeats, not a single lucky pass.

Run the Highspeed variant on a subset if latency matters to you, for example in an interactive coding assistant. At $4/$20 it is double the base price, so only pay for it where the wait is the problem.

The cyber trade-off, in numbers

The unusual part of this release is that Moonshot published the cost of its own safety choice. It says it found cyber-related data in mid-training and in its reinforcement-learning environments, and removed it from both stages while keeping general coding tasks, per The Next Web.

On Moonshot's in-house xdev-bench-hard, using pass@3, a checkpoint without the cuts solved 90.5%. The release model solved 81.0%. K3 solved 33.3%. That is a 9.5-point drop from trimming, and still far above the previous generation.

Two readings are fair. The optimistic one: the cut cost a modest slice of capability and left the coding skills most users need. The sceptical one: this is one in-house benchmark, scored by the vendor, and a model that scores 81.0% on a hard cyber-flavoured test is still a capable one.

Then there is the part nobody can settle yet. Released weights cannot be withdrawn once downloaded. If the trimming is shallow, someone will fine-tune the capability back in, and you cannot patch that after the fact. If it holds, it is a meaningful precedent for open releases. Weights are due by 4 November, and that is when outside researchers can test it instead of quoting the vendor.

What to do this week

Do not switch production on launch-day numbers. Treat 65.7% on Terminal-Bench 4.0 as a reason to test, not a result.
Run the bake-off now against the API, with the model-ID assertion and token logging above. You will have your own cost-per-solved-task figure before the weights land.
Check the license on 4 November. It is not stated on the announcement page, and it decides whether you can self-host or fine-tune.
If you handle security work, test it on that work. Do not assume the cut is irrelevant to you, or that it is airtight. Measure the tasks you actually run.
Set reasoning_effort explicitly in every client rather than relying on the max default, and cap output tokens per request.

The lasting lesson of this launch is not the leaderboard position. It is that a vendor handed you a rare, concrete trade-off figure, and then asked you to take the rest on trust. Verify the rest.