Ubicloud Achieves 100% Tool-Call Accuracy Using Constrained Decoding

Ubicloud reported that serving the Kimi K2.6 model with vLLM version 0.28 resulted in only 92% tool call schema accuracy. The company subsequently enabled constrained decoding in a fork of vLLM to address this issue. This change allowed Ubicloud to achieve 100% tool-call schema accuracy for both Kimi and DeepSeek models. The improvement was measured using the K2VV ToolCall benchmark, which consists of 2000 requests selected to evaluate tool call generation and syntactic correctness.

vLLM version 0.31 introduced support for constrained decoding as a server-side flag to guarantee schema adherence. The system implements this feature using structural tags from the XGrammar library. Constrained decoding prevents models from sampling tokens that would result in a syntactically invalid tool call. XGrammar guarantees correct tool calls with negligible performance overhead by precomputing token masks for each phase. Schema accuracy quantifies the fraction of tool-calls that adhere to the tool's schema.

Moonshot AI published the K2VV ToolCall benchmark to evaluate tool call trigger similarity and schema accuracy. vLLM maintainers reported problems with several models after constrained decoding was enabled, leading to it being disabled by default. To allow users to enable tool-call strictness for their deployments, vLLM added the --tool-strict-level flag.