Post #1753538
2026-04-19 06:11 UTC
@astraleureka I don't know what Anthropic do but llama-cpp (open-source inference) apparently does masked decoding for tool calls. It recognises a magic token indicating the start of a tool call and from that point it forces probability to 0 for tokens that don't match an FSM for JSON syntax + tool call schema. This is done at inference level and might not be visible in the Claude leak, which afaik was just the harness.
So it's not quite as dumb as I made it sound because the LLM is constrained to only produce syntactically and schematically correct JSON during tool calls. It's still funny that it just... types the JSON though
Replies (1)
-
@wren6991@types.pl 2026-04-19 06:17
@astraleureka There's a little bit of info here: https://github.com/ggml-org/llama.cpp/blob/master/grammars/README.md There is some plumbing to make this match whatever the model is post-trained to emit for tool calls. No idea where that is. The whole file format situation is absolutely fucked in general