Post #4491420
2026-08-03 11:08 UTC
random Chinese character
It’s beneficial for reasoning to have models trained in a few languages. Chinese is a good one because one character is one word is one token.
Replies (1)
-
@Meron35@lemmy.world 2026-08-04 03:59
Chinese being more token efficient is a myth, and seems to stem from the superficial fact that characters are only visually more space efficient. The fact that each Chinese character takes up 3 bytes (as opposed to 1 byte of English), words in Chinese typically require compounds of several characters, and that tokenizers have a limited vocabulary limited to mostly English means that Chinese is actually token inefficient. No, Chinese Is Not More Token-Efficient Than English for LLMs | markhuang.ai - markhuang.ai/blog/chinese-token-myth