Post #4491418

Chinese being more token efficient is a myth, and seems to stem from the superficial fact that characters are only visually more space efficient. The fact that each Chinese character takes up 3 bytes (as opposed to 1 byte of English), words in Chinese typically require compounds of several characters, and that tokenizers have a limited vocabulary limited to mostly English means that Chinese is actually token inefficient. No, Chinese Is Not More Token-Efficient Than English for LLMs | markhuang.ai - markhuang.ai/blog/chinese-token-myth