It costs a lot more to use AI models in some languages
The Coalition for Language Equity in AI Research found that languages with less digitized data require disproportionately more tokens to process in large language models. This "tokenizer tax" increases costs for users, slows response times, and reduces context window capacity. The disparity affects major languages with hundreds of millions of speakers, exacerbating digital and economic inequality in AI access.
Key points
- Underrepresented languages use 1.5–17x more tokens than English in LLMs
- Higher token costs hit speakers of Bengali, Punjabi, Urdu, and Burmese
- Inefficiency stems from less digitized training data for non-English languages
- Issue raises concerns about digital equity and AI service affordability
Summary and key points written by AI from the headline and a short excerpt, so they can be incomplete or wrong. Computerworld did the reporting; read the original for the full story.
Read the full story at Computerworld
