Use token counting when a language model’s context limit or usage is measured in tokens. Use character counting when a form, message, or other requirement sets a character limit. The measures answer different questions, and there is no dependable universal conversion between them.
What tokens and characters measure
A character count measures text according to a particular counting convention. A token count measures how a specific model’s tokenizer divides input into units. A token may represent one character, part of a word, a whole word, punctuation, or another common sequence; it is not the same thing as a character or a word. OpenAI describes tokens as chunks used to process text in its Key concepts documentation.
For ordinary English prose, OpenAI gives rough estimates of about four characters per token and about 0.75 words per token. These are ballparks, not conversion formulas: actual counts vary with the text, language, encoding, and model. Use them only for rough planning, and leave room rather than treating an estimate as an exact fit. See OpenAI’s token-counting explanation and concepts documentation.
Choose the count that matches your limit
| What you need to decide | Use | Why |
|---|---|---|
| Whether text fits a form or specification with a character limit | Character count, using the target system’s definition | The requirement is in characters; a token estimate cannot guarantee compliance. |
| Whether text fits a model’s context window | Token count for the target model | The model processes input in tokens, and character-to-token ratios vary. |
| How many tokens an API request will use | The provider’s counter for the intended model and request format, where available | Request structure and non-text inputs can affect the count beyond a plain-text tokenizer. |
| Comparing lengths across languages or formats | Report both measures, defining each; include the model tokenizer if model use matters | Neither measure is a universal substitute for the other. |
How to count tokens for a model request
Plain text
For a plain-text estimate, use the tokenizer corresponding to the model. OpenAI’s Help Center points to tiktoken for programmatic plain-text tokenization and advises selecting the encoding for the target model. A tokenizer for one model or provider should not be assumed exact for another.
OpenAI Responses API input
For a complete OpenAI Responses input, use OpenAI’s input-token counting endpoint with the intended request format. The documented counter accounts for request formatting such as message roles and boundaries, and supports inputs including messages, images, files, tools, and conversations. A local text tokenizer does not capture all of those factors; model-specific behavior can also affect tokenization.
Anthropic Messages input
For Anthropic Messages, Anthropic documents POST /v1/messages/count_tokens. Its count uses the tokenizer of the specified model and can include messages, system prompts, tools, images, and PDFs. Anthropic also documents limits for some server tools and URL or file sources, so check the current count-tokens documentation for the input types you plan to send.
When character counting gets complicated
For an application limit, use that application’s counter and definition if available. “Character” is not a single universal technical counting convention for software. If you implement your own counter, specify whether it counts bytes, Unicode code points, UTF-16 code units, or user-perceived grapheme clusters. These can differ for Unicode text, so one visible symbol does not necessarily equal one unit under every implementation. The cited provider documentation explains token counting but does not define the character convention for arbitrary third-party fields.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Why a plain-text count may not match API usage
A string tokenizer counts text, not necessarily the full request that reaches a model. Roles, message boundaries, tools, schemas, files, and images can affect request sizing. Some reported usage can also include formatting or model-generated tokens that are not visible in the text. Character-based estimates are especially unsuitable for images and files. For context planning or usage validation, count the actual request with the provider’s tool for the intended model and format where available; do not treat visible text length as a guarantee of reported API usage.
Free tools Windows power users keep installed
One-click scans. No signup required.
Quick Recap
Best Value
Rank #3
What to check before sending
- For a character-limited field: follow the field’s own counting convention and counter; do not use a token estimate to establish compliance.
- For a context limit: count tokens with the target model’s tokenizer or provider counter, and leave room for other request content and output as appropriate.
- For API usage: use the provider’s counter with the intended model and structured input when available; verify that its documented scope covers your tools, files, images, or other inputs.
- For cost: check the current pricing for the specific model and usage type. Token counting tells you quantity, not the price, and pricing is not established here.
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
