---
title: "Understanding Tokens — AVCodex Docs"
description: "Understanding Tokens — AVCodex documentation for AV integrators, programmers, and ops teams."
lang: en
json-ld:
---

[](/)

Solutions

[Pricing](/pricing)[The Signal](/blog)[Resources](/resources)

Learn

[Free AI Assessment](/scorecard)[Get Started →](/pricing)

[Documentation Home](/docs)

Guides 

Getting Started

-   [The Alchemist Copilot](/docs/guides/the-alchemist-copilot)
-   [Choosing a Model](/docs/guides/choosing-a-model)
-   [Skills & Templates](/docs/guides/skills-and-templates)
-   [Pricing & Usage](/docs/guides/pricing-and-usage)
-   [Understanding Tokens](/docs/guides/understanding-tokens)
-   [Maximize AVCodex Capabilities](/docs/guides/maximize-avcodex-capabilities)

Knowledge & Memory

-   [How Knowledge Sources Work](/docs/guides/how-knowledge-sources-work)
-   [Knowledge Retrieval Settings](/docs/guides/knowledge-retrieval-settings)
-   [User Memory](/docs/guides/user-memory)
-   [Consumer Brain](/docs/guides/consumer-brain)

Agent Capabilities

-   [Image Recognition](/docs/guides/image-recognition)
-   [Image Generation](/docs/guides/image-generation)
-   [Video Generation](/docs/guides/video-generation)
-   [Deep Research and Deep Thinking](/docs/guides/deep-research-and-deep-thinking)
-   [Heartbeat (Proactive AI Outreach)](/docs/guides/heartbeat-proactive-ai-outreach)
-   [Database Connections](/docs/guides/database-connections)
-   [Agent-to-Agent Links](/docs/guides/agent-to-agent-links)
-   [Message Tagging](/docs/guides/message-tagging)
-   [Lead Generation Forms](/docs/guides/lead-generation-forms)
-   [Multilingual Apps](/docs/guides/multilingual-apps)
-   [Understanding Evaluations](/docs/guides/understanding-evaluations)

Design & Experience

-   [Style Studio](/docs/guides/style-studio)
-   [Component Studio](/docs/guides/component-studio)
-   [HQ Profile](/docs/guides/hq-profile)
-   [Multiplayer Chat](/docs/guides/multiplayer-chat)
-   [Circles](/docs/guides/circles)
-   [Desktop Agent](/docs/guides/desktop-agent)

Voice & Phone

-   [Phone Numbers](/docs/guides/phone-numbers)
-   [Outbound Calling](/docs/guides/outbound-calling)
-   [Voice Cloning](/docs/guides/voice-cloning)

Publish & Share

-   [Embed Chat Widget](/docs/guides/embed-chat-widget)
-   [Custom Domains](/docs/guides/custom-domains)
-   [PWA Installation](/docs/guides/pwa-installation)
-   [AVCodex Sites](/docs/guides/avcodex-sites)
-   [Embed on Kajabi](/docs/guides/embed-on-kajabi)
-   [How to Use AVCodex with Claude Code](/docs/guides/how-to-use-avcodex-with-claude-code)

Monetization & Access

-   [Selling Access](/docs/guides/selling-access)
-   [Consumer Monetization](/docs/guides/consumer-monetization)
-   [Access Control](/docs/guides/access-control)
-   [Bring Your Own Auth](/docs/guides/bring-your-own-auth)
-   [Clever SSO for Schools](/docs/guides/clever-sso-for-schools)

Analytics & Operations

-   [Analytics & Chat History](/docs/guides/analytics-and-chat-history)
-   [Performance Dashboard](/docs/guides/performance-dashboard)
-   [Programmatic Usage Stats](/docs/guides/programmatic-usage-stats)
-   [Session Lifecycle Webhooks](/docs/guides/session-lifecycle-webhooks)
-   [Audit Logs](/docs/guides/audit-logs)

Teams & White-Label

-   [Team Management](/docs/guides/team-management)
-   [Enterprise Whitelabel](/docs/guides/enterprise-whitelabel)

Alchemist Platform

-   [Alchemist Tickets](/docs/guides/alchemist-tickets)
-   [Alchemist Getting Started](/docs/guides/alchemist-getting-started)
-   [Alchemist Working with Tickets](/docs/guides/alchemist-working-with-tickets)
-   [Alchemist Local Development](/docs/guides/alchemist-local-development)

Alchemist Operations

-   [Alchemist Environment Variables](/docs/guides/alchemist-environment-variables)
-   [Alchemist Deploys and Domains](/docs/guides/alchemist-deploys-and-domains)
-   [Alchemist Self-Healing](/docs/guides/alchemist-self-healing)

Alchemist API & Automation

-   [Alchemist API Keys](/docs/guides/alchemist-api-keys)
-   [Alchemist MCP Server](/docs/guides/alchemist-mcp-server)
-   [Alchemist Pipeline Configuration](/docs/guides/alchemist-pipeline-configuration)
-   [Alchemist Pipeline Permutations](/docs/guides/alchemist-pipeline-permutations)

Developer Platform

-   [Building Custom MCP Servers](/docs/guides/building-custom-mcp-servers)
-   [Consumer OAuth for Custom MCP Servers](/docs/guides/consumer-oauth-for-custom-mcp-servers)

AVCodex MCP Server

-   [Overview](/docs/guides/overview)
-   [MCP Reference](/docs/guides/mcp-reference)
-   [Setup & Installation](/docs/guides/setup-and-installation)
-   [Authentication](/docs/guides/authentication)
-   [Tools Reference](/docs/guides/tools-reference)
-   [Common Workflows](/docs/guides/common-workflows)
-   [Rate Limits](/docs/guides/rate-limits)

Custom Actions 

Pro Actions 

API 

Builder API 

Agentic Commerce (ACP) 

Integrations 

[Docs](/docs)/ Guides / Getting Started 

# Understanding Tokens

Last updated · MAR 2026 · [Read as Markdown](/docs/guides/understanding-tokens.md)

If you have used AI tools, you have heard about "tokens." This guide explains tokens in depth, with interactive tools to help you see how your text gets tokenized.

## [Try it yourself# ](#try-it-yourself)

Use the tokenizer tool in the dashboard to see how your text gets broken into tokens.

## [What is a token# ](#what-is-a-token)

A token is the basic unit AI models use to process text. When you send a message, the model does not read characters or words. It reads tokens.

Tokens are the AI's vocabulary. Like humans break sentences into words, AI models break text into tokens. But tokens do not always match what we think of as words.

### [Tokens are not words# ](#tokens-are-not-words)

Where it gets interesting:

Text

Tokens

Count

"hello"

`hello`

1

"Hello"

`Hello`

1

"HELLO"

`HE`, `LLO`

2

"tokenization"

`token`, `ization`

2

"Crestron"

`Cres`, `tron`

2

"ChatGPT"

`Chat`, `G`, `PT`

3

Notice:

-   Common words are often 1 token.
-   Capitalization can change tokenization.
-   Long or uncommon words split into pieces.
-   Technical terms (manufacturer names, part numbers) often become multiple tokens.

### [The 4-character rule (and why it is wrong)# ](#the-4-character-rule-and-why-it-is-wrong)

You will hear "1 token is roughly 4 characters" as a quick estimate. Useful, but not accurate.

Text

Characters

Actual tokens

4-char estimate

"The quick brown fox"

19

4

5

"supercalifragilisticexpialidocious"

34

9

9

"DM-NVX-E30C"

11

6

3

"日本語"

3

3

1

`EISC.SystemReboot()`

20

8

5

The approximation works for plain English. It breaks down for:

-   **Part numbers and SKUs**: Crestron, QSC, and Extron model strings tokenize into many pieces.
-   **Non-English languages**: usually more tokens per character.
-   **Code**: varies based on syntax and naming.
-   **Numbers**: can tokenize digit-by-digit.

## [Input vs output tokens# ](#input-vs-output-tokens)

Two kinds of tokens flow through every conversation.

### [Input tokens (prompt tokens)# ](#input-tokens-prompt-tokens)

Everything you send to the AI:

-   Your message.
-   System instructions (set by the agent builder).
-   Conversation history.
-   Any retrieved knowledge or context (chunks pulled from a Crestron programming guide, for example).

### [Output tokens (completion tokens)# ](#output-tokens-completion-tokens)

Everything the AI generates:

-   The response text.
-   Any formatted content (markdown, code, etc.).

**Why this matters for pricing.**

Output tokens usually cost two to four times more than input tokens. Why?

1.  **Input**: the model reads tokens (relatively fast).
2.  **Output**: the model generates tokens one by one (compute-heavy).

That is why concise responses cost less than long, verbose ones. If your field tech agent answers in two crisp sentences instead of three paragraphs, your bill goes down.

## [How tokenization works# ](#how-tokenization-works)

AI models use a technique called **Byte Pair Encoding (BPE)** to build their vocabulary. Simplified:

1.  **Start with characters**: individual characters as the base vocabulary.
2.  **Find common pairs**: look for character pairs that appear together often.
3.  **Merge pairs**: combine the most common pairs into single tokens.
4.  **Repeat**: keep merging until you have a vocabulary of around 100,000 tokens.

The result is a vocabulary where:

-   Very common words ("the", "is", "and") are single tokens.
-   Common word parts ("-ing", "-tion", "un-") are single tokens.
-   Rare words get split into known pieces.

### [Different models, different tokenizers# ](#different-models-different-tokenizers)

Not every AI model tokenizes text the same way:

Model family

Tokenizer

Vocabulary size

GPT-4.1, GPT-5.x

o200k\_base

about 200,000

Claude 4.x

Claude tokenizer

about 100,000

Gemini 2.5/3

SentencePiece

about 256,000

The same text can produce different token counts across models, though the differences are usually small for English.

## [Practical tips for managing tokens# ](#practical-tips-for-managing-tokens)

### [1\. Be concise in your instructions# ](#1-be-concise-in-your-instructions)

Shorter system prompts mean fewer input tokens on every message.

**Verbose (45 tokens):**

> "I would like you to act as a helpful assistant that can answer questions about various topics related to AV systems. Please provide detailed and informative responses to any questions that AV technicians might ask you."

**Concise (15 tokens):**

> "You are an AV support assistant. Give detailed, informative answers."

### [2\. Consider response length# ](#2-consider-response-length)

If you do not need long responses, tell the agent:

-   "Answer in 2 to 3 sentences."
-   "Be brief."
-   "Summarize in one paragraph."

### [3\. Watch conversation length# ](#3-watch-conversation-length)

Every message in conversation history counts as input tokens. Longer conversations equal more tokens.

### [4\. Code and technical content# ](#4-code-and-technical-content)

Code often tokenizes efficiently because:

-   Common keywords (`function`, `return`, `if`) are single tokens.
-   Variable names, model strings, and inline strings add up fast. A SIMPL# class with descriptive names will tokenize heavier than a tight Lua script.

## [Token limits (context windows)# ](#token-limits-context-windows)

Every AI model has a maximum number of tokens it can process at once. This is the **context window**. Models also have a **max output** limit on how many tokens they can generate in a single response.

Model

Context window

Max output

GPT-5.4

1,000,000 tokens

32,768 tokens

GPT-5.4 Mini

1,000,000 tokens

32,768 tokens

GPT-4.1

1,000,000 tokens

32,768 tokens

Claude Opus 4.6

1,000,000 tokens

128,000 tokens

Claude Sonnet 4.6

1,000,000 tokens

64,000 tokens

Claude Haiku 4.5

200,000 tokens

64,000 tokens

Gemini 2.5 Pro

1,000,000 tokens

65,536 tokens

Gemini 2.5 Flash

1,000,000 tokens

65,536 tokens

o3

200,000 tokens

65,536 tokens

o4-mini

200,000 tokens

65,536 tokens

The context window has to fit:

-   System prompt.
-   Conversation history.
-   Your current message.
-   Space for the response.

If you exceed the limit, older messages get trimmed from the conversation.

## [Model pricing comparison# ](#model-pricing-comparison)

Prices are per 1 million tokens (including AVCodex platform markup):

Model

Input $/M

Output $/M

Best for

GPT-5.4

$3.25

$19.50

General purpose, latest capabilities.

GPT-5.4 Mini

$0.98

$5.85

Fast, cost-effective.

GPT-5.4 Nano

$0.26

$1.63

Ultra-fast, simple lookups.

GPT-4.1

$2.60

$10.40

Long context, reliable coding.

Claude Opus 4.6

$6.50

$32.50

Deep reasoning, complex tasks (RFP scoring, design).

Claude Sonnet 4.6

$3.90

$19.50

Writing, analysis, balanced.

Claude Haiku 4.5

$1.30

$6.50

Fast, affordable.

Gemini 2.5 Pro

$1.63

$6.50

Long context, multimodal (rack photos, drawings).

Gemini 2.5 Flash

$0.10

$0.39

Fast, very affordable.

Gemini 2.5 Flash Lite

$0.05

$0.20

Ultra-cheap, simple tasks.

o3

$13.00

$52.00

Advanced reasoning.

o4-mini

$3.90

$15.60

Reasoning on a budget.

## [Cost examples# ](#cost-examples)

Real token costs using GPT-5.4 ($3.25/M input, $19.50/M output):

### [Example 1: Quick room support question# ](#example-1-quick-room-support-question)

-   Tech's question: 50 tokens ($0.000163).
-   Agent response: 100 tokens ($0.00195).
-   **Total: $0.002** (less than 1 cent).

### [Example 2: Long document analysis (Biamp programming guide)# ](#example-2-long-document-analysis-biamp-programming-guide)

-   Document plus question: 5,000 tokens ($0.01625).
-   Detailed response: 1,000 tokens ($0.0195).
-   **Total: $0.04** (4 cents).

### [Example 3: Extended troubleshooting session# ](#example-3-extended-troubleshooting-session)

-   20 back-and-forth messages on a Crestron commissioning issue.
-   Average input per turn: 2,000 tokens (includes history).
-   Average output: 300 tokens.
-   **Total: about $0.25** (25 cents).

### [Budget-friendly alternative (Gemini 2.5 Flash)# ](#budget-friendly-alternative-gemini-2-5-flash)

The same extended troubleshooting session on Gemini 2.5 Flash ($0.10/M input, $0.39/M output):

-   **Total: about $0.006** (less than 1 cent).

\*AVCodex · Your AV expertise. Amplified by AI.\*

Was this helpful? 

[Edit this page →](#)

[

Previous

Pricing & Usage

](/docs/guides/pricing-and-usage)[

Next

Maximize AVCodex Capabilities

](/docs/guides/maximize-avcodex-capabilities)

On this page

-   [Try it yourself](#try-it-yourself)
-   [What is a token](#what-is-a-token)
-   [Tokens are not words](#tokens-are-not-words)
-   [The 4-character rule (and why it is wrong)](#the-4-character-rule-and-why-it-is-wrong)
-   [Input vs output tokens](#input-vs-output-tokens)
-   [Input tokens (prompt tokens)](#input-tokens-prompt-tokens)
-   [Output tokens (completion tokens)](#output-tokens-completion-tokens)
-   [How tokenization works](#how-tokenization-works)
-   [Different models, different tokenizers](#different-models-different-tokenizers)
-   [Practical tips for managing tokens](#practical-tips-for-managing-tokens)
-   [1\. Be concise in your instructions](#1-be-concise-in-your-instructions)
-   [2\. Consider response length](#2-consider-response-length)
-   [3\. Watch conversation length](#3-watch-conversation-length)
-   [4\. Code and technical content](#4-code-and-technical-content)
-   [Token limits (context windows)](#token-limits-context-windows)
-   [Model pricing comparison](#model-pricing-comparison)
-   [Cost examples](#cost-examples)
-   [Example 1: Quick room support question](#example-1-quick-room-support-question)
-   [Example 2: Long document analysis (Biamp programming guide)](#example-2-long-document-analysis-biamp-programming-guide)
-   [Example 3: Extended troubleshooting session](#example-3-extended-troubleshooting-session)
-   [Budget-friendly alternative (Gemini 2.5 Flash)](#budget-friendly-alternative-gemini-2-5-flash)

[](/)

The AI platform built exclusively for professional AV. Build, deploy, and sell AI tools that understand your industry.

### Platform

-   What You Can Build
-   Templates
-   [Pricing](/pricing)

### Services

-   [Done-For-You](/pricing)
-   [Academy](/academy)
-   [Contact](/contact)

### Company

-   About
-   [The Signal](/blog)
-   [Docs](/docs)
-   [LinkedIn](#)

© 2026 AVCodex. A Future Ready Holdings Inc. product. SOC 2 Type II Certified · HIPAA Compliant