LLM Tokens & Prompt Caching: The Practical Guide

The $1,400 Monday Morning Surprise

A few months ago, our team rolled out a multi-turn conversational agent to help developers debug database migrations. It worked beautifully in staging. On Friday afternoon, we expanded the beta to a few hundred active users and logged off for the weekend.

On Monday morning, our API dashboard displayed a $1,420 weekend burn on what should have been a $40 experiment.

The agent hadn't entered an infinite loop, and our users hadn't abused the system. The culprit was much simpler: on every user turn, our backend dutifully concatenated the entire conversation history, the complete database schema (roughly 85 tables with indexes and comments), and a static 1,800-word system prompt.

By turn seven of a standard chat session, a single user prompt like "Can you add an index on user_id?" was submitting 48,000 input tokens to process a 120-token response.

Turn 1:  6,200 tokens in  → 350 tokens out
Turn 2: 12,800 tokens in  → 410 tokens out
Turn 3: 19,500 tokens in  → 290 tokens out
Turn 4: 26,400 tokens in  → 380 tokens out
Turn 5: 33,600 tokens in  → 420 tokens out
Turn 6: 40,800 tokens in  → 310 tokens out
Turn 7: 48,200 tokens in  → 120 tokens out
Total input tokens billed across one 7-turn session: 187,500 tokens

If you treat Large Language Models as black-box text generators, context accumulation will quietly destroy your unit economics. To build sustainable AI applications, you have to understand how tokenizers chew on your strings, where hidden serialization overhead lives, and how modern prompt caching changes the entire cost equation.

Here is the technical breakdown of how tokens actually function, the math behind context windows, and the four architecture patterns we implemented to cut our token bill by 68%.

---

How Tokenizers Actually Slice Your Text

Large Language Models do not see letters, words, or sentences. They operate exclusively on arrays of integer IDs generated by a tokenizer. Most modern foundation models (including OpenAI's GPT-4o, Anthropic's Claude 3.5/3.7 series, and Meta's Llama 3) rely on variants of Byte-Pair Encoding (BPE).

BPE is a subword compression algorithm. It starts with a base vocabulary of individual bytes (256 values) and iteratively merges the most frequently occurring byte pairs into new tokens until it reaches a target vocabulary size (typically 100,000 to 200,000 distinct tokens).

Raw text: "unbelievable"
Step 1: ['u', 'n', 'b', 'e', 'l', 'i', 'e', 'v', 'a', 'b', 'l', 'e']
Step 2: ['un', 'b', 'e', 'l', 'i', 'e', 'v', 'a', 'b', 'l', 'e']
Step 3: ['un', 'be', 'l', 'i', 'e', 'v', 'a', 'b', 'l', 'e']
Step 4: ['un', 'believ', 'able']  → 3 Tokens [Token IDs: 834, 18429, 1204]

Because BPE relies on frequency statistics across training corpora, how your text is structured dramatically alters the token count.

1. The Leading Space Distinction

In BPE tokenizers, whitespace is baked into the subword token itself. A leading space fundamentally changes the token ID.

// Example using OpenAI's cl100k_base tokenizer:
tokenize("apple")   // -> [17180] (1 token)
tokenize(" apple")  // -> [30571] (1 token, space included)
tokenize("  apple") // -> [256, 30571] (2 tokens: ' ' + ' apple')

When you inadvertently construct prompts with double spaces or inconsistent indentation, you force the tokenizer to fall back to standalone space tokens, inflating token overhead across large payloads.

2. The JSON Indentation and Whitespace Tax

Developers frequently pass structured data to LLMs using pretty-printed JSON. While human-readable, JSON.stringify(payload, null, 2) wastes substantial token budgets on whitespace and structural punctuation.

Look at this real comparison from our database schema payload:

// Pretty-printed JSON (null, 2): 48 Tokens
{
  "tableName": "customer_subscriptions",
  "status": "active",
  "plan": "enterprise_annual",
  "seats": 50,
  "autoRenew": true
}

// Minified JSON: 27 Tokens (43.7% reduction)
{"tableName":"customer_subscriptions","status":"active","plan":"enterprise_annual","seats":50,"autoRenew":true}

// Compact YAML / Delimited Format: 19 Tokens (60.4% reduction)
customer_subscriptions|active|enterprise_annual|50|true

When sending 200 rows of database records or API responses to an LLM context, pretty-printing JSON is effectively throwing money away. You can test your minified payloads using our JSON Formatter & Minifier to preview the exact reduction in raw payload footprint.

3. Syntax and Code Tokenization

In source code, programming language syntax characters like {, }, =>, ===, and multi-space indents are often split into individual tokens depending on the vocabulary.

Consider this TypeScript interface:

// Standard 4-space indented TS: 38 tokens
export interface UserSession {
    userId: string;
    organizationId: string;
    role: "admin" | "member";
    createdAt: number;
}

// Minified TS type alias: 21 tokens (44.7% reduction)
type UserSession={userId:string;organizationId:string;role:"admin"|"member";createdAt:number};

If you feed documentation, schemas, or function definitions to an LLM, trimming unnecessary whitespace and consolidating redundant syntax delivers immediate cost and latency dividends.

---

2026 Token Economics: Pricing Comparison

Token pricing isn't uniform. Different operations carry radically different price tags. Understanding the four core pricing tiers is vital when designing AI workflows:

1. Standard Input Tokens: The base cost to ingest your prompt and context history.
2. Cached Input Tokens: Ingesting prompt segments that hit an active key-value (KV) prompt cache (typically a 75% to 90% discount).
3. Cache Write / Creation Tokens: The one-time cost to write a long prompt prefix into the model's KV cache (usually 25% higher than standard input).
4. Output (Completion) Tokens: The cost of generated tokens. Output tokens are typically 3x to 5x more expensive than input tokens because each generated token requires a sequential forward pass through the transformer.

Here is a breakdown of current production rates across top frontier and efficiency models:

You can model complex multi-turn token calculations and compare total monthly spend across these models with our interactive API Cost Calculator.

---

The Mechanics of Modern Prompt Caching

The single biggest breakthrough in context economics over the past two years is Prompt Caching (KV Cache Reuse).

In standard transformer inference, computing attention across a 50,000-token prompt requires calculating Key ($K$) and Value ($V$) matrices for every token in every attention layer. This calculation is quadratic relative to the sequence length.

With prompt caching, the model provider caches the intermediate KV states of your static prompt prefix in GPU memory. When you send a subsequent request with the exact same prefix, the model bypasses the computation for those tokens, slashing both Time to First Token (TTFT) and input token cost.

Standard Request (No Cache):
[System Prompt: 10k] + [Schema: 20k] + [User Question: 100]
═══════════════════════════════════════════════════════════► Full GPU Attention Computation (30,100 tokens billed at full price)

Cached Request:
[System Prompt: 10k] + [Schema: 20k] │ [User Question: 100]
└───────── KV Cache Hit ──────────┘ │ └─ Only Compute This Part ─►
(30,000 tokens billed at 90% off + 70% lower TTFT)

The Strict Prefix Invariant

Prompt caching relies on exact bitwise prefix matching. The cache checks tokens sequentially from index 0 upwards.

If you change a single character at token position 50, the entire cache from token 50 to token 50,000 is instantly invalidated.

❌ BAD PROMPT STRUCTURE (Cache Busting):
┌────────────────────────────────────────────────────────┐
│ Current Timestamp: 2026-08-20T14:32:01Z  <-- CHANGES!  │  <- Cache breaks at Token 8!
│ Static System Instructions (5,000 tokens)              │  <- Billed at FULL price
│ Static Database Documentation (25,000 tokens)          │  <- Billed at FULL price
│ User Query: "How do I query active users?"             │
└────────────────────────────────────────────────────────┘

✅ GOOD PROMPT STRUCTURE (Cache Optimized):
┌────────────────────────────────────────────────────────┐
│ Static System Instructions (5,000 tokens)              │  <- CACHE HIT (90% off)
│ Static Database Documentation (25,000 tokens)          │  <- CACHE HIT (90% off)
│ Ephemeral Context: Timestamp = 2026-08-20T14:32:01Z    │  <- Computed normally
│ User Query: "How do I query active users?"             │  <- Computed normally
└────────────────────────────────────────────────────────┘

Implementing Explicit Cache Breakpoints (Anthropic SDK Example)

Anthropic’s Messages API allows you to explicitly designate up to four cache breakpoints using the cache_control parameter:

import Anthropic from "@anthropic-ai/sdk";

const anthropic = new Anthropic({
  apiKey: process.env.ANTHROPIC_API_KEY,
});

async function runCachedAgent(userQuery: string, conversationHistory: Array<{ role: string; content: string }>) {
  const response = await anthropic.messages.create({
    model: "claude-3-7-sonnet-20250219",
    max_tokens: 1024,
    // 1. Static System Prompt (Explicit Cache Point)
    system: [
      {
        type: "text",
        text: getStaticSystemPrompt(), // 4,500 tokens of static instructions
        cache_control: { type: "ephemeral" },
      },
      {
        type: "text",
        text: getStaticDatabaseSchema(), // 22,000 tokens of table definitions
        cache_control: { type: "ephemeral" },
      }
    ],
    // 2. Dynamic Conversation History
    messages: [
      ...conversationHistory.map((msg, index) => {
        // Cache the conversation history up to the second-to-last turn
        if (index === conversationHistory.length - 2) {
          return {
            role: msg.role as "user" | "assistant",
            content: [
              {
                type: "text" as const,
                text: msg.content,
                cache_control: { type: "ephemeral" as const },
              }
            ]
          };
        }
        return {
          role: msg.role as "user" | "assistant",
          content: msg.content,
        };
      }),
      {
        role: "user",
        content: userQuery,
      }
    ],
  });

  // Track cache telemetry:
  console.log({
    inputTokens: response.usage.input_tokens,
    cacheReadTokens: response.usage.cache_read_input_tokens,
    cacheCreationTokens: response.usage.cache_creation_input_tokens,
    outputTokens: response.usage.output_tokens,
  });

  return response.content[0];
}

---

Four Production Patterns That Cut Our Spend by 68%

Applying theory to production required restructuring our entire agent workflow. Here are the four specific architectural interventions that produced our savings.

Pattern 1: Three-Tier Context Partitioning

We split every agent context into three isolated tiers based on volatility:

1. Tier 1 (Immutable Core — Never Changes): System instructions, persona guidelines, tool schemas, and output validation rules. Cached permanently.
2. Tier 2 (Session Knowledge — Changes per Repository/Project): Database schema, project files, and domain entities. Cached with a session key.
3. Tier 3 (Volatile State — Changes every turn): User query, transient tool execution results, error logs, and the current clock time. Never placed above Tier 1 or Tier 2.

By never allowing volatile timestamps or dynamic session IDs to sit above Tier 1 or Tier 2, our cache hit rate skyrocketed from 14% to 92.4% across all production requests.

Pattern 2: Rolling Summarization with Entity Retention

Instead of naively appending raw conversation turns until the context window overflows, we implement a rolling context compaction algorithm:

• Turns 1 to 4 are preserved in full fidelity.
• When turn 5 completes, turns 1 through 3 are compacted into a structured <conversation_state> XML block that extracts:
  • Decisions made
  • Active variables or entity IDs
  • Unresolved user constraints
• Raw message text from older turns is discarded, while the structured state block stays below 400 tokens.

<!-- Example of compacted state replacing 12,000 tokens of back-and-forth -->
<conversation_state>
  <target_table>users</target_table>
  <selected_action>add_composite_index</selected_action>
  <fields>["organization_id", "created_at"]</fields>
  <unresolved_question>Confirm index concurrency flag</unresolved_question>
</conversation_state>

Pattern 3: Schema Key Aliasing for Bulk Data

When our agents process bulk records (such as CSV rows, log streams, or analytics tables), standard JSON key duplication inflates token counts by 300%.

We introduced a lightweight bidirectional schema mapping layer:

// Raw JSON data sent to LLM: 145 tokens
const rawData = [
  { transactionId: "tx_98124", customerStatus: "verified", amountInCents: 45000, currencyCode: "USD" },
  { transactionId: "tx_98125", customerStatus: "pending",  amountInCents: 12000, currencyCode: "EUR" },
  { transactionId: "tx_98126", customerStatus: "verified", amountInCents: 8900,  currencyCode: "USD" }
];

// Aliased compact array with legend in system prompt: 62 tokens (57.2% savings)
// Legend: [t: transactionId, s: customerStatus, a: amountInCents, c: currencyCode]
const compactData = [
  ["tx_98124", "verified", 45000, "USD"],
  ["tx_98125", "pending",  12000, "EUR"],
  ["tx_98126", "verified", 8900,  "USD"]
];

The LLM understands the positional matrix natively without needing key names repeated on every single row.

Pattern 4: AST-Driven Code Snippet Pruning

When an agent analyzes code repositories, developers frequently dump entire 1,500-line source files into the prompt.

We built a local AST pre-filter that strips function bodies from unrelated methods, supplying the model with only the type signatures and the specific target function being modified:

// Instead of sending the entire 800-line UserRepository.ts:
// Send the skeleton interface + target method (74% token reduction)
export class UserRepository {
  // [Collapsed: 14 helper methods...]
  
  async findActiveSubscribers(orgId: string): Promise<User[]> {
    // Only full code for the target method under review is passed here
    return this.db.select().from(users).where(eq(users.orgId, orgId));
  }
}

---

Common Token Traps in Production

Even experienced teams stumble into these three subtle token traps:

1. The Autonomous Tool-Calling Cascade

When an autonomous agent encounters a database or API error, standard implementations append the error trace into the message log and prompt the agent to retry.

If the agent tries 5 failed variations of an invalid SQL query, each failed turn resends all previous error traces. A single failing task can burn 100,000+ tokens in 30 seconds.

Fix: Always truncate historical tool errors to the most recent attempt or enforce a hard circuit breaker after two consecutive tool errors.

2. Reasoning/Thinking Tokens Count as Output

Models with internal chain-of-thought (such as Claude 3.7 Sonnet Thinking or OpenAI o1/o3-mini) generate reasoning tokens before writing their final response.

These thinking tokens are billed at the higher output token rate, even though they are not displayed in the final user-facing text. If an agent thinks for 4,000 tokens to write a 50-token answer, you pay for 4,050 output tokens. Always configure explicit budget_tokens caps for internal thinking in non-critical reasoning workflows.

3. Multilingual Token Multipliers

English text averages roughly 0.75 words per token in standard BPE vocabularies. However, languages with non-Latin scripts (Arabic, Cyrillic, Hindi, Japanese, Urdu) often require 2 to 4 tokens per word because the tokenizer splits non-Latin words into raw byte sequences.

If you deploy multilingual AI features, budget your context allowances and rate limits on byte/token counts rather than word or character lengths.

---

Measuring Your Token Efficiency

Context optimization is not a one-time chore; it is an ongoing engineering discipline. Before deploying prompt changes to production:

1. Audit your prompts for static vs volatile ordering: Verify that timestamps, session IDs, and random seeds never precede your heavy documentation blocks.
2. Minify all structural payloads: Eliminate JSON indentation and strip unused fields before passing data to the LLM.
3. Inspect Cache Telemetry: Monitor cache_read_input_tokens in your analytics stream (using tools like PostHog or Langfuse) to ensure your cache hit rate remains above 80%.
4. Use Visual Diffs: Check how prompt refactors impact your raw token count using our Diff Checker and API Cost Calculator.

By treating tokens with the same scrutiny you apply to database queries and cloud compute instances, you can scale responsive, intelligent AI agents without fearing the Monday morning invoice.