Tokenizer

Service for model-specific token counting and prompt truncation. Tokenization depends on the target provider, model, and encoding rules, so this module leaves the actual tokenization function to the service implementation.

The Tokenizer service can count tokens for raw prompt input and shorten a prompt to a token limit by keeping the newest messages that fit. This module defines the service tag, the service interface, and a make constructor that builds a full tokenizer service from a token-counting function.

3 exports Added in v4.0.0 Source

Constructors

make

Added in v4.0.0 Source

Creates a Tokenizer service implementation from tokenization options.

Details

This function constructs a complete Tokenizer service by providing a tokenization function. The service handles both tokenization and truncation operations using the provided tokenizer.

Signature

declare function make(options: {
  readonly tokenize: (content: Prompt) => Effect<Array<number>, AiError>;
}): Service

Example

(Creating a word tokenizer)

import { Effect } from "effect"
import { Tokenizer } from "effect/unstable/ai"

// Simple word-based tokenizer
const wordTokenizer = Tokenizer.make({
  tokenize: (prompt) =>
    Effect.succeed(
      prompt.content
        .flatMap((msg) =>
          typeof msg.content === "string"
            ? msg.content.split(" ")
            : msg.content.flatMap((part) =>
              part.type === "text" ? part.text.split(" ") : []
            )
        )
        .map((_, index) => index)
    )
})

await Effect.runPromise(wordTokenizer.tokenize("hello effect world")) // => [0, 1, 2]

Models

Service interface

Added in v4.0.0 Source

Tokenizer service interface providing text tokenization and truncation operations.

Details

This interface defines the core operations for converting text to tokens and managing content length within token limits for AI model compatibility.

Signature

interface Service {
  readonly tokenize: (input: RawInput) => Effect<Array<number>, AiError>;
  readonly truncate: (input: RawInput, tokens: number) => Effect<Prompt, AiError>;
}

Example

(Implementing a custom tokenizer)

import { Effect } from "effect"
import { Prompt } from "effect/unstable/ai"
import type { Tokenizer } from "effect/unstable/ai"

const customTokenizer: Tokenizer.Service = {
  tokenize: (input) =>
    Effect.succeed(input.toString().split(" ").map((_, i) => i)),
  truncate: (input, maxTokens) =>
    Effect.succeed(Prompt.make(input.toString().slice(0, maxTokens * 5)))
}

const tokenCount = (await Effect.runPromise(customTokenizer.tokenize("one two three"))).length // => 3
const messageCount = (await Effect.runPromise(customTokenizer.truncate("hello world", 1))).content.length // => 1

Services

Tokenizer

Added in v4.0.0 Source

Service tag for model tokenization services.

When to use

Use to access or provide model-specific token counting and prompt truncation operations.

Details

This tag provides access to tokenization functionality throughout your application, enabling token counting and prompt truncation capabilities.

Signature

declare class Tokenizer extends Shape<"effect/ai/Tokenizer", Service, this> {
  constructor(_: never);
}

Example

(Accessing the Tokenizer service)

import { Effect } from "effect"
import { Tokenizer } from "effect/unstable/ai"

const useTokenizer = Effect.gen(function*() {
  const tokenizer = yield* Tokenizer.Tokenizer
  const tokens = yield* tokenizer.tokenize("Hello, world!")
  return tokens.length
})

const tokenizer = Tokenizer.make({
  tokenize: (prompt) => Effect.succeed(prompt.content.map((_, index) => index))
})
const result = useTokenizer.pipe(Effect.provideService(Tokenizer.Tokenizer, tokenizer))
await Effect.runPromise(result) // => 1