Tokenizer unstable

Service for model-specific token counting and prompt truncation. Tokenization depends on the target provider, model, and encoding rules, so this module leaves the actual tokenization function to the service implementation.

The Tokenizer service can count tokens for raw prompt input and shorten a prompt to a token limit by keeping the newest messages that fit. This module defines the service tag, the service interface, and a make constructor that builds a full tokenizer service from a token-counting function.

3 exports Since v4.0.0 Source

Constructors

make

unstable

Since v4.0.0

Creates a Tokenizer service implementation from tokenization options.

Details

This function constructs a complete Tokenizer service by providing a tokenization function. The service handles both tokenization and truncation operations using the provided tokenizer.

Signature

declare function make(options: {
  readonly tokenize: (content: Prompt) => Effect<Array<number>, AiError>;
}): Service

Example Creating a word tokenizer

import { Effect } from "effect"
import { Tokenizer } from "effect/ai"

// Simple word-based tokenizer
const wordTokenizer = Tokenizer.make({
  tokenize: (prompt) =>
    Effect.succeed(
      prompt.content
        .flatMap((msg) =>
          typeof msg.content === "string"
            ? msg.content.split(" ")
            : msg.content.flatMap((part) =>
              part.type === "text" ? part.text.split(" ") : []
            )
        )
        .map((_, index) => index)
    )
})

await Effect.runPromise(wordTokenizer.tokenize("hello effect world")) // => [0, 1, 2]

Models

Service

interface unstable

Since v4.0.0

Tokenizer service interface providing text tokenization and truncation operations.

Details

This interface defines the core operations for converting text to tokens and managing content length within token limits for AI model compatibility.

Signature

interface Service {
  readonly tokenize: (input: RawInput) => Effect<Array<number>, AiError>;
  readonly truncate: (input: RawInput, tokens: number) => Effect<Prompt, AiError>;
}

Example Implementing a custom tokenizer

import { Effect } from "effect"
import { Prompt } from "effect/ai"
import type { Tokenizer } from "effect/ai"

const customTokenizer: Tokenizer.Service = {
  tokenize: (input) =>
    Effect.succeed(input.toString().split(" ").map((_, i) => i)),
  truncate: (input, maxTokens) =>
    Effect.succeed(Prompt.make(input.toString().slice(0, maxTokens * 5)))
}

const tokenCount = (await Effect.runPromise(customTokenizer.tokenize("one two three"))).length // => 3
const messageCount = (await Effect.runPromise(customTokenizer.truncate("hello world", 1))).content.length // => 1

Services

Tokenizer

unstable

Since v4.0.0

Service tag for model tokenization services.

When to use

Use to access or provide model-specific token counting and prompt truncation operations.

Details

This tag provides access to tokenization functionality throughout your application, enabling token counting and prompt truncation capabilities.

Signature

declare class Tokenizer extends Shape<"effect/ai/Tokenizer", Service, this> {
  constructor(_: never);
}

Example Accessing the Tokenizer service

import { Effect } from "effect"
import { Tokenizer } from "effect/ai"

const useTokenizer = Effect.gen(function*() {
  const tokenizer = yield* Tokenizer.Tokenizer
  const tokens = yield* tokenizer.tokenize("Hello, world!")
  return tokens.length
})

const tokenizer = Tokenizer.make({
  tokenize: (prompt) => Effect.succeed(prompt.content.map((_, index) => index))
})
const result = useTokenizer.pipe(Effect.provideService(Tokenizer.Tokenizer, tokenizer))
await Effect.runPromise(result) // => 1