Tool Use

Language models are great at generating text, but often we need them to take real-world actions, such as querying an API, accessing a database, or calling a service. Most LLM providers support this through tool use (also known as function calling), where you expose specific operations in your application that the model can invoke.

Based on the input it receives, a model may choose to invoke (or call) one or more tools to augment its response. Your application then runs the corresponding logic for the tool using the parameters provided by the model. You then return the result to the model, allowing it to include the output in its final response.

The Toolkit simplifies tool integration by offering a structured, type-safe approach to defining tools. It takes care of all the wiring between the model and your application - all you have to do is define the tool and implement its behavior.

Defining a Tool

Let’s walk through a complete example of how to define, implement, and use a tool that fetches a dad joke from the icanhazdadjoke.com API.

1. Define the Tool

We start by defining a tool that the language model will have access to using the Tool.make constructor.

This constructor accepts several parameters that allow us to fully describe the tool to the language model:

• description: Provides an optional description of the tool
• success: The type of value the tool will return if it succeeds
• failure: The type of value the tool will return if it fails
• parameters: The parameters that the tool should be called with

Example (Defining a Tool)

import { Tool } from "@effect/ai"
import { Schema } from "effect"

const GetDadJoke = Tool.make("GetDadJoke", {
  description: "Get a hilarious dad joke from the ICanHazDadJoke API",
  success: Schema.String,
  failure: Schema.Never,
  parameters: {
    searchTerm: Schema.String.annotations({
      description: "The search term to use to find dad jokes"
    })
  }
})

Based on the above, a request to call the GetDadJoke tool:

• Takes a single searchTerm parameter
• Will return a string if it succeeds (i.e. the joke)
• Does not have any expected failure scenarios

2. Create a Toolkit

Once we have a tool request defined, we can create a Toolkit, which is a collection of tools that the model will have access to.

Example (Creating a Toolkit)

import { Tool, Toolkit } from "@effect/ai"
import { Schema } from "effect"

const GetDadJoke = Tool.make("GetDadJoke", {
  description: "Get a hilarious dad joke from the ICanHazDadJoke API",
  success: Schema.String,
  failure: Schema.Never,
  parameters: {
    searchTerm: Schema.String.annotations({
      description: "The search term to use to find dad jokes"
    })
  }
})

const DadJokeTools = Toolkit.make(GetDadJoke)

3. Implement the Logic

The .toLayer(...) method on a Toolkit allows you to define the handlers for each tool in the toolkit. Because .toLayer(...) takes an Effect, we can access services from our application to implement the tool call handlers.

Example (Implementing a Toolkit)

import { Tool, Toolkit } from "@effect/ai"
import {
  HttpClient,
  HttpClientRequest,
  HttpClientResponse
} from "@effect/platform"
import { NodeHttpClient } from "@effect/platform-node"
import { Array, Effect, Schema } from "effect"

class DadJoke extends Schema.Class<DadJoke>("DadJoke")({
  id: Schema.String,
  joke: Schema.String
}) {}

class SearchResponse extends Schema.Class<SearchResponse>("SearchResponse")({
  results: Schema.Array(DadJoke)
}) {}

class ICanHazDadJoke extends Effect.Service<ICanHazDadJoke>()("ICanHazDadJoke", {
  dependencies: [NodeHttpClient.layerUndici],
  effect: Effect.gen(function*() {
    const httpClient = yield* HttpClient.HttpClient
    const httpClientOk = httpClient.pipe(
      HttpClient.filterStatusOk,
      HttpClient.mapRequest(HttpClientRequest.prependUrl("https://icanhazdadjoke.com"))
    )

    const search = Effect.fn("ICanHazDadJoke.search")(
      function*(searchTerm: string) {
        return yield* httpClientOk.get("/search", {
          acceptJson: true,
          urlParams: { searchTerm }
        }).pipe(
          Effect.flatMap(HttpClientResponse.schemaBodyJson(SearchResponse)),
          Effect.flatMap(({ results }) => Array.head(results)),
          Effect.map((joke) => joke.joke),
          Effect.orDie
        )
      }
    )

    return {
      search
    } as const
  })
}) {}

const GetDadJoke = Tool.make("GetDadJoke", {
  description: "Get a hilarious dad joke from the ICanHazDadJoke API",
  success: Schema.String,
  failure: Schema.Never,
  parameters: {
    searchTerm: Schema.String.annotations({
      description: "The search term to use to find dad jokes"
    })
  }
})

const DadJokeTools = Toolkit.make(GetDadJoke)

const DadJokeToolHandlers = DadJokeTools.toLayer(
  Effect.gen(function*() {
    // Access the `ICanHazDadJoke` service
    const icanhazdadjoke = yield* ICanHazDadJoke
    return {
      // Implement the handler for the `GetDadJoke` tool call request
      GetDadJoke: ({ searchTerm }) => icanhazdadjoke.search(searchTerm)
    }
  })
)

In the code above:

• We access the ICanHazDadJoke service from our application
• Register a handler for the GetDadJoke tool using .handle("GetDadJoke", ...)
• Use the .search method on our ICanHazDadJoke service to search for a dad joke based on the tool call parameters

The result of calling .toLayer on a Toolkit is a Layer that contains the handlers for all the tools in our toolkit.

Because of this, it is quite simple to test a Toolkit by using .toLayer to create a separate Layer specifically for testing.

4. Give the Tools to the Model

Once the tools are defined and implemented, you can pass them along to the model at request time. Behind the scenes, the model is given a structured description of each tool and can choose to call one or more of them when responding to input.

Example (Using a Toolkit)

import { LanguageModel, Tool, Toolkit } from "@effect/ai"
import { Effect, Schema } from "effect"

const GetDadJoke = Tool.make("GetDadJoke", {
  description: "Get a hilarious dad joke from the ICanHazDadJoke API",
  success: Schema.String,
  failure: Schema.Never,
  parameters: {
    searchTerm: Schema.String.annotations({
      description: "The search term to use to find dad jokes"
    })
  }
})

const DadJokeTools = Toolkit.make(GetDadJoke)

const generateDadJoke = LanguageModel.generateText({
  prompt: "Generate a dad joke about pirates",
  toolkit: DadJokeTools
})

5. Bring It All Together

To make the program executable, we must provide the implementation of our tool call handlers:

Example (Providing the Tool Call Handlers to a Program)

import { LanguageModel, Tool, Toolkit } from "@effect/ai"
import { OpenAiClient, OpenAiLanguageModel } from "@effect/ai-openai"
import {
  HttpClient,
  HttpClientRequest,
  HttpClientResponse
} from "@effect/platform"
import { NodeHttpClient } from "@effect/platform-node"
import { Array, Config, Console, Effect, Layer, Schema } from "effect"

class DadJoke extends Schema.Class<DadJoke>("DadJoke")({
  id: Schema.String,
  joke: Schema.String
}) {}

class SearchResponse extends Schema.Class<SearchResponse>("SearchResponse")({
  results: Schema.Array(DadJoke)
}) {}

class ICanHazDadJoke extends Effect.Service<ICanHazDadJoke>()("ICanHazDadJoke", {
  dependencies: [NodeHttpClient.layerUndici],
  effect: Effect.gen(function*() {
    const httpClient = yield* HttpClient.HttpClient
    const httpClientOk = httpClient.pipe(
      HttpClient.filterStatusOk,
      HttpClient.mapRequest(HttpClientRequest.prependUrl("https://icanhazdadjoke.com"))
    )

    const search = Effect.fn("ICanHazDadJoke.search")(
      function*(searchTerm: string) {
        return yield* httpClientOk.get("/search", {
          acceptJson: true,
          urlParams: { searchTerm }
        }).pipe(
          Effect.flatMap(HttpClientResponse.schemaBodyJson(SearchResponse)),
          Effect.flatMap(({ results }) => Array.head(results)),
          Effect.map((joke) => joke.joke),
          Effect.scoped,
          Effect.orDie
        )
      }
    )

    return {
      search
    } as const
  })
}) {}

const GetDadJoke = Tool.make("GetDadJoke", {
  description: "Get a hilarious dad joke from the ICanHazDadJoke API",
  success: Schema.String,
  failure: Schema.Never,
  parameters: {
    searchTerm: Schema.String.annotations({
      description: "The search term to use to find dad jokes"
    })
  }
})

const DadJokeTools = Toolkit.make(GetDadJoke)

const DadJokeToolHandlers = DadJokeTools.toLayer(
  Effect.gen(function*() {
    const icanhazdadjoke = yield* ICanHazDadJoke
    return {
      GetDadJoke: ({ searchTerm }) => icanhazdadjoke.search(searchTerm)
    }
  })
).pipe(Layer.provide(ICanHazDadJoke.Default))

const program = LanguageModel.generateText({
  prompt: "Generate a dad joke about pirates",
  toolkit: DadJokeTools
}).pipe(
  Effect.flatMap((response) => Console.log(response.text)),
  Effect.provide(OpenAiLanguageModel.model("gpt-4o"))
)

const OpenAi = OpenAiClient.layerConfig({
  apiKey: Config.redacted("OPENAI_API_KEY")
}).pipe(Layer.provide(NodeHttpClient.layerUndici))

program.pipe(
  Effect.provide([OpenAi, DadJokeToolHandlers]),
  Effect.runPromise
)

Benefits

Type Safe

Every tool is fully described using Effect’s Schema, including inputs, outputs, and descriptions.

Effect Native

Tool call behavior is defined using Effect, so they can leverage all the power of Effect. This is especially useful when you need to access other services to support the implementation of your tool call handlers.

Injectable

Because implementing the handlers for an Toolkit results in a Layer, providing alternate implementation of tool call handlers in different environments is as simple as providing a different Layer to your program.

Separation of Concerns

The definition of a tool call request is cleanly separated from both the implementation of the tool behavior, as well as the business logic that calls the model.