Batching & Caching

Before we dig into the Effect's solution to batching and caching let's start with a description of the problem.

It is very common in apps to depend on a number of external data sources like:

HTTP APIs
Databases
Filesystems

Model Definition

Let's start with a fairly minimal model description:

ts
import * as Effect from "@effect/io/Effect"
 
interface User {
  readonly _tag: "User"
  readonly id: number
  readonly name: string
  readonly email: string
}
 
class GetUserError {
  readonly _tag = "GetUserError"
}
 
interface Todo {
  readonly _tag: "Todo"
  readonly id: number
  readonly message: string
  readonly ownerId: number
}
 
class GetTodosError {
  readonly _tag = "GetTodosError"
}
 
class SendEmailError {
  readonly _tag = "SendEmailError"
}

ts
import * as Effect from "@effect/io/Effect"
 
interface User {
  readonly _tag: "User"
  readonly id: number
  readonly name: string
  readonly email: string
}
 
class GetUserError {
  readonly _tag = "GetUserError"
}
 
interface Todo {
  readonly _tag: "Todo"
  readonly id: number
  readonly message: string
  readonly ownerId: number
}
 
class GetTodosError {
  readonly _tag = "GetTodosError"
}
 
class SendEmailError {
  readonly _tag = "SendEmailError"
}

In a real world scenario we may want to use a more precise types instead of directly using primitives for identifiers (see @effect/data/Brand). We may also want to include more information in the errors.

Classic Approach

Given such a model we usually write up functions to call some API (or database, etc.) like the following:

ts
const getTodos = Effect.tryCatchPromise(
  () => fetch("https://api.example.demo/todos").then(_ => _.json() as Promise<Todo[]>),
  () => new GetTodosError()
)
 
const getUserById = (id: number) => Effect.tryCatchPromise(
  () => fetch(`https://api.example.demo/getUserById?id=${id}`).then(_ => _.json() as Promise<User>),
  () => new GetUserError()
)
 
const sendEmail = (address: string, text: string) => Effect.tryCatchPromise(
  () => fetch("https://api.example.demo/sendEmail", {
    method: "POST",
    headers: {
      "Content-Type": "application/json",
    },
    body: JSON.stringify({ address, text }),
  }).then(_ => _.json() as Promise<void>),
  () => new SendEmailError()
)
 
const sendEmailToUser = (id: number, message: string) => Effect.flatMap(
  getUserById(id),
  (user) => sendEmail(user.email, message)
)
 
const notifyOwner = (todo: Todo) => Effect.flatMap(
  getUserById(todo.ownerId),
  (user) => sendEmailToUser(user.id, `hey ${user.name} you got a todo!`)
)

ts
const getTodos = Effect.tryCatchPromise(
  () => fetch("https://api.example.demo/todos").then(_ => _.json() as Promise<Todo[]>),
  () => new GetTodosError()
)
 
const getUserById = (id: number) => Effect.tryCatchPromise(
  () => fetch(`https://api.example.demo/getUserById?id=${id}`).then(_ => _.json() as Promise<User>),
  () => new GetUserError()
)
 
const sendEmail = (address: string, text: string) => Effect.tryCatchPromise(
  () => fetch("https://api.example.demo/sendEmail", {
    method: "POST",
    headers: {
      "Content-Type": "application/json",
    },
    body: JSON.stringify({ address, text }),
  }).then(_ => _.json() as Promise<void>),
  () => new SendEmailError()
)
 
const sendEmailToUser = (id: number, message: string) => Effect.flatMap(
  getUserById(id),
  (user) => sendEmail(user.email, message)
)
 
const notifyOwner = (todo: Todo) => Effect.flatMap(
  getUserById(todo.ownerId),
  (user) => sendEmailToUser(user.id, `hey ${user.name} you got a todo!`)
)

In a real world scenario we may not want to trust our APIs to actually return the expected data - for doing this properly you can use @effect/schema/Schema or similar alternatives such as zod.

When using the utilities we defined it is normal to end up with code that looks like the following:

ts
const program = Effect.flatMap(getTodos, Effect.forEachParDiscard(notifyOwner))

ts
const program = Effect.flatMap(getTodos, Effect.forEachParDiscard(notifyOwner))

Here we used the Effect.forEachParDiscard to repeat an Effect for every Todo and the Effect repeated first fetches the User who owns the todo and then sends an email.

We like writing code this way because it is very expressive and very easy to read, but is it efficient?

This code will execute tons of individual API calls. Many todos will likely have the same owner and our APIs may also provide batched alternatives where you can request as many users as you would like to in one call.

So what can we do? Rewrite all our code to use a different form of API? Should we really do that?

Well not anymore.

Declaring Requests

Let's rewrite our example to be as efficient as possible - we'll start by writing a model for the requests that our data sources support:

ts
import * as Request from "@effect/io/Request"
 
interface GetTodos extends Request.Request<GetTodosError, Todo[]> {
  readonly _tag: "GetTodos"
}
const GetTodos = Request.tagged<GetTodos>("GetTodos")
 
interface GetUserById extends Request.Request<GetUserError, User> {
  readonly _tag: "GetUserById"
  readonly id: number
}
const GetUserById = Request.tagged<GetUserById>("GetUserById")
 
interface SendEmail extends Request.Request<SendEmailError, void> {
  readonly _tag: "SendEmail"
  readonly address: string
  readonly text: string
}
const SendEmail = Request.tagged<SendEmail>("SendEmail")
 
type ApiRequest = GetTodos | GetUserById | SendEmail

ts
import * as Request from "@effect/io/Request"
 
interface GetTodos extends Request.Request<GetTodosError, Todo[]> {
  readonly _tag: "GetTodos"
}
const GetTodos = Request.tagged<GetTodos>("GetTodos")
 
interface GetUserById extends Request.Request<GetUserError, User> {
  readonly _tag: "GetUserById"
  readonly id: number
}
const GetUserById = Request.tagged<GetUserById>("GetUserById")
 
interface SendEmail extends Request.Request<SendEmailError, void> {
  readonly _tag: "SendEmail"
  readonly address: string
  readonly text: string
}
const SendEmail = Request.tagged<SendEmail>("SendEmail")
 
type ApiRequest = GetTodos | GetUserById | SendEmail

We are using @effect/data/Data behind the scenes and given that requests will be compared using @effect/data/Equal for caching it is important to make sure they are compared by value. If you nest objects / arrays in your model, you should use Data.struct / Data.case / Data.tuple / etc.

Declaring Resolvers

Now that we have our requests defined it is time to tell Effect how to resolve those requests. That's where we would use a RequestResolver.

Here we will define a single resolver per query. There is no right or wrong answer in how granular your resolvers should be but usually you will split up your resolvers based on which API calls can be batched.

ts
import * as RequestResolver from "@effect/io/RequestResolver"
 
// we assume we cannot batch GetTodos, we create a normal resolver
const GetTodosResolver = RequestResolver.fromFunctionEffect((request: GetTodos) =>
  Effect.tryCatchPromise(
    () => fetch("https://api.example.demo/todos").then((_) => _.json()) as Promise<Todo[]>,
    () => new GetTodosError()
  )
)
 
// we assume we can batch GetUserById, we create a batched resolver
const GetUserByIdResolver = RequestResolver.makeBatched((requests: GetUserById[]) =>
  pipe(
    Effect.tryCatchPromise(
      () => fetch("https://api.example.demo/getUserByIdBatch", {
        method: "POST",
        headers: {
          "Content-Type": "application/json",
        },
        body: JSON.stringify({ users: requests.map(({ id }) => ({ id })) }),
      }).then((_) => _.json()) as Promise<User[]>,
      () => new GetUserError()
    ),
    Effect.flatMap((users) => Effect.forEachWithIndex(requests, (request, index) =>
      Request.completeEffect(request, Effect.succeed(users[index]))
    )),
    Effect.catchAll((error) => Effect.forEach(requests, (request) =>
      Request.completeEffect(request, Effect.fail(error))
    ))
  )
)
 
// we assume we can batch SendEmail, we create a batched resolver
const SendEmailResolver = RequestResolver.makeBatched((requests: SendEmail[]) =>
  pipe(
    Effect.tryCatchPromise(
      () => fetch("https://api.example.demo/sendEmailBatch", {
        method: "POST",
        headers: {
          "Content-Type": "application/json",
        },
        body: JSON.stringify({ emails: requests.map(({ address, text }) => ({ address, text })) }),
      }).then((_) => _.json()),
      () => new SendEmailError()
    ),
    Effect.flatMap(() => Effect.forEach(requests, (request) =>
      Request.completeEffect(request, Effect.unit())
    )),
    Effect.catchAll((error) => Effect.forEach(requests, (request) =>
      Request.completeEffect(request, Effect.fail(error))
    ))
  )
)

ts
import * as RequestResolver from "@effect/io/RequestResolver"
 
// we assume we cannot batch GetTodos, we create a normal resolver
const GetTodosResolver = RequestResolver.fromFunctionEffect((request: GetTodos) =>
  Effect.tryCatchPromise(
    () => fetch("https://api.example.demo/todos").then((_) => _.json()) as Promise<Todo[]>,
    () => new GetTodosError()
  )
)
 
// we assume we can batch GetUserById, we create a batched resolver
const GetUserByIdResolver = RequestResolver.makeBatched((requests: GetUserById[]) =>
  pipe(
    Effect.tryCatchPromise(
      () => fetch("https://api.example.demo/getUserByIdBatch", {
        method: "POST",
        headers: {
          "Content-Type": "application/json",
        },
        body: JSON.stringify({ users: requests.map(({ id }) => ({ id })) }),
      }).then((_) => _.json()) as Promise<User[]>,
      () => new GetUserError()
    ),
    Effect.flatMap((users) => Effect.forEachWithIndex(requests, (request, index) =>
      Request.completeEffect(request, Effect.succeed(users[index]))
    )),
    Effect.catchAll((error) => Effect.forEach(requests, (request) =>
      Request.completeEffect(request, Effect.fail(error))
    ))
  )
)
 
// we assume we can batch SendEmail, we create a batched resolver
const SendEmailResolver = RequestResolver.makeBatched((requests: SendEmail[]) =>
  pipe(
    Effect.tryCatchPromise(
      () => fetch("https://api.example.demo/sendEmailBatch", {
        method: "POST",
        headers: {
          "Content-Type": "application/json",
        },
        body: JSON.stringify({ emails: requests.map(({ address, text }) => ({ address, text })) }),
      }).then((_) => _.json()),
      () => new SendEmailError()
    ),
    Effect.flatMap(() => Effect.forEach(requests, (request) =>
      Request.completeEffect(request, Effect.unit())
    )),
    Effect.catchAll((error) => Effect.forEach(requests, (request) =>
      Request.completeEffect(request, Effect.fail(error))
    ))
  )
)

Resolvers can also access context like any other Effect and there are many different ways of creating resolvers. You may want to check the reference documentation of the @effect/io/RequestResolver module next.

Defining Queries

At this point we are ready to plug the pieces together! Let's do just that:

ts
const getTodos = Effect.request(
         
const getTodos: Effect.Effect<never, GetTodosError, Todo[]>
  GetTodos({}),
  GetTodosResolver
)
 
const getUserById = (id: number) => Effect.request(
  GetUserById({ id }),
  GetUserByIdResolver
)
 
const sendEmail = (address: string, text: string) => Effect.request(
  SendEmail({ address, text }),
  SendEmailResolver
)
 
const sendEmailToUser = (id: number, message: string) => Effect.flatMap(
  getUserById(id),
  (user) => sendEmail(user.email, message)
)
 
const notifyOwner = (todo: Todo) => Effect.flatMap(
  getUserById(todo.ownerId),
  (user) => sendEmailToUser(user.id, `hey ${user.name} you got a todo!`)
)

ts
const getTodos = Effect.request(
         
const getTodos: Effect.Effect<never, GetTodosError, Todo[]>
  GetTodos({}),
  GetTodosResolver
)
 
const getUserById = (id: number) => Effect.request(
  GetUserById({ id }),
  GetUserByIdResolver
)
 
const sendEmail = (address: string, text: string) => Effect.request(
  SendEmail({ address, text }),
  SendEmailResolver
)
 
const sendEmailToUser = (id: number, message: string) => Effect.flatMap(
  getUserById(id),
  (user) => sendEmail(user.email, message)
)
 
const notifyOwner = (todo: Todo) => Effect.flatMap(
  getUserById(todo.ownerId),
  (user) => sendEmailToUser(user.id, `hey ${user.name} you got a todo!`)
)

It looks like we are back at the beginning, same exact types and same exact composition.

But now the following program:

ts
const program = Effect.flatMap(getTodos, Effect.forEachParDiscard(notifyOwner))

ts
const program = Effect.flatMap(getTodos, Effect.forEachParDiscard(notifyOwner))

Will only require 3 queries to be executed to our APIs instead of 1 + 2n where n is the number of todos.

Resolvers with Context

There may be cases where you want to access some context as part of the request resolver, in order for requests to be batchable the resolver they reference has to be the same so it is important to avoid over providing context to a resolver because having even two slightly different services makes the resolvers incompatible leading to no batching.

To avoid easy mistakes we decided to force the context of the resolver passed to Effect.request to never so that you always have to specify how context is accessed.

Let's see how we would do it:

ts
import * as RequestResolver from "@effect/io/RequestResolver"
 
interface HttpService {
  fetch: typeof fetch
}
 
const HttpService = Context.Tag<HttpService>(
  Symbol.for("@app/services/HttpService")
)
 
const HttpServiceLive = Layer.sync(HttpService, () => ({ fetch }))
 
const GetTodosResolver = pipe(
  // we create a normal resolver like we did before
  RequestResolver.fromFunctionEffect((request: GetTodos) =>
    Effect.flatMap(HttpService, (http) => Effect.tryCatchPromise(
      () => http.fetch("https://api.example.demo/todos").then((_) => _.json()) as Promise<Todo[]>,
      () => new GetTodosError()
    ))
  ),
  // we list the tags that the resolver can access
  RequestResolver.contextFromServices(HttpService)
)

ts
import * as RequestResolver from "@effect/io/RequestResolver"
 
interface HttpService {
  fetch: typeof fetch
}
 
const HttpService = Context.Tag<HttpService>(
  Symbol.for("@app/services/HttpService")
)
 
const HttpServiceLive = Layer.sync(HttpService, () => ({ fetch }))
 
const GetTodosResolver = pipe(
  // we create a normal resolver like we did before
  RequestResolver.fromFunctionEffect((request: GetTodos) =>
    Effect.flatMap(HttpService, (http) => Effect.tryCatchPromise(
      () => http.fetch("https://api.example.demo/todos").then((_) => _.json()) as Promise<Todo[]>,
      () => new GetTodosError()
    ))
  ),
  // we list the tags that the resolver can access
  RequestResolver.contextFromServices(HttpService)
)

We can see now that the type of GetTodosResolver is no longer a RequestResolver but instead it is Effect.Effect<HttpService, never, RequestResolver.RequestResolver<GetTodos, never>> which is an Effect that access the HttpService and returns a composed resolver that has the minimal context ready to use.

Once we have such Effect we can directly use it in our request definition:

ts
const getTodos = Effect.request(
         
const getTodos: Effect.Effect<HttpService, GetTodosError, Todo[]>
  GetTodos({}),
  GetTodosResolver
)

ts
const getTodos = Effect.request(
         
const getTodos: Effect.Effect<HttpService, GetTodosError, Todo[]>
  GetTodos({}),
  GetTodosResolver
)

We can see that the Effect correctly requires HttpService to be provided.

Alternatively you can create RequestResolvers as part of layers direcly accessing or closing over context from construction.

For example:

ts
interface HttpService {
  fetch: typeof fetch
}
 
const HttpService = Context.Tag<HttpService>(
  Symbol.for("@app/services/HttpService")
)
 
const HttpServiceLive = Layer.sync(HttpService, () => ({ fetch }))
 
interface TodosService {
  getTodos: Effect.Effect<never, GetTodosError, Todo[]>
}
 
const TodosService = Context.Tag<TodosService>(
  Symbol.for("@app/services/TodosService")
)
 
const TodosServiceLive = Layer.effect(
  TodosService,
  Effect.gen(function* ($) {
    const http = yield* $(HttpService)
    const resolver = RequestResolver.fromFunctionEffect((request: GetTodos) =>
      Effect.tryCatchPromise(
        () =>
          http
            .fetch("https://api.example.demo/todos")
            .then((_) => _.json()) as Promise<Todo[]>,
        () => new GetTodosError()
      )
    )
    return {
      getTodos: Effect.request(GetTodos({}), resolver),
    }
  })
)
const getTodos = Effect.flatMap(TodosService, (_) => _.getTodos)
         
const getTodos: Effect.Effect<TodosService, GetTodosError, Todo[]>

ts
interface HttpService {
  fetch: typeof fetch
}
 
const HttpService = Context.Tag<HttpService>(
  Symbol.for("@app/services/HttpService")
)
 
const HttpServiceLive = Layer.sync(HttpService, () => ({ fetch }))
 
interface TodosService {
  getTodos: Effect.Effect<never, GetTodosError, Todo[]>
}
 
const TodosService = Context.Tag<TodosService>(
  Symbol.for("@app/services/TodosService")
)
 
const TodosServiceLive = Layer.effect(
  TodosService,
  Effect.gen(function* ($) {
    const http = yield* $(HttpService)
    const resolver = RequestResolver.fromFunctionEffect((request: GetTodos) =>
      Effect.tryCatchPromise(
        () =>
          http
            .fetch("https://api.example.demo/todos")
            .then((_) => _.json()) as Promise<Todo[]>,
        () => new GetTodosError()
      )
    )
    return {
      getTodos: Effect.request(GetTodos({}), resolver),
    }
  })
)
const getTodos = Effect.flatMap(TodosService, (_) => _.getTodos)
         
const getTodos: Effect.Effect<TodosService, GetTodosError, Todo[]>

This way is probably the best for most of the cases given that layers are the natural primitive where to wire services together.

Controlling Batching

Batching can be locally disabled using the Effect.withRequestBatching("off") utility in the following way:

ts
const program = Effect.withRequestBatching("off")(
  Effect.flatMap(getTodos, Effect.forEachParDiscard(notifyOwner))
)

ts
const program = Effect.withRequestBatching("off")(
  Effect.flatMap(getTodos, Effect.forEachParDiscard(notifyOwner))
)

Request Caching

Up to this point we optimized how requests are executed but there is still a catch - we are not doing any caching.

This leads to request duplication...

Fortunately we also have a primitive for caching in Effect and we use that to automatically cache requests.

ts
const getUserById = (id: number) =>
  Effect.withRequestCaching("on")(
    Effect.request(GetUserById({ id }), GetUserByIdResolver)
  )

ts
const getUserById = (id: number) =>
  Effect.withRequestCaching("on")(
    Effect.request(GetUserById({ id }), GetUserByIdResolver)
  )

Final Program

Assuming you've wired everything up correctly the following program:

ts
import * as Schedule from "@effect/io/Schedule"
 
const main = pipe(
  getTodos,
  Effect.flatMap(Effect.forEachParDiscard(notifyOwner)),
  Effect.repeat(Schedule.fixed(seconds(10)))
)

ts
import * as Schedule from "@effect/io/Schedule"
 
const main = pipe(
  getTodos,
  Effect.flatMap(Effect.forEachParDiscard(notifyOwner)),
  Effect.repeat(Schedule.fixed(seconds(10)))
)

should never execute the same GetUserById twice within a span of 1 minute (assuming less than 65k users) while also making sure emails are sent in batches.

The Request Cache

There may be cases where you want to localize a cache (use a cache only for a part of your program) or maybe you want a global cache with a different setup, or a mix of both.

To cover those scenarios you'd create a custom cache like the following:

ts
import * as Schedule from "@effect/io/Schedule"
 
const main = pipe(
  getTodos,
  Effect.flatMap(Effect.forEachParDiscard(notifyOwner)),
  Effect.repeat(Schedule.fixed(seconds(10))),
  Effect.provideSomeLayer(
    Effect.setRequestCache(Request.makeCache(256, minutes(60)))
  )
)

ts
import * as Schedule from "@effect/io/Schedule"
 
const main = pipe(
  getTodos,
  Effect.flatMap(Effect.forEachParDiscard(notifyOwner)),
  Effect.repeat(Schedule.fixed(seconds(10))),
  Effect.provideSomeLayer(
    Effect.setRequestCache(Request.makeCache(256, minutes(60)))
  )
)

Alternatively you can also directly construct a cache with Request.makeCache(256, minutes(60)) and then use Effect.withRequestCache(myCache)(program) on a program to make sure the requests generated from that program uses the custom cache (when enabled with Effect.withRequestCaching("on"))

How is this possible?

We recently introduced a new key primitive in the fiber that enables an execution to pause when it sees the program requires a request. In the process of pausing, the fiber will reify its stack into a continuation that can be externally performed.

ts
const program = Effect.flatMap(getTodos, Effect.forEachParDiscard(notifyOwner))
 
const nextStep = Effect.flatMapStep(program, (step) => {
  switch (step._tag) {
    // the program is blocked on a number of requests
    case "Blocked": {
      const requests = step.i0
      const continuation = step.i1
      return Effect.flatMap(
        shipRequestsToBeExecutedAndWait(requests),
        () => continuation
      )
    }
    // the program completed
    case "Success":
    case "Failure": {
      return step
    }
  }
})
 
const shipRequestsToBeExecutedAndWait = <R, E, A>(
  requests: Effect.Blocked<R, E, A>["i0"]
): Effect.Effect<R, E, void> => {
  // go on mars and come back
  return Effect.unit()
}

ts
const program = Effect.flatMap(getTodos, Effect.forEachParDiscard(notifyOwner))
 
const nextStep = Effect.flatMapStep(program, (step) => {
  switch (step._tag) {
    // the program is blocked on a number of requests
    case "Blocked": {
      const requests = step.i0
      const continuation = step.i1
      return Effect.flatMap(
        shipRequestsToBeExecutedAndWait(requests),
        () => continuation
      )
    }
    // the program completed
    case "Success":
    case "Failure": {
      return step
    }
  }
})
 
const shipRequestsToBeExecutedAndWait = <R, E, A>(
  requests: Effect.Blocked<R, E, A>["i0"]
): Effect.Effect<R, E, void> => {
  // go on mars and come back
  return Effect.unit()
}

By using the functions provided by the @effect/io/RequestBlock module, you can combine requests from multiple blocked effects. By using the function Effect.blocked(requests, continuation), you can express an effect that is blocked on requests that should continue with continuation.

Using Cache Directly

There are many cases where you have functions (key: Key) => Effect<R, E, A> that you would like to cache and not necessarily every case is a good fit for the request model shown above. For example, non-batchable API calls or intensive work.

Let's see how we would go about using cache:

ts
import * as Effect from "@effect/io/Effect"
import * as Cache from "@effect/io/Cache"
import * as Duration from "@effect/data/Duration"
 
declare const intensiveWork: (
  key: string
) => Effect.Effect<never, never, string>
 
const program = Effect.gen(function* ($) {
  const cache = yield* $(
    Cache.make(Number.MAX_SAFE_INTEGER, Duration.infinity, intensiveWork)
  )
 
  const a0 = yield* $(cache.get("key0"))
  const b0 = yield* $(cache.get("key1"))
  const a1 = yield* $(cache.get("key0"))
  const b1 = yield* $(cache.get("key1"))
 
  if (a0 === a1 && b0 === b1) {
    yield* $(Effect.log("I'll always end up here...."))
  }
})

ts
import * as Effect from "@effect/io/Effect"
import * as Cache from "@effect/io/Cache"
import * as Duration from "@effect/data/Duration"
 
declare const intensiveWork: (
  key: string
) => Effect.Effect<never, never, string>
 
const program = Effect.gen(function* ($) {
  const cache = yield* $(
    Cache.make(Number.MAX_SAFE_INTEGER, Duration.infinity, intensiveWork)
  )
 
  const a0 = yield* $(cache.get("key0"))
  const b0 = yield* $(cache.get("key1"))
  const a1 = yield* $(cache.get("key0"))
  const b1 = yield* $(cache.get("key1"))
 
  if (a0 === a1 && b0 === b1) {
    yield* $(Effect.log("I'll always end up here...."))
  }
})

In order for the cache to correctly compare two Key values if you are not using primitives (e.g. string, boolean, number), you should use types that implement the @effect/data/Equal interface.

There are many more methods available in the Cache module. As a next step, check out the reference docs!

Reference Docs

The following are the reference docs for each of the mentioned modules:

Services Option