Creating Streams

In this section, we’ll explore various methods for creating Effect Streams. These methods will help you generate streams tailored to your needs.

Common Constructors

make

You can create a pure stream by using the Stream.make constructor. This constructor accepts a variable list of values as its arguments.

1
import { Stream, Effect } from "effect"
2

3
const stream = Stream.make(1, 2, 3)
4

5
await Effect.runPromise(Stream.runCollect(stream)) // => [1, 2, 3]

empty

Sometimes, you may require a stream that doesn’t produce any values. In such cases, you can use Stream.empty. This constructor creates a stream that remains empty.

1
import { Stream, Effect } from "effect"
2

3
const stream = Stream.empty
4

5
await Effect.runPromise(Stream.runCollect(stream)) // => []

void

If you need a stream that contains a single void value, you can use Stream.succeed(void 0). This is handy when you want to represent a stream with a single event or signal.

1
import { Stream, Effect } from "effect"
2

3
const stream = Stream.succeed(void 0)
4

5
await Effect.runPromise(Stream.runCollect(stream)) // => [undefined]

range

To create a stream of integers within a specified range [min, max] (including both endpoints, min and max), you can use Stream.range. This is particularly useful for generating a stream of sequential numbers.

1
import { Stream, Effect } from "effect"
2

3
// Creating a stream of numbers from 1 to 5
4
const stream = Stream.range(1, 5)
5

6
await Effect.runPromise(Stream.runCollect(stream)) // => [1, 2, 3, 4, 5]

iterate

With Stream.iterate, you can generate a stream by applying a function iteratively to an initial value. The initial value becomes the first element produced by the stream, followed by subsequent values produced by f(init), f(f(init)), and so on.

1
import { Stream, Effect } from "effect"
2

3
// Creating a stream of incrementing numbers
4
const stream = Stream.iterate(1, (n) => n + 1) // Produces 1, 2, 3, ...
5

6
await Effect.runPromise(Stream.runCollect(stream.pipe(Stream.take(5)))) // => [1, 2, 3, 4, 5]

scoped

Stream.scoped is used to create a single-valued stream from a scoped resource. It can be handy when dealing with resources that require explicit acquisition, usage, and release.

1
import { Stream, Effect, Console } from "effect"
2

3
// Creating a single-valued stream from a scoped resource
4
const stream = Stream.scoped(
5
  Stream.fromEffect(
6
    Effect.acquireUseRelease(
7
      Console.log("acquire"),
8
      () => Console.log("use"),
9
      () => Console.log("release"),
10
    ),
11
  ),
12
)
13

14
await Effect.runPromise(Stream.runCollect(stream)) // => [undefined]
15
/*
16
Output:
17
acquire
18
use
19
release
20
*/

From Success and Failure

Much like the Effect data type, you can generate a Stream using the fail and succeed functions:

1
import { Stream, Effect } from "effect"
2

3
// Creating a stream that can emit errors
4
const streamWithError: Stream.Stream<never, string> = Stream.fail("Uh oh!")
5

6
Effect.runPromise(Stream.runCollect(streamWithError))
7
// throws Error: Uh oh!
8

9
// Creating a stream that emits a numeric value
10
const streamWithNumber: Stream.Stream<number> = Stream.succeed(5)
11

12
Effect.runPromise(Stream.runCollect(streamWithNumber)).then(console.log)
13
// [ 5 ]

From Arrays

You can construct a stream from an array like this:

1
import { Stream, Effect } from "effect"
2

3
// Creating a stream with values from a single array
4
const stream = Stream.fromArray([1, 2, 3])
5

6
await Effect.runPromise(Stream.runCollect(stream)) // => [1, 2, 3]

Moreover, you can create a stream from multiple arrays as well:

1
import { Stream, Effect } from "effect"
2

3
// Creating a stream with values from multiple arrays
4
const stream = Stream.fromArrays([1, 2, 3], [4, 5, 6])
5

6
await Effect.runPromise(Stream.runCollect(stream)) // => [1, 2, 3, 4, 5, 6]

From Effect

You can generate a stream from an Effect workflow by employing the Stream.fromEffect constructor. For instance, consider the following stream, which generates a single random number:

1
import { Stream, Random, Effect } from "effect"
2

3
const stream = Stream.fromEffect(Random.nextInt)
4

5
Effect.runPromise(Stream.runCollect(stream)).then(console.log)
6
// Example Output: [ 1042302242 ]
7

8
// The value is random, but the stream always emits exactly one element
9
const result = await Effect.runPromise(Stream.runCollect(stream))
10
result.length // => 1

This method allows you to seamlessly transform the output of an Effect into a stream, providing a straightforward way to work with asynchronous operations within your streams.

From Asynchronous Callback

Imagine you have an asynchronous function that relies on callbacks. If you want to capture the results emitted by those callbacks as a stream, you can use the Stream.callback function. This function is designed to adapt functions that invoke their callbacks multiple times and emit the results as a stream.

Let’s break down how to use it in the following example:

1
import { Stream, Effect, Queue } from "effect"
2

3
const events = [1, 2, 3, 4]
4

5
const stream = Stream.callback<number>((queue) =>
6
  Effect.sync(() => {
7
    events.forEach((n) => {
8
      setTimeout(() => {
9
        if (n === 3) {
10
          // Terminate the stream
11
          Queue.endUnsafe(queue)
12
        } else {
13
          // Add the current item to the stream
14
          Queue.offerUnsafe(queue, n)
15
        }
16
      }, 100 * n)
17
    })
18
  }),
19
)
20

21
await Effect.runPromise(Stream.runCollect(stream)) // => [1, 2]

The function you pass to Stream.callback receives a Queue that you use to drive the stream from your asynchronous code. Here’s what each of the possible outcomes means:

• Calling Queue.offerUnsafe (or the effectful Queue.offer) on the queue emits the given element as part of the stream.

• Calling Queue.fail (or Queue.failCauseUnsafe/Queue.failCause) on the queue terminates the stream with the specified error.

• Calling Queue.endUnsafe/Queue.end on the queue signals the end of the stream, terminating it successfully.

To put it simply, this gives you full control over how your asynchronous callback interacts with the stream, determining when to emit elements, when to terminate with an error, or when to signal the end of the stream.

From Iterables

fromIterable

You can create a pure stream from an Iterable of values using the Stream.fromIterable constructor. It’s a straightforward way to convert a collection of values into a stream.

1
import { Stream, Effect } from "effect"
2

3
const numbers = [1, 2, 3]
4

5
const stream = Stream.fromIterable(numbers)
6

7
await Effect.runPromise(Stream.runCollect(stream)) // => [1, 2, 3]

fromIterableEffect

When you have an effect that produces a value of type Iterable, you can employ the Stream.fromIterableEffect constructor to generate a stream from that effect.

For instance, let’s say you have a database operation that retrieves a list of users. Since this operation involves effects, you can utilize Stream.fromIterableEffect to convert the result into a Stream:

1
import { Stream, Effect, Context } from "effect"
2

3
class Database extends Context.Service<
4
  Database,
5
  { readonly getUsers: Effect.Effect<Array<string>> }
6
>()("Database") {}
7

8
const getUsers = Database.use((_) => _.getUsers)
9

10
const stream = Stream.fromIterableEffect(getUsers)
11

12
await Effect.runPromise(
13
  Stream.runCollect(
14
    stream.pipe(
15
      Stream.provideService(Database, {
16
        getUsers: Effect.succeed(["user1", "user2"]),
17
      }),
18
    ),
19
  ),
20
) // => ["user1", "user2"]

This enables you to work seamlessly with effects and convert their results into streams for further processing.

fromAsyncIterable

Async iterables are another type of data source that can be converted into a stream. With the Stream.fromAsyncIterable constructor, you can work with asynchronous data sources and handle potential errors gracefully.

1
import { Stream, Effect } from "effect"
2

3
const myAsyncIterable = async function* () {
4
  yield 1
5
  yield 2
6
}
7

8
const stream = Stream.fromAsyncIterable(
9
  myAsyncIterable(),
10
  (e) => new Error(String(e)), // Error Handling
11
)
12

13
await Effect.runPromise(Stream.runCollect(stream)) // => [1, 2]

In this code, we define an async iterable and then create a stream named stream from it. Additionally, we provide an error handler function to manage any potential errors that may occur during the conversion.

From Repetition

Repeating a Single Value

You can create a stream that endlessly repeats a specific value using Stream.forever(Stream.succeed(value)):

1
import { Stream, Effect } from "effect"
2

3
const stream = Stream.forever(Stream.succeed(0))
4

5
await Effect.runPromise(Stream.runCollect(stream.pipe(Stream.take(5)))) // => [0, 0, 0, 0, 0]

Repeating a Stream’s Content

Stream.repeat allows you to create a stream that repeats a specified stream’s content according to a schedule. This can be useful for generating recurring events or values.

1
import { Stream, Effect, Schedule } from "effect"
2

3
// Creating a stream that repeats a value indefinitely
4
const stream = Stream.repeat(Stream.succeed(1), Schedule.forever)
5

6
await Effect.runPromise(Stream.runCollect(stream.pipe(Stream.take(5)))) // => [1, 1, 1, 1, 1]

Repeating an Effect’s Result

Imagine you have an effectful API call, and you want to use the result of that call to create a stream. You can achieve this by creating a stream from the effect and repeating it indefinitely.

Here’s an example of generating a stream of random numbers:

1
import { Stream, Effect, Random } from "effect"
2

3
const stream = Stream.fromEffectRepeat(Random.nextInt)
4

5
Effect.runPromise(Stream.runCollect(stream.pipe(Stream.take(5)))).then(
6
  console.log,
7
)
8
/*
9
Example Output:
10
[ 1666935266, 604851965, 2194299958, 3393707011, 4090317618 ]
11
*/
12

13
// The values are random, but the stream always emits exactly 5 elements
14
const result = await Effect.runPromise(
15
  Stream.runCollect(stream.pipe(Stream.take(5))),
16
)
17
result.length // => 5

Repeating an Effect with Termination

You can repeatedly evaluate a given effect and terminate the stream based on specific conditions.

In this example, we’re draining an Iterator to create a stream from it:

1
import { Stream, Effect, Cause } from "effect"
2

3
const drainIterator = <A>(it: Iterator<A>): Stream.Stream<A> =>
4
  Stream.fromEffectRepeat(
5
    Effect.sync(() => it.next()).pipe(
6
      Effect.andThen((res) => {
7
        if (res.done) {
8
          return Cause.done()
9
        }
10
        return Effect.succeed(res.value)
11
      }),
12
    ),
13
  )
14

15
const numbers = [10, 20, 30]
16

17
await Effect.runPromise(
18
  Stream.runCollect(drainIterator(numbers[Symbol.iterator]())),
19
) // => [10, 20, 30]

Generating Ticks

You can create a stream that emits void values at specified intervals using the Stream.tick constructor. This is useful for creating periodic events.

1
import { Stream, Effect } from "effect"
2

3
const stream = Stream.tick("100 millis")
4

5
await Effect.runPromise(Stream.runCollect(stream.pipe(Stream.take(5)))) // => [undefined, undefined, undefined, undefined, undefined]

From Unfolding/Pagination

In functional programming, the concept of unfold can be thought of as the counterpart to fold.

With fold, we process a data structure and produce a return value. For example, we can take an Array<number> and calculate the sum of its elements.

On the other hand, unfold represents an operation where we start with an initial value and generate a recursive data structure, adding one element at a time using a specified state function. For example, we can create a sequence of natural numbers starting from 1 and using the increment function as the state function.

Unfold

unfold

The Stream module includes an unfold function defined as follows:

declare const unfold: <S, A, E, R>(
  initialState: S,
  step: (s: S) => Effect.Effect<readonly [A, S] | undefined, E, R>,
) => Stream<A, E, R>

Here’s how it works:

• initialState. This is the initial state value.
• step. The state function step takes the current state s as input and returns an effect. If the effect resolves to undefined, the stream ends. If it resolves to a tuple [A, S], the next element in the stream is A, and the state S is updated for the next step process.

For example, let’s create a stream of natural numbers using Stream.unfold:

1
import { Stream, Effect } from "effect"
2

3
const stream = Stream.unfold(1, (n) => Effect.succeed([n, n + 1] as const))
4

5
await Effect.runPromise(Stream.runCollect(stream.pipe(Stream.take(5)))) // => [1, 2, 3, 4, 5]

Unfolding with Effects

Sometimes, we may need to perform effectful state transformations during the unfolding operation. Since the step function passed to Stream.unfold already returns an Effect, it can depend on any effectful computation, such as generating a random value, while producing the next element and state.

Here’s an example of creating an infinite stream of random 1 and -1 values using Stream.unfold:

1
import { Stream, Effect, Random } from "effect"
2

3
const stream = Stream.unfold(1, (n) =>
4
  Random.nextBoolean.pipe(
5
    Effect.map((b) => (b ? ([n, -n] as const) : ([n, n] as const))),
6
  ),
7
)
8

9
Effect.runPromise(Stream.runCollect(stream.pipe(Stream.take(5)))).then(
10
  console.log,
11
)
12
// Example Output: [ 1, 1, 1, 1, -1 ]
13

14
// The sign is random, but the state starts at 1 and only ever flips sign,
15
// so its absolute value is deterministically always 1
16
const result = await Effect.runPromise(
17
  Stream.runCollect(stream.pipe(Stream.take(5))),
18
)
19
result.map(Math.abs) // => [1, 1, 1, 1, 1]

Pagination

paginate

Stream.paginate is similar to Stream.unfold but allows emitting values one step further.

For example, the following stream emits 0, 1, 2, 3 elements:

1
import { Stream, Effect, Option } from "effect"
2

3
const stream = Stream.paginate(0, (n) =>
4
  Effect.succeed([[n], n < 3 ? Option.some(n + 1) : Option.none()] as const),
5
)
6

7
await Effect.runPromise(Stream.runCollect(stream)) // => [0, 1, 2, 3]

Here’s how it works:

• We start with an initial value of 0.
• The provided function takes the current value n and returns a tuple. The first element of the tuple is the value to emit (n), and the second element determines whether to continue (Option.some(n + 1)) or stop (Option.none()).

Unfolding vs. Pagination

You might wonder about the difference between the unfold and paginate combinators and when to use one over the other. Stream.unfold produces exactly one value per step, so it can’t consume an API whose step naturally returns a batch of values at once. Stream.paginate is built for that shape: each step returns an array of values together with the next state, so a single call can emit zero, one, or many elements before deciding whether to continue.

This is exactly the shape of a paginated API. Imagine a fetchUsers request that, given a cursor, returns a page of users and, if there’s more data, the cursor for the next page:

1
import { Effect, Option, Stream } from "effect"
2

3
interface Page<A> {
4
  readonly items: ReadonlyArray<A>
5
  readonly nextCursor: number | undefined
6
}
7

8
// A mock paginated API: six users, two per page.
9
const fetchUsers = (cursor: number): Effect.Effect<Page<string>> => {
10
  const users = ["Alice", "Bob", "Carol", "Dave", "Erin", "Frank"]
11
  const pageSize = 2
12
  const items = users.slice(cursor, cursor + pageSize)
13
  const nextCursor =
14
    cursor + pageSize < users.length ? cursor + pageSize : undefined
15
  return Effect.succeed({ items, nextCursor })
16
}
17

18
const stream = Stream.paginate(0, (cursor) =>
19
  fetchUsers(cursor).pipe(
20
    Effect.map(
21
      (page) => [page.items, Option.fromUndefinedOr(page.nextCursor)] as const,
22
    ),
23
  ),
24
)
25

26
await Effect.runPromise(Stream.runCollect(stream)) // => ["Alice", "Bob", "Carol", "Dave", "Erin", "Frank"]

Each call to fetchUsers hands back a whole page of items, and Stream.paginate flattens every page into the resulting stream, stopping once nextCursor is undefined. Modeling this with Stream.unfold would require peeling one item off the current page at a time and keeping the rest around as extra state. Stream.paginate already handles that logic by accepting an array per step.

From Queue and PubSub

In Effect, there are two essential asynchronous messaging data types: Queue and PubSub. You can easily transform these data types into Streams by utilizing Stream.fromQueue and Stream.fromPubSub, respectively.

From Schedule

We can create a stream from a Schedule that does not require any further input. The stream will emit an element for each value output from the schedule, continuing for as long as the schedule continues:

1
import { Effect, Stream, Schedule } from "effect"
2

3
// Emits values every 100 milliseconds for a total of 10 emissions
4
const schedule = Schedule.spaced("100 millis").pipe(
5
  Schedule.upTo({ times: 10 }),
6
)
7

8
const stream = Stream.fromSchedule(schedule)
9

10
await Effect.runPromise(Stream.runCollect(stream)) // => [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]