Schema 0.69 (Release) – Effect Blog

#Recap of 0.68 patches

Several features were introduced in version 0.68 through various patches, here's a recap of the most important updates in case you missed them.

#Improvements in JSON Schema Generation

#Handling undefined Types

The behavior of JSON Schema generation has been refined to enhance the handling of optional fields. Previously, schemas containing undefined could lead to exceptions; now, they are treated as optional automatically.

Before Update:

ts
import { JSONSchema, Schema } from "@effect/schema"
const schema = Schema.Struct({
  a: Schema.NullishOr(Schema.Number)
})
const jsonSchema = JSONSchema.make(schema)
console.log(JSON.stringify(jsonSchema, null, 2))
/*
throws
Error: Missing annotation
at path: ["a"]
details: Generating a JSON Schema for this schema requires a "jsonSchema" annotation
schema (UndefinedKeyword): undefined
*/

ts
import { JSONSchema, Schema } from "@effect/schema"
const schema = Schema.Struct({
  a: Schema.NullishOr(Schema.Number)
})
const jsonSchema = JSONSchema.make(schema)
console.log(JSON.stringify(jsonSchema, null, 2))
/*
throws
Error: Missing annotation
at path: ["a"]
details: Generating a JSON Schema for this schema requires a "jsonSchema" annotation
schema (UndefinedKeyword): undefined
*/

After Update:

ts
import { JSONSchema, Schema } from "@effect/schema"
const schema = Schema.Struct({
  a: Schema.NullishOr(Schema.Number)
})
const jsonSchema = JSONSchema.make(schema)
console.log(JSON.stringify(jsonSchema, null, 2))
/*
{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "type": "object",
  "required": [], // <=== empty
  "properties": {
    "a": {
      "anyOf": [
        {
          "type": "number"
        },
        {
          "$ref": "#/$defs/null"
        }
      ]
    }
  },
  "additionalProperties": false,
  "$defs": {
    "null": {
      "const": null
    }
  }
}
*/

ts
import { JSONSchema, Schema } from "@effect/schema"
const schema = Schema.Struct({
  a: Schema.NullishOr(Schema.Number)
})
const jsonSchema = JSONSchema.make(schema)
console.log(JSON.stringify(jsonSchema, null, 2))
/*
{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "type": "object",
  "required": [], // <=== empty
  "properties": {
    "a": {
      "anyOf": [
        {
          "type": "number"
        },
        {
          "$ref": "#/$defs/null"
        }
      ]
    }
  },
  "additionalProperties": false,
  "$defs": {
    "null": {
      "const": null
    }
  }
}
*/

#Refinements in Records

The generation of JSON schemas from records that utilize refinements has been improved to ensure error-free outputs.

Before Update:

ts
import { JSONSchema, Schema } from "@effect/schema"
const schema = Schema.Record({
  key: Schema.String.pipe(Schema.minLength(1)),
  value: Schema.Number
})
console.log(JSONSchema.make(schema))
/*
throws
Error: Unsupported index signature parameter
schema (Refinement): a string at least 1 character(s) long
*/

ts
import { JSONSchema, Schema } from "@effect/schema"
const schema = Schema.Record({
  key: Schema.String.pipe(Schema.minLength(1)),
  value: Schema.Number
})
console.log(JSONSchema.make(schema))
/*
throws
Error: Unsupported index signature parameter
schema (Refinement): a string at least 1 character(s) long
*/

After Update:

ts
import { JSONSchema, Schema } from "@effect/schema"
const schema = Schema.Record({
  key: Schema.String.pipe(Schema.minLength(1)),
  value: Schema.Number
})
console.log(JSONSchema.make(schema))
/*
Output:
{
  '$schema': 'http://json-schema.org/draft-07/schema#',
  type: 'object',
  required: [],
  properties: {},
  patternProperties: { '': { type: 'number' } },
  propertyNames: {
    type: 'string',
    description: 'a string at least 1 character(s) long',
    minLength: 1
  }
}
*/

ts
import { JSONSchema, Schema } from "@effect/schema"
const schema = Schema.Record({
  key: Schema.String.pipe(Schema.minLength(1)),
  value: Schema.Number
})
console.log(JSONSchema.make(schema))
/*
Output:
{
  '$schema': 'http://json-schema.org/draft-07/schema#',
  type: 'object',
  required: [],
  properties: {},
  patternProperties: { '': { type: 'number' } },
  propertyNames: {
    type: 'string',
    description: 'a string at least 1 character(s) long',
    minLength: 1
  }
}
*/

#parseJson Schemas

Resolved an issue where JSONSchema.make improperly generated JSON Schemas for schemas defined with S.parseJson(<real schema>). Previously, invoking JSONSchema.make on these transformed schemas produced a JSON Schema corresponding to a string type rather than the underlying real schema.

Before Update:

ts
import { JSONSchema, Schema } from "@effect/schema"
// Define a schema that parses a JSON string into a structured object
const schema = Schema.parseJson(
  Schema.Struct({
    a: Schema.parseJson(Schema.NumberFromString) // Nested parsing from JSON string to number
  })
)
console.log(JSONSchema.make(schema))
/*
{
  '$schema': 'http://json-schema.org/draft-07/schema#',
  '$ref': '#/$defs/JsonString',
  '$defs': {
    JsonString: {
      type: 'string',
      description: 'a JSON string',
      title: 'JsonString'
    }
  }
}
*/

ts
import { JSONSchema, Schema } from "@effect/schema"
// Define a schema that parses a JSON string into a structured object
const schema = Schema.parseJson(
  Schema.Struct({
    a: Schema.parseJson(Schema.NumberFromString) // Nested parsing from JSON string to number
  })
)
console.log(JSONSchema.make(schema))
/*
{
  '$schema': 'http://json-schema.org/draft-07/schema#',
  '$ref': '#/$defs/JsonString',
  '$defs': {
    JsonString: {
      type: 'string',
      description: 'a JSON string',
      title: 'JsonString'
    }
  }
}
*/

After Update:

ts
import { JSONSchema, Schema } from "@effect/schema"
// Define a schema that parses a JSON string into a structured object
const schema = Schema.parseJson(
  Schema.Struct({
    a: Schema.parseJson(Schema.NumberFromString) // Nested parsing from JSON string to number
  })
)
console.log(JSONSchema.make(schema))
/*
{
  '$schema': 'http://json-schema.org/draft-07/schema#',
  type: 'object',
  required: [ 'a' ],
  properties: { a: { type: 'string', description: 'a string', title: 'string' } },
  additionalProperties: false
}
*/

ts
import { JSONSchema, Schema } from "@effect/schema"
// Define a schema that parses a JSON string into a structured object
const schema = Schema.parseJson(
  Schema.Struct({
    a: Schema.parseJson(Schema.NumberFromString) // Nested parsing from JSON string to number
  })
)
console.log(JSONSchema.make(schema))
/*
{
  '$schema': 'http://json-schema.org/draft-07/schema#',
  type: 'object',
  required: [ 'a' ],
  properties: { a: { type: 'string', description: 'a string', title: 'string' } },
  additionalProperties: false
}
*/

#New String Transformations and filters

We have introduced new transformations and filters to enhance string processing capabilities:

Transformations: Capitalize, Uncapitalize
Filters: Capitalized, Uncapitalized

#Asynchronous Validation with filterEffect

The filterEffect function enhances the filter functionality by allowing the integration of effects, thus enabling asynchronous or dynamic validation scenarios. This is particularly useful when validations need to perform operations that require side effects, such as network requests or database queries.

Example: Validating Usernames Asynchronously

ts
import { Schema } from "@effect/schema"
import { Effect } from "effect"
async function validateUsername(username: string) {
  return Promise.resolve(username === "gcanti")
}
const ValidUsername = Schema.String.pipe(
  Schema.filterEffect((username) =>
    Effect.promise(() =>
      validateUsername(username).then((valid) => valid || "Invalid username")
    )
  )
).annotations({ identifier: "ValidUsername" })
Effect.runPromise(Schema.decodeUnknown(ValidUsername)("xxx")).then(
  console.log
)
/*
ParseError: ValidUsername
└─ Transformation process failure
   └─ Invalid username
*/

ts
import { Schema } from "@effect/schema"
import { Effect } from "effect"
async function validateUsername(username: string) {
  return Promise.resolve(username === "gcanti")
}
const ValidUsername = Schema.String.pipe(
  Schema.filterEffect((username) =>
    Effect.promise(() =>
      validateUsername(username).then((valid) => valid || "Invalid username")
    )
  )
).annotations({ identifier: "ValidUsername" })
Effect.runPromise(Schema.decodeUnknown(ValidUsername)("xxx")).then(
  console.log
)
/*
ParseError: ValidUsername
└─ Transformation process failure
   └─ Invalid username
*/

#Introducing ReadonlyMapFromRecord and MapFromRecord

These new functionalities provide efficient transformations between records and maps, supporting both encoding and decoding processes.

decoding
- { readonly [x: string]: VI } -> ReadonlyMap<KA, VA>
encoding
- ReadonlyMap<KA, VA> -> { readonly [x: string]: VI }

Example:

ts
import { Schema } from "@effect/schema"
const schema = Schema.ReadonlyMapFromRecord({
  key: Schema.BigInt,
  value: Schema.NumberFromString
})
const decode = Schema.decodeUnknownSync(schema)
const encode = Schema.encodeSync(schema)
console.log(
  decode({
    "1": "4",
    "2": "5",
    "3": "6"
  })
) // Map(3) { 1n => 4, 2n => 5, 3n => 6 }
console.log(
  encode(
    new Map([
      [1n, 4],
      [2n, 5],
      [3n, 6]
    ])
  )
) // { '1': '4', '2': '5', '3': '6' }

ts
import { Schema } from "@effect/schema"
const schema = Schema.ReadonlyMapFromRecord({
  key: Schema.BigInt,
  value: Schema.NumberFromString
})
const decode = Schema.decodeUnknownSync(schema)
const encode = Schema.encodeSync(schema)
console.log(
  decode({
    "1": "4",
    "2": "5",
    "3": "6"
  })
) // Map(3) { 1n => 4, 2n => 5, 3n => 6 }
console.log(
  encode(
    new Map([
      [1n, 4],
      [2n, 5],
      [3n, 6]
    ])
  )
) // { '1': '4', '2': '5', '3': '6' }

#Expanded extend Functionality

The extend function now supports combinations with Union, Suspend, and Refinement, broadening its applicability and flexibility.

#Enhanced Struct Operations: pick and omit

These operations allow selective inclusion or exclusion of properties from structs, providing more control over schema composition.

Using pick:

The pick static function available in each struct schema can be used to create a new Struct by selecting particular properties from an existing Struct.

ts
import { Schema } from "@effect/schema"
const MyStruct = Schema.Struct({
  a: Schema.String,
  b: Schema.Number,
  c: Schema.Boolean
})
// Schema.Struct<{ a: typeof Schema.String; c: typeof Schema.Boolean; }>
const PickedSchema = MyStruct.pick("a", "c")

ts
import { Schema } from "@effect/schema"
const MyStruct = Schema.Struct({
  a: Schema.String,
  b: Schema.Number,
  c: Schema.Boolean
})
// Schema.Struct<{ a: typeof Schema.String; c: typeof Schema.Boolean; }>
const PickedSchema = MyStruct.pick("a", "c")

Using omit:

The omit static function available in each struct schema can be used to create a new Struct by excluding particular properties from an existing Struct.

ts
import { Schema } from "@effect/schema"
const MyStruct = Schema.Struct({
  a: Schema.String,
  b: Schema.Number,
  c: Schema.Boolean
})
// Schema.Struct<{ a: typeof Schema.String; c: typeof Schema.Boolean; }>
const PickedSchema = MyStruct.omit("b")

ts
import { Schema } from "@effect/schema"
const MyStruct = Schema.Struct({
  a: Schema.String,
  b: Schema.Number,
  c: Schema.Boolean
})
// Schema.Struct<{ a: typeof Schema.String; c: typeof Schema.Boolean; }>
const PickedSchema = MyStruct.omit("b")

#New make Constructor for Class APIs

The introduction of a make constructor simplifies class instantiation, avoiding direct use of the new keyword and enhancing usability.

Example

ts
import { Schema } from "@effect/schema"
class MyClass extends Schema.Class<MyClass>("MyClass")({
  someField: Schema.String
}) {
  someMethod() {
    return this.someField + "bar"
  }
}
// Create an instance of MyClass using the make constructor
const instance = MyClass.make({ someField: "foo" }) // same as new MyClass({ someField: "foo" })
// Outputs to console to demonstrate that the instance is correctly created
console.log(instance instanceof MyClass) // true
console.log(instance.someField) // "foo"
console.log(instance.someMethod()) // "foobar"

ts
import { Schema } from "@effect/schema"
class MyClass extends Schema.Class<MyClass>("MyClass")({
  someField: Schema.String
}) {
  someMethod() {
    return this.someField + "bar"
  }
}
// Create an instance of MyClass using the make constructor
const instance = MyClass.make({ someField: "foo" }) // same as new MyClass({ someField: "foo" })
// Outputs to console to demonstrate that the instance is correctly created
console.log(instance instanceof MyClass) // true
console.log(instance.someField) // "foo"
console.log(instance.someMethod()) // "foobar"

#New Customizable Parsing Options at the Schema Level

This update allows setting specific parse options at the schema level (using the parseOptions annotation), ensuring precise control over parsing behaviors throughout your schemas.

Example Configuration:

ts
import { Schema } from "@effect/schema"
import { Either } from "effect"
const schema = Schema.Struct({
  a: Schema.Struct({
    b: Schema.String,
    c: Schema.String
  }).annotations({
    title: "first error only",
    parseOptions: { errors: "first" } // Only the first error in this sub-schema is reported
  }),
  d: Schema.String
}).annotations({
  title: "all errors",
  parseOptions: { errors: "all" } // All errors in the main schema are reported
})
const result = Schema.decodeUnknownEither(schema)(
  { a: {} },
  { errors: "first" }
)
if (Either.isLeft(result)) {
  console.log(result.left.message)
}
/*
all errors
├─ ["d"]
│  └─ is missing
└─ ["a"]
   └─ first error only
      └─ ["b"]
         └─ is missing
*/

ts
import { Schema } from "@effect/schema"
import { Either } from "effect"
const schema = Schema.Struct({
  a: Schema.Struct({
    b: Schema.String,
    c: Schema.String
  }).annotations({
    title: "first error only",
    parseOptions: { errors: "first" } // Only the first error in this sub-schema is reported
  }),
  d: Schema.String
}).annotations({
  title: "all errors",
  parseOptions: { errors: "all" } // All errors in the main schema are reported
})
const result = Schema.decodeUnknownEither(schema)(
  { a: {} },
  { errors: "first" }
)
if (Either.isLeft(result)) {
  console.log(result.left.message)
}
/*
all errors
├─ ["d"]
│  └─ is missing
└─ ["a"]
   └─ first error only
      └─ ["b"]
         └─ is missing
*/

Detailed Output Explanation:

In this example:

The main schema is configured to display all errors. Hence, you will see errors related to both the d field (since it's missing) and any errors from the a subschema.
The subschema (a) is set to display only the first error. Although both b and c fields are missing, only the first missing field (b) is reported.

#Schema 0.69 release

#Codemod

For some of the breaking changes, a code-mod has been released to make migration as easy as possible.

You can run it by executing:

bash
npx @effect/codemod schema-0.69 src/**/*

bash
npx @effect/codemod schema-0.69 src/**/*

It might not be perfect - if you encounter issues, let us know! Also make sure you commit any changes before running it, in case you need to revert anything.

#Enhanced API for TaggedRequest

We've improved the TaggedRequest API to make it more intuitive by grouping parameters into a single object:

Before Update:

ts
class Sample extends Schema.TaggedRequest<Sample>()(
  "Sample",
  Schema.String, // Failure Schema
  Schema.Number, // Success Schema
  { id: Schema.String, foo: Schema.Number } // Payload Schema
) {}

ts
class Sample extends Schema.TaggedRequest<Sample>()(
  "Sample",
  Schema.String, // Failure Schema
  Schema.Number, // Success Schema
  { id: Schema.String, foo: Schema.Number } // Payload Schema
) {}

After Update:

ts
class Sample extends Schema.TaggedRequest<Sample>()("Sample", {
  payload: {
    id: Schema.String,
    foo: Schema.Number
  },
  success: Schema.Number,
  failure: Schema.String
}) {}

ts
class Sample extends Schema.TaggedRequest<Sample>()("Sample", {
  payload: {
    id: Schema.String,
    foo: Schema.Number
  },
  success: Schema.Number,
  failure: Schema.String
}) {}

#Record Constructor Update

The Record constructor now consistently accepts an object argument, aligning it with similar constructors such as Map and HashMap:

Before Update:

ts
import { Schema } from "@effect/schema"
const schema = Schema.Record(Schema.String, Schema.Number)

ts
import { Schema } from "@effect/schema"
const schema = Schema.Record(Schema.String, Schema.Number)

After Update:

ts
import { Schema } from "@effect/schema"
const schema = Schema.Record({ key: Schema.String, value: Schema.Number })

ts
import { Schema } from "@effect/schema"
const schema = Schema.Record({ key: Schema.String, value: Schema.Number })

#Refinement and extend Enhancements

Support for extending Schema.String, Schema.Number, and Schema.Boolean with refinements has been added:

ts
import { Schema } from "@effect/schema"
const Integer = Schema.Int.pipe(Schema.brand("Int"))
const Positive = Schema.Positive.pipe(Schema.brand("Positive"))
// Schema.Schema<number & Brand<"Positive"> & Brand<"Int">, number, never>
const PositiveInteger = Schema.asSchema(Schema.extend(Positive, Integer))
Schema.decodeUnknownSync(PositiveInteger)(-1)
/*
throws
ParseError: Int & Brand<"Int">
└─ From side refinement failure
  └─ Positive & Brand<"Positive">
      └─ Predicate refinement failure
        └─ Expected Positive & Brand<"Positive">, actual -1
*/
Schema.decodeUnknownSync(PositiveInteger)(1.1)
/*
throws
ParseError: Int & Brand<"Int">
└─ Predicate refinement failure
  └─ Expected Int & Brand<"Int">, actual 1.1
*/

ts
import { Schema } from "@effect/schema"
const Integer = Schema.Int.pipe(Schema.brand("Int"))
const Positive = Schema.Positive.pipe(Schema.brand("Positive"))
// Schema.Schema<number & Brand<"Positive"> & Brand<"Int">, number, never>
const PositiveInteger = Schema.asSchema(Schema.extend(Positive, Integer))
Schema.decodeUnknownSync(PositiveInteger)(-1)
/*
throws
ParseError: Int & Brand<"Int">
└─ From side refinement failure
  └─ Positive & Brand<"Positive">
      └─ Predicate refinement failure
        └─ Expected Positive & Brand<"Positive">, actual -1
*/
Schema.decodeUnknownSync(PositiveInteger)(1.1)
/*
throws
ParseError: Int & Brand<"Int">
└─ Predicate refinement failure
  └─ Expected Int & Brand<"Int">, actual 1.1
*/

#Clarification in Naming Schemas

To improve clarity, we have renamed nonEmpty filter to nonEmptyString and NonEmpty schema to NonEmptyString.

#Improved Optional and Partial APIs

We've refined the optional and partial APIs by splitting them into two distinct methods: one without options (optional and partial) and another with options (optionalWith and partialWith).

This change resolves issues with previous implementations when used with the pipe method:

ts
Schema.String.pipe(Schema.optional)

ts
Schema.String.pipe(Schema.optional)

#New String Transformations

The following transformations have been added:

StringFromBase64
StringFromBase64Url
StringFromHex

#Changelog

For all the details, head over to our changelog.