Xml

Details

• maxTokenSize: the maximum length of a single token in UTF-16 code units, as written in the input: a tag with its attributes, a comment, a processing instruction, a DOCTYPE, or a text node including its references and CDATA sections. Defaults to 10 MiB; Infinity disables the limit.
• maxDepth: the maximum number of nested elements. Defaults to 1024; Infinity disables the limit.

Memory per token and for open element names is bounded by maxTokenSize. Invalid options throw when the channel is created.

interface DecodeOptions {
  readonly maxDepth?: number;
  readonly maxTokenSize?: number;
}

Details

maxElementSize is the maximum size of one collected element, counted as the UTF-16 code units of its names, attribute names and values, and text. Defaults to 16 MiB; Infinity disables the limit. Invalid options throw when the channel is created.

interface ElementsOptions {
  readonly maxElementSize?: number;
}

Details

Works like decodeString after decoding the bytes as UTF-8. Multi-byte characters can be split between chunks, and a leading byte order mark is skipped.

Gotchas

• Invalid or truncated UTF-8 fails with a Syntax XmlError.
• An initial UTF-16 byte order mark or an XML declaration naming an encoding other than UTF-8 or US-ASCII fails with an Unsupported XmlError. Decode such bytes with a TextDecoder for that encoding and use decodeString.

See

• decodeString for text input and the shared gotchas

declare function decode<IE = never, Done = unknown>(options?: DecodeOptions): Channel<readonly [Event, Event], XmlError | IE, Done, readonly [Uint8Array<ArrayBufferLike>, Uint8Array<ArrayBufferLike>], IE, Done>

Details

• Tokens can be split anywhere between chunks. Each output chunk holds the events completed by one input chunk.
• The document must have one root element. Whitespace outside it is not reported.
• A leading byte order mark is skipped, and the encoding named by the XML declaration is ignored.
• Events completed before an error are emitted before the channel fails.

Gotchas

• End of input inside the root fails with a Syntax error ("unclosed element"); interruption ends the stream without this check.
• A DOCTYPE is reported as a Doctype event, and its internal subset is skipped without checking its declarations. They are not applied, so default attribute values are not added, and parameter entities are not expanded.
• References to entities other than <, >, &, ' and " fail with an Unsupported XmlError when the document has a DOCTYPE, and with a Syntax XmlError otherwise. Character references such as é are always decoded.
• Names are reported as written, including namespace prefixes, and xmlns declarations are ordinary attributes.
• Tokens or aggregate open element names longer than maxTokenSize, and elements nested deeper than maxDepth, fail with a Limit XmlError. A long token can instead fail with an error found in its first maxTokenSize + 1 characters.
• Strings in events can share memory with their input chunk, so keeping a short string, such as an id from each record, can keep the whole chunk in memory. Copy strings that are kept long, for example with structuredClone.

See

• decode for UTF-8 byte input
• elements for collecting repeated elements into trees
• encodeString for the inverse operation

declare function decodeString<IE = never, Done = unknown>(options?: DecodeOptions): Channel<readonly [Event, Event], XmlError | IE, Done, readonly [string, string], IE, Done>

import { Effect, Stream } from "effect"
import { Xml } from "effect/encoding"

const events = Stream.make("<note id=\"1\">Hi &amp; bye<br/></note>").pipe(
  Stream.pipeThroughChannel(Xml.decodeString()),
  Stream.map((event) => event._tag),
  Stream.runCollect
)
await Effect.runPromise(events) // => ["StartElement", "Text", "StartElement", "EndElement", "EndElement"]

When to use

Use to process large documents such as feeds, sitemaps or exports one record at a time, without holding the whole document in memory.

Details

path lists element names from the root, and "*" matches any name. Each element whose ancestors and own name match the path is emitted once its end tag arrives, with all of its descendants; everything outside the matched elements is skipped. Because paths start at the root, an element nested in a matched element is part of its tree and is not emitted separately.

Gotchas

• A path that does not start at the root element matches nothing.
• Whitespace between child elements, such as indentation, is kept as string children.
• A collected element larger than maxElementSize fails with a Limit XmlError. The size counts characters, not memory: an element with many small children can take many times its size in memory, so use a lower limit for untrusted input.
• Events that end inside an element, or end an element that was never started, fail with an InvalidEvent XmlError.
• A path that is empty or contains an invalid name throws when the channel is created.

See

• decode for producing events from bytes

declare function elements<IE = never, Done = unknown>(path: readonly Array<string>, options?: ElementsOptions): Channel<readonly [Element, Element], XmlError | IE, Done, readonly [Event, Event], IE, Done>

import { Effect, Stream } from "effect"
import { Xml } from "effect/encoding"

const feed = `<rss><channel><title>News</title>
<item><title>First</title></item>
<item><title>Second</title></item>
</channel></rss>`

const items = await Effect.runPromise(
  Stream.make(feed).pipe(
    Stream.pipeThroughChannel(Xml.decodeString()),
    Stream.pipeThroughChannel(Xml.elements(["rss", "channel", "item"])),
    Stream.runCollect
  )
)
items.length // => 2
items[1] // => { name: "item", attributes: {}, children: [{ name: "title", attributes: {}, children: ["Second"] }] }

Details

Works like encodeString, with one byte array per input chunk.

Gotchas

A Declaration event naming an encoding other than UTF-8 fails with an InvalidEvent XmlError.

See

• encodeString for text output and the shared gotchas

declare function encode<IE = never, Done = unknown>(): Channel<readonly [Uint8Array<ArrayBufferLike>, Uint8Array<ArrayBufferLike>], XmlError | IE, Done, readonly [Event, Event], IE, Done>

Details

• Each input chunk is written as one string.
• Text escapes &, < and >. Attribute values are double-quoted and escape &, <, ", tabs and line breaks. Carriage returns are written as 
 so that they survive decoding.
• Start tags are written completely on their event. Empty elements use explicit closing tags, such as <br></br>, never self-closing tags.
• Nothing is added between events, so whitespace comes only from Text events.

Gotchas

Events that do not form one well-formed document fail with an InvalidEvent XmlError, such as an end tag that does not match the open element, a second root element, non-whitespace text outside the root element, invalid names or characters, -- in a comment, ?> in processing instruction data, or a declaration that is not the first event. The stream must end with the root element closed.

See

• encode for UTF-8 byte output
• decodeString for the inverse operation

declare function encodeString<IE = never, Done = unknown>(): Channel<readonly [string, string], XmlError | IE, Done, readonly [Event, Event], IE, Done>

import { Effect, Stream } from "effect"
import { Xml } from "effect/encoding"

const events: ReadonlyArray<Xml.Event> = [
  { _tag: "StartElement", name: "a", attributes: { href: "x?a=1&b=2" } },
  { _tag: "Text", text: "1 < 2" },
  { _tag: "EndElement", name: "a" }
]

const xml = Stream.fromIterable(events).pipe(
  Stream.pipeThroughChannel(Xml.encodeString()),
  Stream.mkString
)
await Effect.runPromise(xml) // => '<a href="x?a=1&amp;b=2">1 &lt; 2</a>'

Details

The kind field identifies the failure:

• Syntax: the input is not well-formed XML or not valid UTF-8.
• Unsupported: the input uses a feature this module does not process, such as an entity declared in a DOCTYPE or a byte encoding other than UTF-8.
• Limit: the input exceeds maxTokenSize, maxDepth or maxElementSize.
• InvalidEvent: the events do not form a well-formed document, such as an end tag that does not match its start tag.

Errors from decode and decodeString carry the one-based line and column where the problem was found, also appended to message. Columns count UTF-16 code units, and lines end at LF, CR or CRLF.

declare class XmlError extends YieldableError<this> & {
  readonly _tag: "XmlError";
} & Readonly<{
  readonly column?: number;
  readonly kind: "Syntax" | "Limit" | "Unsupported" | "InvalidEvent";
  readonly line?: number;
  readonly message: string;
}> {
  constructor(args: {
    readonly column?: number;
    readonly kind: "Syntax" | "Limit" | "Unsupported" | "InvalidEvent";
    readonly line?: number;
    readonly message: string;
  });
  readonly "~effect/encoding/Xml/XmlError": "~effect/encoding/Xml/XmlError";
}

Details

encoding and standalone are present only when the declaration sets them.

interface Declaration {
  readonly _tag: "Declaration";
  readonly encoding?: string;
  readonly standalone?: boolean;
  readonly version: string;
}

Details

publicId and systemId are present only when the declaration names an external DTD; the DTD is never read. Whitespace in publicId is collapsed to single spaces and trimmed. The internal subset is not reported.

interface Doctype {
  readonly _tag: "Doctype";
  readonly name: string;
  readonly publicId?: string;
  readonly systemId?: string;
}

Details

Children are elements and strings of character data, in document order. Adjacent text is merged into one string; comments and processing instructions are dropped.

interface Element {
  readonly attributes: Readonly<Record<string, string>>;
  readonly children: readonly Array<string | Element>;
  readonly name: string;
}

Details

data is the text after the target, without leading whitespace.

interface ProcessingInstruction {
  readonly _tag: "ProcessingInstruction";
  readonly data: string;
  readonly target: string;
}

Details

name is the qualified name as written and is the only element-name field the encoder reads.

Attribute values have their references replaced and whitespace normalized as XML requires. Decoded attributes inherit nothing, so only the attributes of the tag are present and __proto__ is an ordinary key. Their key order is document order, as XML names are never integer-like. A structuredClone of a decoded map gets Object.prototype back. A self-closing tag such as <br/> is decoded as a StartElement followed by an EndElement.

interface StartElement {
  readonly _tag: "StartElement";
  readonly attributes: Readonly<Record<string, string>>;
  readonly name: string;
}

Details

Adjacent text, references and CDATA sections are decoded into one Text event that ends at the next tag, comment or processing instruction, however the input is chunked. Line endings are normalized to \n.

interface Text {
  readonly _tag: "Text";
  readonly text: string;
}