Zip

Details

A trailing / is added to a non-empty path when missing, as zip tools expect for directories.

declare function directory(path: string, options?: EntryOptions): ContentEntry

Details

String content is encoded as UTF-8, and size is the byte length of the content. Because the content is known up front, its sizes and CRC-32 are written in the local header, which every zip reader accepts.

See

• fileStream for content that is streamed instead of held in memory
• encode for writing entries to an archive

declare function file(path: string, content: string | Uint8Array<ArrayBufferLike>, options?: FileOptions): ContentEntry

When to use

Use when the content is too large to hold in memory, or is read from a file or another stream.

Details

body is not run until the entry is encoded. Its sizes and CRC-32 are written in a data descriptor after the data.

Gotchas

• A stored entry without size fails with an InvalidHeader ZipError.
• When size is set, encode fails with a SizeMismatch ZipError if the body produces fewer or more bytes.
• Non-empty stored entries are followed by a data descriptor, which some streaming readers mishandle. Java's ZipInputStream rejects them, and libarchive reading from a pipe (as bsdtar does) ends them at the first bytes that look like a descriptor, so content can cut the entry short and add entries. Use Deflate or an in-memory file for untrusted content and for those readers.

See

• file for in-memory content
• encode for writing entries to an archive

declare function fileStream<E = never, R = never>(path: string, body: Stream<Uint8Array<ArrayBufferLike>, E, R>, options?: FileStreamOptions): Entry<E, R>

Details

Entries are read in archive order from their local headers, so archives can be decoded as they arrive, for example from an HTTP response. Each output chunk holds one entry, and each body streams the uncompressed bytes of the entry. Pulling the next entry skips whatever is left of the current body. Stored and deflated entries are read, including ZIP64 sizes and deflated entries whose sizes follow their data in a data descriptor. Every body is checked against its CRC-32 and sizes. Reading stops at the central directory.

Paths use the UTF-8 flag when set, and otherwise the first Info-ZIP Unicode path field when it matches the name, as Info-ZIP and 7-Zip do; other paths are read as UTF-8 when valid and as CP437 otherwise. Times are read from the extended timestamp field when present, and otherwise from the DOS time and date, which have no time zone and are read as UTC.

Gotchas

• Read each body before pulling the next entry: a body read afterwards fails with a BodyConsumed ZipError. Operators that pull ahead, such as Stream.buffer, therefore can't sit between decode and the body reads.
• Read bodies while the entry stream runs, for example in Stream.mapEffect: once it has ended, as after Stream.runCollect, Stream.runHead or Stream.take, every body fails with a BodyConsumed ZipError.
• A body reads the entry's bytes once: running it again continues where the previous run stopped.
• The central directory is not read, so mode, comments and entries that only the central directory lists are not reported. Tools that read the central directory can see other entries and paths, for example in archives updated by appending.
• Archives that do not start with an entry or a central directory, such as self-extracting archives, fail with an InvalidHeader ZipError.
• Stored entries whose sizes only follow their data fail with an Unsupported ZipError. Python's zipfile writes them for stored entries on unseekable output, and Info-ZIP's zip for empty files when writing to a pipe. Searching for their end would let the content of an entry forge the entries after it.
• A body that grows past its size fails with a SizeMismatch ZipError before the extra bytes are emitted, so size can be checked before reading.
• Deflated data can expand about a thousand times, and pulling the next entry inflates the rest of the current body to find its end. To bound the work on untrusted archives, check size, count the bytes of bodies without one, and end the stream at an entry that is too large rather than skip it.
• Every ZipError ends the archive: later entry and body reads fail with the same error.
• The channel completes only when its input ends: input after the central directory is read and discarded, so that the upstream done value can be passed on.
• Paths are returned as written in the archive: they can be absolute, contain .., drive letters, control characters or backslashes, which some tools read as separators, and can repeat. Validate them before writing entries to disk; encode rejects absolute paths, drive letters and backslashes. Paths with a NUL character, which other tools take as the end of the name, fail with an InvalidHeader ZipError.

See

• encode for the inverse operation

declare function decode<IE = never, Done = unknown>(): Channel<readonly [Entry<ZipError | IE, never>, Entry<ZipError | IE, never>], ZipError | IE, Done, readonly [Uint8Array<ArrayBufferLike>, Uint8Array<ArrayBufferLike>], IE, Done>

Details

Each entry is written as soon as it arrives: a local header, the stored or deflated body and, for entries with a streamed body, a data descriptor with the CRC-32 and sizes. A ContentEntry has them in its local header. The central directory is written once the input ends. Paths are written as UTF-8, with the UTF-8 flag set when they are not ASCII. mtime is written both as a DOS time in UTC, which keeps two-second resolution and covers 1980 to 2107, and as an extended timestamp in whole seconds when it falls between 1970 and 2038. ZIP64 fields are added when a size, an offset or the entry count does not fit the classic fields, and to the local header of a deflated entry streamed without a size or with one near 4 GiB, so that readers know the width of the sizes after its data.

Headers and body chunks below 16 KiB are gathered into output chunks of about 64 KiB, and larger body chunks pass through without copying. Output is handed out before waiting for more entries, but up to 64 KiB of it can be held while a body is pulled. Each body chunk of a streamed Deflate entry is a separate compressor job, so larger chunks compress faster. A Deflate body with a size of at most 64 KiB is instead collected and compressed in one job, and still gets a data descriptor; omit size to stream it.

Gotchas

• Deflated entries need CompressionStream("deflate-raw"). On a runtime without it, encoding a deflated entry dies with the runtime's error, as it does when the compressor fails: check for the API or use Store.
• Deflated bytes come from the runtime's CompressionStream, so they can differ between runtimes. Archives of stored entries are byte-identical everywhere.
• A body that produces a byte count other than a set header.size, or any bytes for a directory, fails with a SizeMismatch ZipError.
• Directories are always stored, whatever their method.
• Empty paths, paths starting with / or a drive letter such as C:, paths containing \ or NUL, File paths ending with /, Directory paths not ending with /, and invalid modes, sizes or times fail with an InvalidHeader ZipError. Other paths are written as given, including .. segments, so validate untrusted paths before encoding them.

See

• decode for the inverse operation
• file for building entries from in-memory content
• fileStream for building entries from streamed content

declare function encode<IE = never, Done = unknown, In extends ContentEntry | Entry<any, any> = ContentEntry | Entry<never, never>>(): Channel<readonly [Uint8Array<ArrayBufferLike>, Uint8Array<ArrayBufferLike>], ZipError | IE | In extends Entry<E, any> ? E : never, Done, readonly [In, In], IE, Done, In extends Entry<any, R> ? R : never>

import { Effect, Stream } from "effect"
import { Zip } from "effect/encoding"

const archive = Stream.make(
  Zip.directory("docs"),
  Zip.file("docs/readme.txt", "Hello, zip!")
).pipe(Stream.pipeThroughChannel(Zip.encode()))

const listing = archive.pipe(
  Stream.pipeThroughChannel(Zip.decode()),
  Stream.mapEffect((entry) =>
    Stream.mkString(Stream.decodeText(entry.body)).pipe(
      Effect.map((text) => `${entry.header.path} ${JSON.stringify(text)}`)
    )
  ),
  Stream.runCollect
)
await Effect.runPromise(listing) // => ['docs/ ""', 'docs/readme.txt "Hello, zip!"']

Details

The kind field identifies the failure:

• InvalidHeader: a header has an unknown signature or a malformed field, or an entry has a value that cannot be written.
• InvalidData: the compressed data of an entry is corrupt.
• Truncated: the input ended inside an entry or before the central directory.
• Checksum: the data of an entry does not match its CRC-32.
• SizeMismatch: an entry body produced a byte count other than its declared size, or the data of an entry does not match the sizes in the archive.
• Unsupported: the archive uses a feature this module does not read, such as encryption or a compression method other than store and deflate.
• BodyConsumed: an entry body was read after the decoder had moved past it.

declare class ZipError extends YieldableError<this> & {
  readonly _tag: "ZipError";
} & Readonly<{
  readonly kind: "InvalidData" | "InvalidHeader" | "Truncated" | "SizeMismatch" | "Unsupported" | "BodyConsumed" | "Checksum";
  readonly message: string;
}> {
  constructor(args: {
    readonly kind: "InvalidData" | "InvalidHeader" | "Truncated" | "SizeMismatch" | "Unsupported" | "BodyConsumed" | "Checksum";
    readonly message: string;
  });
  readonly "~effect/encoding/Zip/ZipError": "~effect/encoding/Zip/ZipError";
}

Details

file and directory build these. Because the content is known before the entry is written, encode puts its sizes and CRC-32 in the local header and writes no data descriptor, which every zip reader accepts. content and body exclude each other, so an entry whose body is replaced must drop content.

See

• Entry for entries whose content is streamed

interface ContentEntry {
  readonly body?: undefined;
  readonly content: Uint8Array;
  readonly header: Header;
}

Details

decode returns these, and fileStream builds them. encode writes the sizes and CRC-32 of a streamed body in a data descriptor after the data.

See

• ContentEntry for entries whose content is held in memory

interface Entry<E = never, R = never> {
  readonly body: Stream<Uint8Array<ArrayBufferLike>, E, R>;
  readonly content?: undefined;
  readonly header: Header;
}

Details

Unset fields default to an mtime of 1980-01-01T00:00:00Z, the earliest time a zip header can hold, and mode 0o644 (0o755 for directories). The defaults make archives reproducible.

interface EntryOptions {
  readonly mode?: number;
  readonly mtime?: Date;
}

Details

method defaults to "Deflate". Use "Store" for content that is already compressed, such as images or archives.

interface FileOptions extends EntryOptions {
  readonly method?: Method;
}

Details

size is the exact byte length of the body. It is required for Store, because streaming readers can only find the end of stored data from its size, and optional for Deflate, where it is checked against the body.

type FileStreamOptions = EntryOptions & {
  readonly method?: "Deflate";
  readonly size?: number;
} | {
  readonly method: "Store";
  readonly size: number;
}

Details

path is the archive path, with / separators; directory paths end with /. size is the uncompressed byte length when it is known before the body: decoded entries whose sizes follow their data have no size. mode holds Unix permission bits and is written to the central directory, which decode does not read, so decoded headers have no mode. mtime has a resolution of one second when the archive has an extended timestamp and two seconds otherwise.

interface Header {
  readonly method: Method;
  readonly mode?: number;
  readonly mtime: Date;
  readonly path: string;
  readonly size?: number;
  readonly type: EntryType;
}