Skip to main content
Version: Next

StorageTransaction

A storage transaction scoped to a request's lifecycle. Writes made through the storage frontends (Dataset, KeyValueStore, RequestQueue) while the transaction is active are recorded rather than applied; on commit() they are replayed into real storage, on rollback() they are dropped. Reads consult the recorded writes first, so a handler sees its own writes.

Create one with createStorageTransaction (explicit commit/rollback) or withStorageTransaction (scoped sugar). Crawlers open one automatically around every request handler unless transactionalStorage: false is set.

Implements

Index

Properties

Accessors

Methods

Properties

readonlyjournal

journal: JournalEntry[] = []

The ordered, append-only journal — the source of truth for commit, introspection and reads.

readonlypolicy

policy: StorageWritePolicy

Per-storage-type write policy.

Accessors

datasetItems

  • get datasetItems(): { datasetId: string; item: Dictionary }[]

  • Items pushed to datasets during the transaction, in push order.

    Returns { datasetId: string; item: Dictionary }[]

enqueuedUrls

  • get enqueuedUrls(): { label?: string; url: string }[]

  • URLs enqueued to request queues during the transaction, under either write policy. Recorded as requested, so duplicate, already-present and backend-rejected URLs are included.

    One gap: unless a caller of addRequestsBatched() waits for every chunk (waitForAllRequestsToBeAdded or maxNewRequests, both of which enqueueLinks sets when a crawl limit applies), the chunks after the first are added by a background writer that outlives the transaction and is not recorded here.

    Returns { label?: string; url: string }[]

isActive

  • get isActive(): boolean

  • true only while state === 'open'. This is the single predicate every storage operation consults — operations performed after the transaction is closed pass through to the real backend.

    Returns boolean

keyValueStoreChanges

  • get keyValueStoreChanges(): Record<string, Record<string, { changedValue: unknown; options?: RecordOptions }>>

  • Key-value store changes made during the transaction, keyed by store id, last write per key.

    Returns Record<string, Record<string, { changedValue: unknown; options?: RecordOptions }>>

state

Methods

commit

  • commit(): Promise<void>

  • Replays the journaled writes into real storage. A no-op unless the transaction is open.

    The transaction transitions to committing before anything is flushed, so a commit that throws partway lands in failed (never back in open) and subsequent storage operations pass through rather than recording into a dead transaction. Delivery is at-least-once — a commit that fails partway may have applied some of the writes already.

    Returns Promise<void>

dispose

  • dispose(): void

  • Releases the journal and the write-time snapshots it holds. Must be called for every terminal state, failed included. Idempotent, never throws, and does not change state. Any StorageTransactionView of this transaction is only valid until this is called.

    Returns void

rollback

  • rollback(): void

  • Discards the journaled writes. A no-op unless the transaction is open — in particular, calling it after a successful commit() (which the crawler's error handling can legitimately do) does nothing and never throws.

    Returns void

run

  • run<T>(callback): Promise<T>

  • Runs callback with this transaction installed in the async context.

    Parameters

    • callback: () => Awaitable<T>

      Returns Promise<T>

    Page Options