RestrictedCrawlingContext <UserData>

Hierarchy

• RestrictedCrawlingContext
  • CrawlingContext

Index

Properties

• addRequests
• getKeyValueStore
• id
• log
• proxyInfo
• request
• session
• useState

Methods

• pushData

Properties

addRequests

addRequests: (requestsLike, options) => Promise<AddRequestsBatchedResult>

Add requests directly to the request queue currently used by the crawler.

Optionally, the function allows you to filter the target URLs using an array of glob or regexp patterns, the same way enqueueLinks does for extracted links.

---

Type declaration

• • (requestsLike, options): Promise<AddRequestsBatchedResult>

  • Parameters

    • requestsLike: readonly (string | ReadonlyObjectDeep<Partial<RequestOptions<Dictionary>> & { regex?: RegExp; requestsFromUrl?: string }> | ReadonlyObjectDeep<CrawleeRequest<Dictionary>>)[]

    • optionaloptions: ReadonlyObjectDeep<EnqueueUrlsOptions>

      Options for the request queue

    Returns Promise<AddRequestsBatchedResult>

getKeyValueStore

getKeyValueStore: (identifier) => Promise<Pick<KeyValueStore, id | name | getValue | getAutoSavedValue | setValue | getPublicUrl>>

Get a key-value store with given name or id, or the default one for the crawler.

---

Type declaration

• • (identifier): Promise<Pick<KeyValueStore, id | name | getValue | getAutoSavedValue | setValue | getPublicUrl>>

  • Parameters

    • optionalidentifier: string | StorageIdentifier

    Returns Promise<Pick<KeyValueStore, id | name | getValue | getAutoSavedValue | setValue | getPublicUrl>>

id

id: string

log

log: CrawleeLogger

A preconfigured logger for the request handler.

optionalproxyInfo

proxyInfo?: ProxyInfo

An object with information about currently used proxy by the crawler and configured by the ProxyConfiguration class.

request

request: CrawleeRequest<UserData>

The original Request object.

session

session: ISession

useState

useState: <State>(defaultValue) => Promise<State>

Returns the state - a piece of mutable persistent data shared across all the request handler runs.

---

Type declaration

• • <State>(defaultValue): Promise<State>

  • Parameters

    • optionaldefaultValue: State

    Returns Promise<State>

Methods

pushData

• pushData(data, datasetIdentifier): Promise<void>

• This function allows you to push data to a Dataset specified by name, or the one currently used by the crawler.

  Shortcut for crawler.pushData().

  ---

  Parameters

  • optionaldata: ReadonlyDeep<Dictionary | Dictionary[]>

    Data to be pushed to the default dataset.

  • optionaldatasetIdentifier: string | StorageIdentifier

  Returns Promise<void>