Skip to main content
Version: Next

BasicCrawlingContext <UserData>

Hierarchy

Index

Properties

addRequests

addRequests: (requestsLike: readonly (string | ReadonlyObjectDeep<Partial<RequestOptions<Dictionary>> & { regex?: RegExp; requestsFromUrl?: string }> | ReadonlyObjectDeep<Request<Dictionary>>)[], options?: ReadonlyObjectDeep<RequestQueueOperationOptions>) => Promise<void>

Add requests directly to the request queue.

@param

The requests to add

@param

Options for the request queue


Type declaration

    • (requestsLike: readonly (string | ReadonlyObjectDeep<Partial<RequestOptions<Dictionary>> & { regex?: RegExp; requestsFromUrl?: string }> | ReadonlyObjectDeep<Request<Dictionary>>)[], options?: ReadonlyObjectDeep<RequestQueueOperationOptions>): Promise<void>
    • Add requests directly to the request queue.


      Parameters

      • requestsLike: readonly (string | ReadonlyObjectDeep<Partial<RequestOptions<Dictionary>> & { regex?: RegExp; requestsFromUrl?: string }> | ReadonlyObjectDeep<Request<Dictionary>>)[]
      • optionaloptions: ReadonlyObjectDeep<RequestQueueOperationOptions>

        Options for the request queue

      Returns Promise<void>

crawler

crawler: BasicCrawler<BasicCrawlingContext<Dictionary>>

getKeyValueStore

getKeyValueStore: (idOrName?: string) => Promise<KeyValueStore>

Get a key-value store with given name or id, or the default one for the crawler.


Type declaration

    • Get a key-value store with given name or id, or the default one for the crawler.


      Parameters

      • optionalidOrName: string

      Returns Promise<KeyValueStore>

id

id: string

log

log: Log

A preconfigured logger for the request handler.

optionalproxyInfo

proxyInfo?: ProxyInfo

An object with information about currently used proxy by the crawler and configured by the ProxyConfiguration class.

request

request: Request<UserData>

The original Request object.

optionalsession

session?: Session

useState

useState: <State>(defaultValue?: State) => Promise<State>

Returns the state - a piece of mutable persistent data shared across all the request handler runs.


Type declaration

    • <State>(defaultValue?: State): Promise<State>
    • Returns the state - a piece of mutable persistent data shared across all the request handler runs.


      Type parameters

      • State: Dictionary = Dictionary

      Parameters

      • optionaldefaultValue: State

      Returns Promise<State>

Methods

enqueueLinks

  • This function automatically finds and enqueues links from the current page, adding them to the RequestQueue currently used by the crawler.

    Optionally, the function allows you to filter the target links' URLs using an array of globs or regular expressions and override settings of the enqueued Request objects.

    Check out the Crawl a website with relative links example for more details regarding its usage.

    Example usage

    async requestHandler({ enqueueLinks }) {
    await enqueueLinks({
    urls: [...],
    });
    },

    Parameters

    • optionaloptions: { baseUrl?: string; exclude?: readonly (GlobInput | RegExpInput)[]; forefront?: boolean; globs?: readonly GlobInput[]; label?: string; limit?: number; pseudoUrls?: readonly PseudoUrlInput[]; regexps?: readonly RegExpInput[]; requestQueue?: RequestProvider; selector?: string; skipNavigation?: boolean; strategy?: EnqueueStrategy | all | same-domain | same-hostname | same-origin; transformRequestFunction?: RequestTransform; urls: readonly string[]; userData?: Dictionary }

      All enqueueLinks() parameters are passed via an options object.

    Returns Promise<BatchAddRequestsResult>

    Promise that resolves to BatchAddRequestsResult object.

pushData

  • pushData(data?: ReadonlyDeep<Dictionary | Dictionary[]>, datasetIdOrName?: string): Promise<void>
  • This function allows you to push data to a Dataset specified by name, or the one currently used by the crawler.

    Shortcut for crawler.pushData().


    Parameters

    • optionaldata: ReadonlyDeep<Dictionary | Dictionary[]>

      Data to be pushed to the default dataset.

    • optionaldatasetIdOrName: string

    Returns Promise<void>

sendRequest

  • sendRequest<Response>(overrideOptions?: Partial<OptionsInit>): Promise<Response<Response>>
  • Fires HTTP request via got-scraping, allowing to override the request options on the fly.

    This is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests). Check the Skipping navigations for certain requests example for more detailed explanation of how to do that.

    async requestHandler({ sendRequest }) {
    const { body } = await sendRequest({
    // override headers only
    headers: { ... },
    });
    },

    Type parameters

    • Response = string

    Parameters

    • optionaloverrideOptions: Partial<OptionsInit>

    Returns Promise<Response<Response>>