BrowserCrawlingContext <Page, Response, UserData, GoToOptions>
Hierarchy
- CrawlingContext<UserData>
- BrowserCrawlingContext
Index
Properties
inheritedaddRequests
Type declaration
Parameters
requestsLike: readonly (string | ReadonlyObjectDeep<Partial<RequestOptions<Dictionary>> & { regex?: RegExp; requestsFromUrl?: string }> | ReadonlyObjectDeep<CrawleeRequest<Dictionary>>)[]
optionaloptions: ReadonlyObjectDeep<EnqueueUrlsOptions>
Options for the request queue
Returns Promise<AddRequestsBatchedResult>
enqueueLinks
Helper function for extracting URLs from the current page and adding them to the request queue.
Type declaration
Parameters
optionaloptions: EnqueueLinksOptions
Returns Promise<AddRequestsBatchedResult>
extractLinks
Extracts URLs from the current page, without adding them to the request queue.
Type declaration
Parameters
optionaloptions: ExtractLinksOptions
Returns Promise<string[]>
inheritedgetKeyValueStore
Get a key-value store with given name or id, or the default one for the crawler.
Type declaration
Parameters
optionalidentifier: string | StorageIdentifier
Returns Promise<Pick<KeyValueStore, id | name | getValue | getAutoSavedValue | setValue | getPublicUrl>>
gotoOptions
Options object passed to the underlying page.goto() call. preNavigationHooks can mutate this
object (or return { gotoOptions: ... }) to influence the navigation.
inheritedid
inheritedlog
A preconfigured logger for the request handler.
page
The browser page object where the web page is loaded and rendered.
optionalinheritedproxyInfo
An object with information about currently used proxy by the crawler and configured by the ProxyConfiguration class.
request
The request object that was successfully loaded and navigated to, including the loadedUrl property.
response
The HTTP response object returned by the browser's navigation.
inheritedsendRequest
Fires HTTP request via the internal HTTP client, allowing to override the request options on the fly.
This is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests). Check the Skipping navigations for certain requests example for more detailed explanation of how to do that.
async requestHandler({ sendRequest }) {
const { body } = await sendRequest({
// override headers only
headers: { ... },
});
},
Type declaration
Parameters
optionalrequestOverrides: Partial<HttpRequestOptions>
optionaloptionsOverrides: SendRequestOptions
Returns Promise<Response>
inheritedsession
inheriteduseState
Returns the state - a piece of mutable persistent data shared across all the request handler runs.
Type declaration
Parameters
optionaldefaultValue: State
Returns Promise<State>
Methods
inheritedextendTimeout
Gives the current request
secsmore seconds to finish, for when how long it needs is only apparent once it is already running - a listing page that turns out to have far more to scroll through than usual, say. PreferrequestHandlerTimeoutSecs, or a per-route override viarouter.addHandler, whenever the time needed is known up front.router.addHandler('LIST', async ({ extendTimeout, page }) => {const pageCount = await countPages(page);extendTimeout(pageCount * 10);await scrapeAllPages(page);});Extends the request handler's own timeout and the crawler's internal one together, so the extension is not immediately undone by the latter. Calling it from a handler that has already timed out does nothing.
Parameters
secs: number
Returns void
inheritedpushData
This function allows you to push data to a Dataset specified by name, or the one currently used by the crawler.
Shortcut for
crawler.pushData().Parameters
optionaldata: ReadonlyDeep<Dictionary | Dictionary[]>
Data to be pushed to the default dataset.
optionaldatasetIdentifier: string | StorageIdentifier
Returns Promise<void>
inheritedregisterDeferredCleanup
Register a function to be called at the very end of the request handling process. This is useful for resources that should be accessible to error handlers, for instance.
The callback runs outside the request's storage transaction, so storage writes made here are applied immediately and are not rolled back when the request fails. In AdaptivePlaywrightCrawler it also runs once per request handler attempt, so a write here can land more than once for a single request. Push results from the request handler itself.
Parameters
cleanup: () => Promise<unknown>
Returns void
Add requests directly to the request queue currently used by the crawler.
Optionally, the function allows you to filter the target URLs using an array of glob or regexp patterns, the same way
enqueueLinksdoes for extracted links.