CrawlingContext <UserData>
Hierarchy
Index
Properties
inheritedaddRequests
Type declaration
Parameters
requestsLike: readonly (string | ReadonlyObjectDeep<Partial<RequestOptions<Dictionary>> & { regex?: RegExp; requestsFromUrl?: string }> | ReadonlyObjectDeep<CrawleeRequest<Dictionary>>)[]
optionaloptions: ReadonlyObjectDeep<EnqueueUrlsOptions>
Options for the request queue
Returns Promise<AddRequestsBatchedResult>
inheritedgetKeyValueStore
Get a key-value store with given name or id, or the default one for the crawler.
Type declaration
Parameters
optionalidentifier: string | StorageIdentifier
Returns Promise<Pick<KeyValueStore, id | name | getValue | getAutoSavedValue | setValue | getPublicUrl>>
inheritedid
inheritedlog
A preconfigured logger for the request handler.
optionalinheritedproxyInfo
An object with information about currently used proxy by the crawler and configured by the ProxyConfiguration class.
inheritedrequest
The original Request object.
sendRequest
Fires HTTP request via the internal HTTP client, allowing to override the request options on the fly.
This is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests). Check the Skipping navigations for certain requests example for more detailed explanation of how to do that.
async requestHandler({ sendRequest }) {
const { body } = await sendRequest({
// override headers only
headers: { ... },
});
},
Type declaration
Parameters
optionalrequestOverrides: Partial<HttpRequestOptions>
optionaloptionsOverrides: SendRequestOptions
Returns Promise<Response>
inheritedsession
inheriteduseState
Returns the state - a piece of mutable persistent data shared across all the request handler runs.
Type declaration
Parameters
optionaldefaultValue: State
Returns Promise<State>
Methods
afterStorageCommit
Registers
callbackto run once the request's storage writes have been committed, or once committing them has failed - the point where a write made in the handler can finally be reacted to where it was made. Two things belong here: side effects that must agree with what actually landed (result counters, progress reporting), and handling of a write that did not land at all.async requestHandler({ pushData, afterStorageCommit, useState }) {const state = await useState({ itemCount: 0 });await pushData(item);afterStorageCommit((error) => {if (error) {throw new NonRetryableError('The page is too big to store', { cause: error });}state.itemCount++;});}useState()is deliberately not transactional, which is what makes the counter case necessary: incrementing in the handler would keep the increment for a request that pushes an item and then fails, while the item itself is rolled back.Callbacks run in registration order, after the commit and before the request is marked as handled. The first one to throw propagates and the rest do not run; on a failed commit its error replaces the commit error, so a callback can decide what the request fails with. After a successful commit, a callback that throws fails the request without a retry whatever it throws - the writes are already durable and retrying would duplicate them, so anything that is not already a NonRetryableError is wrapped in an AfterCommitError.
Callbacks are not run when the request handler itself fails - nothing was written, and
errorHandler/failedRequestHandlercover that.Throws when storage is not transactional (
transactionalStorage: false) - writes are then applied as they are made and throw at the call site on their own. In AdaptivePlaywrightCrawler the callback belongs to the current request handler attempt, so it runs only for the attempt whose writes are the ones being committed.Parameters
callback: (error) => Awaitable<void>
Returns void
extendTimeout
Gives the current request
secsmore seconds to finish, for when how long it needs is only apparent once it is already running - a listing page that turns out to have far more to scroll through than usual, say. PreferrequestHandlerTimeoutSecs, or a per-route override viarouter.addHandler, whenever the time needed is known up front.router.addHandler('LIST', async ({ extendTimeout, page }) => {const pageCount = await countPages(page);extendTimeout(pageCount * 10);await scrapeAllPages(page);});Extends the request handler's own timeout and the crawler's internal one together, so the extension is not immediately undone by the latter. Calling it from a handler that has already timed out does nothing.
Parameters
secs: number
Returns void
inheritedpushData
This function allows you to push data to a Dataset specified by name, or the one currently used by the crawler.
Shortcut for
crawler.pushData().Parameters
optionaldata: ReadonlyDeep<Dictionary | Dictionary[]>
Data to be pushed to the default dataset.
optionaldatasetIdentifier: string | StorageIdentifier
Returns Promise<void>
registerDeferredCleanup
Register a function to be called at the very end of the request handling process. This is useful for resources that should be accessible to error handlers, for instance.
The callback runs outside the request's storage transaction, so storage writes made here are applied immediately and are not rolled back when the request fails. In AdaptivePlaywrightCrawler it also runs once per request handler attempt, so a write here can land more than once for a single request. Push results from the request handler itself.
Parameters
cleanup: () => Promise<unknown>
Returns void
Add requests directly to the request queue currently used by the crawler.
Optionally, the function allows you to filter the target URLs using an array of glob or regexp patterns, the same way
enqueueLinksdoes for extracted links.