CheerioCrawlingContext <UserData, JSONData>
Hierarchy
- DOMCrawlingContext<CheerioParseResult, UserData, JSONData>
- CheerioCrawlingContext
Index
Properties
inherited$
inheritedaddRequests
Type declaration
Parameters
requestsLike: readonly (string | ReadonlyObjectDeep<Partial<RequestOptions<Dictionary>> & { regex?: RegExp; requestsFromUrl?: string }> | ReadonlyObjectDeep<CrawleeRequest<Dictionary>>)[]
optionaloptions: ReadonlyObjectDeep<EnqueueUrlsOptions>
Options for the request queue
Returns Promise<AddRequestsBatchedResult>
inheritedbody
The request body of the web page.
The type depends on the Content-Type header of the web page:
- String for
text/html,application/xhtml+xml,application/xmlMIME content types - Buffer for others MIME content types
inheritedcontentType
Parsed Content-Type header: { type, encoding }.
Type declaration
encoding: BufferEncoding
type: string
inheritedgetKeyValueStore
Get a key-value store with given name or id, or the default one for the crawler.
Type declaration
Parameters
optionalidentifier: string | StorageIdentifier
Returns Promise<Pick<KeyValueStore, id | name | getValue | getAutoSavedValue | setValue | getPublicUrl>>
inheritedid
inheritedjson
The parsed object from JSON string if the response contains the content type application/json.
inheritedlog
A preconfigured logger for the request handler.
optionalinheritedproxyInfo
An object with information about currently used proxy by the crawler and configured by the ProxyConfiguration class.
inheritedrequest
The request object that was successfully loaded and navigated to, including the loadedUrl property.
inheritedresponse
The HTTP response object containing status code, headers, and other response metadata.
inheritedsendRequest
Fires HTTP request via the internal HTTP client, allowing to override the request options on the fly.
This is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests). Check the Skipping navigations for certain requests example for more detailed explanation of how to do that.
async requestHandler({ sendRequest }) {
const { body } = await sendRequest({
// override headers only
headers: { ... },
});
},
Type declaration
Parameters
optionalrequestOverrides: Partial<HttpRequestOptions>
optionaloptionsOverrides: SendRequestOptions
Returns Promise<Response>
inheritedsession
inheriteduseState
Returns the state - a piece of mutable persistent data shared across all the request handler runs.
Type declaration
Parameters
optionaldefaultValue: State
Returns Promise<State>
Methods
inheritedafterStorageCommit
Registers
callbackto run once the request's storage writes have been committed, or once committing them has failed - the point where a write made in the handler can finally be reacted to where it was made. Two things belong here: side effects that must agree with what actually landed (result counters, progress reporting), and handling of a write that did not land at all.async requestHandler({ pushData, afterStorageCommit, useState }) {const state = await useState({ itemCount: 0 });await pushData(item);afterStorageCommit((error) => {if (error) {throw new NonRetryableError('The page is too big to store', { cause: error });}state.itemCount++;});}useState()is deliberately not transactional, which is what makes the counter case necessary: incrementing in the handler would keep the increment for a request that pushes an item and then fails, while the item itself is rolled back.Callbacks run in registration order, after the commit and before the request is marked as handled. The first one to throw propagates and the rest do not run; on a failed commit its error replaces the commit error, so a callback can decide what the request fails with. After a successful commit, a callback that throws fails the request without a retry whatever it throws - the writes are already durable and retrying would duplicate them, so anything that is not already a NonRetryableError is wrapped in an AfterCommitError.
Callbacks are not run when the request handler itself fails - nothing was written, and
errorHandler/failedRequestHandlercover that.Throws when storage is not transactional (
transactionalStorage: false) - writes are then applied as they are made and throw at the call site on their own. In AdaptivePlaywrightCrawler the callback belongs to the current request handler attempt, so it runs only for the attempt whose writes are the ones being committed.Parameters
callback: (error) => Awaitable<void>
Returns void
inheritedenqueueLinks
Helper function for extracting URLs from the parsed DOM and adding them to the request queue.
Parameters
optionaloptions: EnqueueLinksOptions
Returns Promise<AddRequestsBatchedResult>
inheritedextendTimeout
Gives the current request
secsmore seconds to finish, for when how long it needs is only apparent once it is already running - a listing page that turns out to have far more to scroll through than usual, say. PreferrequestHandlerTimeoutSecs, or a per-route override viarouter.addHandler, whenever the time needed is known up front.router.addHandler('LIST', async ({ extendTimeout, page }) => {const pageCount = await countPages(page);extendTimeout(pageCount * 10);await scrapeAllPages(page);});Extends the request handler's own timeout and the crawler's internal one together, so the extension is not immediately undone by the latter. Calling it from a handler that has already timed out does nothing.
Parameters
secs: number
Returns void
inheritedextractLinks
Extracts URLs from the parsed DOM, without adding them to the request queue.
Parameters
optionaloptions: ExtractLinksOptions
Returns Promise<string[]>
inheritedparseWithCheerio
Returns Cheerio handle for
page.content(), allowing to work with the data same way as with CheerioCrawler. When provided with theselectorargument, it will throw if it's not available.Example usage:
async requestHandler({ parseWithCheerio }) {const $ = await parseWithCheerio();const title = $('title').text();});Parameters
optionalselector: string
optionaltimeoutMs: number
Returns Promise<CheerioAPI>
inheritedpushData
This function allows you to push data to a Dataset specified by name, or the one currently used by the crawler.
Shortcut for
crawler.pushData().Parameters
optionaldata: ReadonlyDeep<Dictionary | Dictionary[]>
Data to be pushed to the default dataset.
optionaldatasetIdentifier: string | StorageIdentifier
Returns Promise<void>
inheritedregisterDeferredCleanup
Register a function to be called at the very end of the request handling process. This is useful for resources that should be accessible to error handlers, for instance.
The callback runs outside the request's storage transaction, so storage writes made here are applied immediately and are not rolled back when the request fails. In AdaptivePlaywrightCrawler it also runs once per request handler attempt, so a write here can land more than once for a single request. Push results from the request handler itself.
Parameters
cleanup: () => Promise<unknown>
Returns void
inheritedwaitForSelector
Wait for an element matching the selector to appear. Timeout is ignored.
Example usage:
async requestHandler({ waitForSelector, parseWithCheerio }) {await waitForSelector('article h1');const $ = await parseWithCheerio();const title = $('title').text();});Parameters
selector: string
optionaltimeoutMs: number
Returns Promise<void>
Add requests directly to the request queue currently used by the crawler.
Optionally, the function allows you to filter the target URLs using an array of glob or regexp patterns, the same way
enqueueLinksdoes for extracted links.