AdaptivePlaywrightCrawlerContext
Hierarchy
- RestrictedCrawlingContext
- AdaptivePlaywrightCrawlerContext
Index
Properties
addRequests
Type declaration
Parameters
requestsLike: readonly (string | ReadonlyObjectDeep<Partial<RequestOptions<Dictionary>> & { regex?: RegExp; requestsFromUrl?: string }> | ReadonlyObjectDeep<Request<Dictionary>>)[]
optionaloptions: ReadonlyObjectDeep<RequestQueueOperationOptions>
Options for the request queue
Returns Promise<void>
enqueueLinks
This function automatically finds and enqueues links from the current page, adding them to the RequestQueue currently used by the crawler.
Optionally, the function allows you to filter the target links' URLs using an array of globs or regular expressions and override settings of the enqueued Request objects.
Check out the Crawl a website with relative links example for more details regarding its usage.
Example usage
async requestHandler({ enqueueLinks }) {
await enqueueLinks({
globs: [
'https://www.example.com/handbags/*',
],
});
},
Type declaration
Parameters
optionaloptions: ReadonlyObjectDeep<Omit<EnqueueLinksOptions, requestQueue>>
All
enqueueLinks()
parameters are passed via an options object.
Returns Promise<unknown>
getKeyValueStore
Get a key-value store with given name or id, or the default one for the crawler.
Type declaration
Parameters
optionalidOrName: string
Returns Promise<Pick<KeyValueStore, id | name | getValue | getAutoSavedValue | setValue>>
id
log
A preconfigured logger for the request handler.
optionalproxyInfo
An object with information about currently used proxy by the crawler and configured by the ProxyConfiguration class.
request
The original Request object.
optionalsession
useState
Returns the state - a piece of mutable persistent data shared across all the request handler runs.
Type declaration
Type parameters
- State: Dictionary = Dictionary
Parameters
optionaldefaultValue: State
Returns Promise<State>
Methods
parseWithCheerio
Returns Cheerio handle for
page.content()
, allowing to work with the data same way as with CheerioCrawler. When provided with theselector
argument, it will first look for the selector with a 5s timeout.Example usage:
async requestHandler({ parseWithCheerio }) {
const $ = await parseWithCheerio();
const title = $('title').text();
});Parameters
optionalselector: string
optionaltimeoutMs: number
Returns Promise<CheerioAPI>
pushData
This function allows you to push data to a Dataset specified by name, or the one currently used by the crawler.
Shortcut for
crawler.pushData()
.Parameters
optionaldata: ReadonlyDeep<Dictionary | Dictionary[]>
Data to be pushed to the default dataset.
optionaldatasetIdOrName: string
Returns Promise<void>
querySelector
Wait for an element matching the selector to appear and return a Cheerio object of matched elements. Timeout defaults to 5s.
Parameters
selector: string
optionaltimeoutMs: number
Returns Promise<Cheerio<Element>>
waitForSelector
Wait for an element matching the selector to appear. Timeout defaults to 5s.
Example usage:
async requestHandler({ waitForSelector, parseWithCheerio }) {
await waitForSelector('article h1');
const $ = await parseWithCheerio();
const title = $('title').text();
});Parameters
selector: string
optionaltimeoutMs: number
Returns Promise<void>
Add requests directly to the request queue.