PlaywrightCrawlingContext <UserData>
Hierarchy
- BrowserCrawlingContext<PlaywrightCrawler, Page, Response, PlaywrightController, UserData>
- PlaywrightContextUtils
- PlaywrightCrawlingContext
Index
Properties
addRequests
Type declaration
Parameters
requestsLike: readonly (string | ReadonlyObjectDeep<Partial<RequestOptions<Dictionary>> & { regex?: RegExp; requestsFromUrl?: string }> | ReadonlyObjectDeep<Request<Dictionary>>)[]
optionaloptions: ReadonlyObjectDeep<RequestQueueOperationOptions>
Options for the request queue
Returns Promise<void>
browserController
crawler
getKeyValueStore
Type declaration
Get a key-value store with given name or id, or the default one for the crawler.
Parameters
optionalidOrName: string
Returns Promise<KeyValueStore>
id
log
A preconfigured logger for the request handler.
page
optionalproxyInfo
An object with information about currently used proxy by the crawler and configured by the ProxyConfiguration class.
request
The original Request object.
optionalresponse
optionalsession
useState
Type declaration
Returns the state - a piece of mutable persistent data shared across all the request handler runs.
Type parameters
- State: Dictionary = Dictionary
Parameters
optionaldefaultValue: State
Returns Promise<State>
Methods
blockRequests
Forces the Playwright browser tab to block loading URLs that match a provided pattern. This is useful to speed up crawling of websites, since it reduces the amount of data that needs to be downloaded from the web, but it may break some websites or unexpectedly prevent loading of resources.
By default, the function will block all URLs including the following patterns:
[".css", ".jpg", ".jpeg", ".png", ".svg", ".gif", ".woff", ".pdf", ".zip"]If you want to extend this list further, use the
extraUrlPatternsoption, which will keep blocking the default patterns, as well as add your custom ones. If you would like to block only specific patterns, use theurlPatternsoption, which will override the defaults and block only URLs with your custom patterns.This function does not use Playwright's request interception and therefore does not interfere with browser cache. It's also faster than blocking requests using interception, because the blocking happens directly in the browser without the round-trip to Node.js, but it does not provide the extra benefits of request interception.
The function will never block main document loads and their respective redirects.
Example usage
preNavigationHooks: [
async ({ blockRequests }) => {
// Block all requests to URLs that include `adsbygoogle.js` and also all defaults.
await blockRequests({
extraUrlPatterns: ['adsbygoogle.js'],
});
},
],Parameters
optionaloptions: BlockRequestsOptions
Returns Promise<void>
closeCookieModals
Tries to close cookie consent modals on the page. Based on the I Don't Care About Cookies browser extension.
Returns Promise<void>
compileScript
Compiles a Playwright script into an async function that may be executed at any time by providing it with the following object:
{
page: Page,
request: Request,
}Where
pageis a PlaywrightPageandrequestis a Request.The function is compiled by using the
scriptStringparameter as the function's body, so any limitations to function bodies apply. Return value of the compiled function is the return value of the function body = thescriptStringparameter.As a security measure, no globals such as
processorrequireare accessible from within the function body. Note that the function does not provide a safe sandbox and even though globals are not easily accessible, malicious code may still execute in the main process via prototype manipulation. Therefore you should only use this function to execute sanitized or safe code.Custom context may also be provided using the
contextparameter. To improve security, make sure to only pass the really necessary objects to the context. Preferably making secured copies beforehand.Parameters
scriptString: string
optionalctx: Dictionary
Returns CompiledScriptFunction
enqueueLinks
This function automatically finds and enqueues links from the current page, adding them to the RequestQueue currently used by the crawler.
Optionally, the function allows you to filter the target links' URLs using an array of globs or regular expressions and override settings of the enqueued Request objects.
Check out the Crawl a website with relative links example for more details regarding its usage.
Example usage
async requestHandler({ enqueueLinks }) {
await enqueueLinks({
globs: [
'https://www.example.com/handbags/*',
],
});
},Parameters
optionaloptions: ReadonlyObjectDeep<Omit<EnqueueLinksOptions, requestQueue>> & Pick<EnqueueLinksOptions, requestQueue>
All
enqueueLinks()parameters are passed via an options object.
Returns Promise<BatchAddRequestsResult>
Promise that resolves to BatchAddRequestsResult object.
enqueueLinksByClickingElements
The function finds elements matching a specific CSS selector in a Playwright page, clicks all those elements using a mouse move and a left mouse button click and intercepts all the navigation requests that are subsequently produced by the page. The intercepted requests, including their methods, headers and payloads are then enqueued to a provided RequestQueue. This is useful to crawl JavaScript heavy pages where links are not available in
hrefelements, but rather navigations are triggered in click handlers. If you're looking to find URLs inhrefattributes of the page, see enqueueLinks.Optionally, the function allows you to filter the target links' URLs using an array of PseudoUrl objects and override settings of the enqueued Request objects.
IMPORTANT: To be able to do this, this function uses various mutations on the page, such as changing the Z-index of elements being clicked and their visibility. Therefore, it is recommended to only use this function as the last operation in the page.
USING HEADFUL BROWSER: When using a headful browser, this function will only be able to click elements in the focused tab, effectively limiting concurrency to 1. In headless mode, full concurrency can be achieved.
PERFORMANCE: Clicking elements with a mouse and intercepting requests is not a low level operation that takes nanoseconds. It's not very CPU intensive, but it takes time. We strongly recommend limiting the scope of the clicking as much as possible by using a specific selector that targets only the elements that you assume or know will produce a navigation. You can certainly click everything by using the
*selector, but be prepared to wait minutes to get results on a large and complex page.Example usage
async requestHandler({ enqueueLinksByClickingElements }) {
await enqueueLinksByClickingElements({
selector: 'a.product-detail',
globs: [
'https://www.example.com/handbags/**'
'https://www.example.com/purses/**'
],
});
});Parameters
options: Omit<EnqueueLinksByClickingElementsOptions, requestQueue | page>
Returns Promise<BatchAddRequestsResult>
Promise that resolves to BatchAddRequestsResult object.
infiniteScroll
Scrolls to the bottom of a page, or until it times out. Loads dynamic content when it hits the bottom of a page, and then continues scrolling.
Parameters
optionaloptions: InfiniteScrollOptions
Returns Promise<void>
injectFile
Injects a JavaScript file into current
page. Unlike Playwright'saddScriptTagfunction, this function works on pages with arbitrary Cross-Origin Resource Sharing (CORS) policies.File contents are cached for up to 10 files to limit file system access.
Parameters
filePath: string
optionaloptions: InjectFileOptions
Returns Promise<unknown>
injectJQuery
Injects the jQuery library into current
page. jQuery is often useful for various web scraping and crawling tasks. For example, it can help extract text from HTML elements using CSS selectors.Beware that the injected jQuery object will be set to the
window.$variable and thus it might cause conflicts with other libraries included by the page that use the same variable name (e.g. another version of jQuery). This can affect functionality of page's scripts.The injected jQuery will survive page navigations and reloads.
Example usage:
async requestHandler({ page, injectJQuery }) {
await injectJQuery();
const title = await page.evaluate(() => {
return $('head title').text();
});
});Note that
injectJQuery()does not affect the Playwrightpage.$()function in any way.Returns Promise<unknown>
parseWithCheerio
Returns Cheerio handle for
page.content(), allowing to work with the data same way as with CheerioCrawler.Example usage:
async requestHandler({ parseWithCheerio }) {
const $ = await parseWithCheerio();
const title = $('title').text();
});Returns Promise<CheerioAPI>
pushData
This function allows you to push data to a Dataset specified by name, or the one currently used by the crawler.
Shortcut for
crawler.pushData().Parameters
optionaldata: ReadonlyDeep<Dictionary | Dictionary[]>
Data to be pushed to the default dataset.
optionaldatasetIdOrName: string
Returns Promise<void>
saveSnapshot
Saves a full screenshot and HTML of the current page into a Key-Value store.
Parameters
optionaloptions: SaveSnapshotOptions
Returns Promise<void>
sendRequest
Fires HTTP request via
got-scraping, allowing to override the request options on the fly.This is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests). Check the Skipping navigations for certain requests example for more detailed explanation of how to do that.
async requestHandler({ sendRequest }) {
const { body } = await sendRequest({
// override headers only
headers: { ... },
});
},Type parameters
- Response = string
Parameters
optionaloverrideOptions: Partial<OptionsInit>
Returns Promise<Response<Response>>
Add requests directly to the request queue.