HttpCrawlingContext <UserData, JSONData>
Hierarchy
- InternalHttpCrawlingContext<UserData, JSONData, HttpCrawler<HttpCrawlingContext<UserData, JSONData>>>
- HttpCrawlingContext
Index
Properties
body
contentType
Parsed Content-Type header: { type, encoding }
.
Type declaration
encoding: BufferEncoding
type: string
crawler
id
json
The parsed object from JSON string if the response contains the content type application/json.
log
optionalproxyInfo
An object with information about currently used proxy by the crawler and configured by the ProxyConfiguration class.
request
The original Request object.
response
optionalsession
Methods
enqueueLinks
This function automatically finds and enqueues links from the current page, adding them to the RequestQueue currently used by the crawler.
Optionally, the function allows you to filter the target links' URLs using an array of globs or regular expressions and override settings of the enqueued Request objects.
Check out the Crawl a website with relative links example for more details regarding its usage.
Example usage
async requestHandler({ enqueueLinks }) {
await enqueueLinks({
globs: [
'https://www.example.com/handbags/*',
],
});
},Parameters
optionaloptions: EnqueueLinksOptions
All
enqueueLinks()
parameters are passed via an options object.
Returns Promise<BatchAddRequestsResult>
Promise that resolves to BatchAddRequestsResult object.
parseWithCheerio
Returns Promise<CheerioAPI>
sendRequest
Fires HTTP request via
got-scraping
, allowing to override the request options on the fly.This is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests). Check the Skipping navigations for certain requests example for more detailed explanation of how to do that.
async requestHandler({ sendRequest }) {
const { body } = await sendRequest({
// override headers only
headers: { ... },
});
},Type parameters
- Response = string
Parameters
optionaloverrideOptions: Partial<OptionsInit>
Returns Promise<Response<Response>>
The request body of the web page. The type depends on the
Content-Type
header of the web page:text/html
,application/xhtml+xml
,application/xml
MIME content types