PydanticAiCrawlingContext
Hierarchy
- ParselCrawlingContext
- PydanticAiCrawlingContext
Index
Methods
Properties
Methods
__hash__
Return hash of the context. Each context is considered unique.
Returns int
create_modified_copy
Create a modified copy of the crawling context with specified changes.
Parameters
optionalpush_data: PushDataFunction | None = None
optionaladd_requests: AddRequestsFunction | None = None
optionalget_key_value_store: GetKeyValueStoreFromRequestHandlerFunction | None = None
Returns Self
extract_form_requests
Create requests submitting the forms matching
selectorthe way a browser does.By default, only one form is submitted: the first one a browser can submit among those sharing the most field names with
fields.Parameters
optionalkeyword-onlyselector: str = 'form'
CSS selector for the forms. An invalid one matches no form.
optionalkeyword-onlyfields: Mapping[str, str | Sequence[str] | None] | None = None
Values replacing the form fields or adding new ones. A
Nonevalue drops the field and a sequence submits it once per value. Withall_forms, only the fields a form has are replaced.optionalkeyword-onlyclick: bool | Mapping[str, str] = True
The submit button to click:
Truefor the first enabled one, if any,Falsefor none, or a mapping of attributes the button must have, disabled or not, which skips forms without such a button.optionalkeyword-onlyall_forms: bool = False
Submit each matching form instead of only one.
kwargs: Unpack[FormRequestOptions]
Returns list[Request]
from_basic_crawling_context
Initialize a new instance from an existing
BasicCrawlingContext.Parameters
context: BasicCrawlingContext
http_response: HttpResponse
Returns Self
from_http_crawling_context
Initialize a new instance from an existing
HttpCrawlingContext.Parameters
context: HttpCrawlingContext
parsed_content: TParseResult
enqueue_links: EnqueueLinksFunction
extract_links: ExtractLinksFunction
Returns Self
from_parsed_http_crawling_context
Create a new context from an existing
ParsedHttpCrawlingContext[Selector].Parameters
context: ParsedHttpCrawlingContext[Selector]
Returns Self
from_parsel_crawling_context
Create a new context from an existing
ParselCrawlingContext.Parameters
context: ParselCrawlingContext
keyword-onlyextract: ExtractFunction
keyword-onlyai_usage: PydanticAiUsageStats
Returns Self
get_snapshot
Get snapshot of crawled page.
Returns PageSnapshot
html_to_text
Convert the parsed HTML content to newline-separated plain text without tags.
Returns str
Properties
add_requests
Add requests crawling context helper function.
ai_usage
The cumulative token usage stats of the extractor across calls in this crawl.
enqueue_links
extract
Extract a structured Pydantic model from the page using the configured AI extractor.
extract_links
get_key_value_store
Get key-value store crawling context helper function.
http_response
The HTTP response received from the server.
log
Logger instance.
parsed_content
proxy_info
Proxy information for the current page being processed.
push_data
Push data crawling context helper function.
register_deferred_cleanup
Register an async callback to be called after request processing completes (including error handlers).
request
Request object for the current page being processed.
selector
Convenience alias.
send_request
Send request crawling context helper function.
session
Session object for the current page being processed.
use_state
Use state crawling context helper function.
The crawling context used by the
PydanticAiCrawler.It extends
ParselCrawlingContext, so the full Parselselector(andenqueue_links) remain available alongside the AI-poweredextracthelper. Handlers can mix cheap manual selectors with AI extraction on the same page.