Skip to main content

Request

crawlee.models.Request

A crawling request (as returned from a request queue).

Index

Methods

enqueue_strategy

  • enqueue_strategy(new_enqueue_strategy): None
  • Parameters

    • new_enqueue_strategy: EnqueueStrategy

    Returns None

forefront

  • forefront(new_value): None
  • Parameters

    • new_value: bool

    Returns None

from_base_request_data

  • from_base_request_data(base_request_data, *, id): Self
  • Create a complete Request object based on a BaseRequestData instance.


    Parameters

    • base_request_data: BaseRequestData
    • id: str | None = Nonekeyword-only

    Returns Self

from_url

  • from_url(url, *, label, unique_key, id, kwargs): Self
  • Create a new RequestData instance from a URL.


    Parameters

    • url: str
    • label: str | None = Nonekeyword-only
    • unique_key: str | None = Nonekeyword-only
    • id: str | None = Nonekeyword-only
    • kwargs: Any

    Returns Self

last_proxy_tier

  • last_proxy_tier(new_value): None
  • Parameters

    • new_value: int

    Returns None

max_retries

  • max_retries(new_max_retries): None
  • Parameters

    • new_max_retries: int

    Returns None

session_rotation_count

  • session_rotation_count(new_session_rotation_count): None
  • Parameters

    • new_session_rotation_count: int

    Returns None

state

  • state(new_state): None
  • Parameters

    • new_state: RequestState

    Returns None

Properties

crawlee_data

crawlee_data: CrawleeRequestData

Crawlee-specific configuration stored in the user_data.

enqueue_strategy

enqueue_strategy: EnqueueStrategy

The strategy used when enqueueing the request.

forefront

forefront: bool

Should the request be enqueued at the start of the queue?

id

id: str

json_

json_: str | None

TODO: get rid of this

label

label: str | None

A string used to differentiate between arbitrary request types.

last_proxy_tier

last_proxy_tier: int | None

The last proxy tier used to process the request.

max_retries

max_retries: int | None

Crawlee-specific limit on the number of retries of the request.

order_no

order_no: Decimal | None

TODO: get rid of this

session_rotation_count

session_rotation_count: int | None

Crawlee-specific number of finished session rotations for the request.

state

state: RequestState | None

Crawlee-specific request handling state.