SessionPool

crawlee.sessions._session_pool.SessionPool

Session pool is a pool of sessions that are rotated based on the usage count or age.

Index

Constructors

• __init__

Methods

• __aenter__
• __aexit__
• __repr__
• add_session
• get_session
• get_session_by_id
• get_state
• get_state
• get_state
• reset_store

Properties

• retired_session_count
• session_count
• usable_session_count

Constructors

__init__

• __init__(*, max_pool_size, create_session_settings, create_session_function, event_manager, persistence_enabled, persist_state_kvs_name, persist_state_key): None

• Create a new instance.

  ---

  Parameters

  • max_pool_size: int = 1000keyword-only

  • create_session_settings: dict | None = Nonekeyword-only

  • create_session_function: CreateSessionFunctionType | None = Nonekeyword-only

  • event_manager: EventManager | None = Nonekeyword-only

  • persistence_enabled: bool = Falsekeyword-only

  • persist_state_kvs_name: str = 'default'keyword-only

  • persist_state_key: str = 'CRAWLEE_SESSION_POOL_STATE'keyword-only

  Returns None

Methods

__aenter__

• async __aenter__(): SessionPool

• Initialize the pool upon entering the context manager.

  ---

  Returns SessionPool

__aexit__

• async __aexit__(exc_type, exc_value, exc_traceback): None

• Deinitialize the pool upon exiting the context manager.

  ---

  Parameters

  • exc_type: type[BaseException] | None

  • exc_value: BaseException | None

  • exc_traceback: TracebackType | None

  Returns None

__repr__

• __repr__(): str

• Get a string representation.

  ---

  Returns str

add_session

• add_session(session): None

• Add a specific session to the pool.

  This is intened only for the cases when you want to add a session that was created outside of the pool. Otherwise, the pool will create new sessions automatically.

  ---

  Parameters

  • session: Session

  Returns None

get_session

• async get_session(): Session

• Retrieve a random session from the pool.

  This method first ensures the session pool is at its maximum capacity. If the random session is not usable, retired sessions are removed and a new session is created and returned.

  ---

  Returns Session

get_session_by_id

• async get_session_by_id(session_id): Session | None

• Retrieve a session by ID from the pool.

  This method first ensures the session pool is at its maximum capacity. It then tries to retrieve a specific session by ID. If the session is not found or not usable, None is returned.

  ---

  Parameters

  • session_id: str

  Returns Session | None

get_state

• get_state(*, as_dict): dict

• Parameters

  • as_dict: Literal[True]keyword-only

  Returns dict

get_state

• get_state(*, as_dict): SessionPoolModel

• Parameters

  • as_dict: Literal[False]keyword-only

  Returns SessionPoolModel

get_state

• get_state(*, as_dict): SessionPoolModel | dict

• Retrieve the current state of the pool either as a model or as a dictionary.

  ---

  Parameters

  • as_dict: bool = Falsekeyword-only

  Returns SessionPoolModel | dict

reset_store

• async reset_store(): None

• Reset the KVS where the pool state is persisted.

  ---

  Returns None

Properties

retired_session_count

retired_session_count: int

Get the number of sessions that are no longer usable.

session_count

session_count: int

Get the total number of sessions currently maintained in the pool.

usable_session_count

usable_session_count: int

Get the number of sessions that are currently usable.