Skip to main content
Version: Next

SessionPool

Handles the rotation, creation and persistence of user-like sessions. Creates a pool of Session instances, that are randomly rotated. When some session is marked as blocked, it is removed and new one is created instead (the pool never returns an unusable session). Learn more in the Session management guide.

Session pool is already integrated into crawlers and is always active. All public methods are lazy-initialized — the pool initializes itself on first use.

You can configure the pool with many options. See the SessionPoolOptions. Session pool is by default persisted in default KeyValueStore. If you want to have one pool for all runs you have to specify SessionPoolOptions.persistStateKeyValueStoreId.

Advanced usage:

const sessionPool = new SessionPool({
    maxPoolSize: 25,
    sessionOptions:{
         maxAgeSecs: 10,
         maxUsageCount: 150, // for example when you know that the site blocks after 150 requests.
    },
    persistStateKeyValueStoreId: 'my-key-value-store-for-sessions',
    persistStateKey: 'my-session-pool',
});

// Get random session from the pool
const session1 = await sessionPool.getSession();
const session2 = await sessionPool.getSession();
const session3 = await sessionPool.getSession();

// Now you can mark the session either failed or successful

// Marks session as bad after unsuccessful usage -> it increases error count (soft retire)
session1.markBad()

// Marks as successful.
session2.markGood()

// Retires session -> session is removed from the pool
session3.retire()

*Default session allocation flow:

  1. Until the SessionPool reaches maxPoolSize, new sessions are created, provided to the user and added to the pool
  2. Blocked/retired sessions stay in the pool but are never provided to the user
  3. Once the pool is full (live plus blocked session count reaches maxPoolSize), a random session from the pool is provided.
  4. If a blocked session would be picked, instead all blocked sessions are evicted from the pool and a new session is created and provided

Implements

Index

Constructors

Properties

Methods

Constructors

constructor

Properties

readonlyid

id: string

Methods

[asyncDispose]

  • [asyncDispose](): Promise<void>

  • Returns Promise<void>

addSession

  • addSession(options): Promise<void>

  • Adds a new session to the session pool. The pool automatically creates sessions up to the maximum size of the pool, but this allows you to add more sessions once the max pool size is reached. This also allows you to add session with overridden session options (e.g. with specific session id).

    Parameters

    • optionaloptions: SessionOptions | Session = {}

      The configuration options for the session being added to the session pool.

    Returns Promise<void>

getSession

  • getSession(sessionId): Promise<undefined | Session>

  • Gets session. If there is space for new session, it creates and returns new session. If the session pool is full, it picks a session from the pool, If the picked session is usable it is returned, otherwise it creates and returns a new one.

    Parameters

    • optionalsessionId: string

      If provided, it returns the usable session with this id, undefined otherwise.

    Returns Promise<undefined | Session>

newSession

  • newSession(sessionOptions): Promise<Session>

  • Adds a new session to the session pool. The pool automatically creates sessions up to the maximum size of the pool, but this allows you to add more sessions once the max pool size is reached. This also allows you to add session with overridden session options (e.g. with specific session id).

    Parameters

    Returns Promise<Session>

persistState

  • persistState(): Promise<void>

  • Persists the current state of the SessionPool into the default KeyValueStore. The state is persisted automatically in regular intervals.

    Returns Promise<void>

reset

  • reset(): void

  • Discards all sessions in the pool, both usable and retired, resetting it to a blank state.

    The persisted record is left alone - use SessionPool.resetStore to clear that as well.

    Returns void

resetStore

  • resetStore(): Promise<void>

  • Clears the persisted pool record, leaving the in-memory sessions alone.

    Throws while the state is still being persisted periodically - the next PERSIST_STATE event would write the record straight back. Call SessionPool.teardown first, or use SessionPool.reset to discard the sessions themselves. A no-op if persistence is disabled.

    Returns Promise<void>

retiredSessionsCount

  • retiredSessionsCount(): Promise<number>

  • Gets count of retired sessions in the pool.

    Returns Promise<number>

teardown

  • teardown(): Promise<void>

  • Stops the periodic state persistence and persists the state one last time. This function should be called after you are done with using the SessionPool instance. Using the pool again afterwards initializes it anew.

    Returns Promise<void>

usableSessionsCount

  • usableSessionsCount(): Promise<number>

  • Gets count of usable sessions in the pool.

    Returns Promise<number>

Page Options