Skip to main content
Version: 3.11

ProxyConfiguration

Configures connection to a proxy server with the provided options. Proxy servers are used to prevent target websites from blocking your crawlers based on IP address rate limits or blacklists. Setting proxy configuration in your crawlers automatically configures them to use the selected proxies for all connections. You can get information about the currently used proxy by inspecting the ProxyInfo property in your crawler's page function. There, you can inspect the proxy's URL and other attributes.

If you want to use your own proxies, use the ProxyConfigurationOptions.proxyUrls option. Your list of proxy URLs will be rotated by the configuration if this option is provided.

Example usage:


const proxyConfiguration = new ProxyConfiguration({
  proxyUrls: ['...', '...'],
});

const crawler = new CheerioCrawler({
  // ...
  proxyConfiguration,
  requestHandler({ proxyInfo }) {
     const usedProxyUrl = proxyInfo.url; // Getting the proxy URL
  }
})

Index

Constructors

Properties

Methods

Constructors

constructor

  • Creates a ProxyConfiguration instance based on the provided options. Proxy servers are used to prevent target websites from blocking your crawlers based on IP address rate limits or blacklists. Setting proxy configuration in your crawlers automatically configures them to use the selected proxies for all connections.

    const proxyConfiguration = new ProxyConfiguration({
        proxyUrls: ['http://user:pass@proxy-1.com', 'http://user:pass@proxy-2.com'],
    });

    const crawler = new CheerioCrawler({
      // ...
      proxyConfiguration,
      requestHandler({ proxyInfo }) {
          const usedProxyUrl = proxyInfo.url; // Getting the proxy URL
      }
    })

    Parameters

    Returns ProxyConfiguration

Properties

isManInTheMiddle

isManInTheMiddle: boolean = false

Methods

newProxyInfo

  • newProxyInfo(sessionId?: string | number, options?: TieredProxyOptions): Promise<undefined | ProxyInfo>

  • This function creates a new ProxyInfo info object. It is used by CheerioCrawler and PuppeteerCrawler to generate proxy URLs and also to allow the user to inspect the currently used proxy via the requestHandler parameter proxyInfo. Use it if you want to work with a rich representation of a proxy URL. If you need the URL string only, use ProxyConfiguration.newUrl.

    Parameters

    • optionalsessionId: string | number

      Represents the identifier of user Session that can be managed by the SessionPool or you can use the Apify Proxy Session identifier. When the provided sessionId is a number, it's converted to a string. Property sessionId of ProxyInfo is always returned as a type string.

      All the HTTP requests going through the proxy with the same session identifier will use the same target proxy server (i.e. the same IP address). The identifier must not be longer than 50 characters and include only the following: 0-9, a-z, A-Z, ".", "_" and "~".

    • optionaloptions: TieredProxyOptions

    Returns Promise<undefined | ProxyInfo>

    Represents information about used proxy and its configuration.

newUrl

  • newUrl(sessionId?: string | number, options?: TieredProxyOptions): Promise<undefined | string>

  • Returns a new proxy URL based on provided configuration options and the sessionId parameter.

    Parameters

    • optionalsessionId: string | number

      Represents the identifier of user Session that can be managed by the SessionPool or you can use the Apify Proxy Session identifier. When the provided sessionId is a number, it's converted to a string.

      All the HTTP requests going through the proxy with the same session identifier will use the same target proxy server (i.e. the same IP address). The identifier must not be longer than 50 characters and include only the following: 0-9, a-z, A-Z, ".", "_" and "~".

    • optionaloptions: TieredProxyOptions

    Returns Promise<undefined | string>

    A string with a proxy URL, including authentication credentials and port number. For example, http://bob:password123@proxy.example.com:8000

Page Options