Skip to main content
Version: Next

ProxyConfiguration

Configures connection to a proxy server with the provided options. Proxy servers are used to prevent target websites from blocking your crawlers based on IP address rate limits or blacklists. Setting proxy configuration in your crawlers automatically configures them to use the selected proxies for all connections. You can get information about the currently used proxy by inspecting the ProxyInfo property in your crawler's page function. There, you can inspect the proxy's URL and other attributes.

If you want to use your own proxies, use the ProxyConfigurationOptions.proxyUrls option. Your list of proxy URLs will be rotated by the configuration if this option is provided.

Example usage:


const proxyConfiguration = new ProxyConfiguration({
  proxyUrls: ['...', '...'],
});

const crawler = new CheerioCrawler({
  // ...
  proxyConfiguration,
  requestHandler({ proxyInfo }) {
     const usedProxyUrl = proxyInfo.url; // Getting the proxy URL
  }
})

Implements

Index

Constructors

Properties

Methods

Constructors

constructor

  • Creates a ProxyConfiguration instance based on the provided options. Proxy servers are used to prevent target websites from blocking your crawlers based on IP address rate limits or blacklists. Setting proxy configuration in your crawlers automatically configures them to use the selected proxies for all connections.

    const proxyConfiguration = new ProxyConfiguration({
        proxyUrls: ['http://user:pass@proxy-1.com', 'http://user:pass@proxy-2.com'],
    });

    const crawler = new CheerioCrawler({
      // ...
      proxyConfiguration,
      requestHandler({ proxyInfo }) {
          const usedProxyUrl = proxyInfo.url; // Getting the proxy URL
      }
    })

    Parameters

    Returns ProxyConfiguration

Properties

readonlyisManInTheMiddle

isManInTheMiddle: false = false

Methods

newProxyInfo

  • newProxyInfo(options): Promise<undefined | ProxyInfo>

  • This function creates a new ProxyInfo info object. It is used by CheerioCrawler and PuppeteerCrawler to generate proxy URLs and also to allow the user to inspect the currently used proxy via the requestHandler parameter proxyInfo. Use it if you want to work with a rich representation of a proxy URL. If you need the URL string only, use ProxyConfiguration.newUrl.

    Parameters

    • optionaloptions: NewUrlOptions

    Returns Promise<undefined | ProxyInfo>

    Represents information about used proxy and its configuration.

newUrl

  • newUrl(options): Promise<undefined | string>

  • Returns a new proxy URL based on provided configuration options.

    Parameters

    • optionaloptions: NewUrlOptions

    Returns Promise<undefined | string>

    A string with a proxy URL, including authentication credentials and port number. For example, http://bob:password123@proxy.example.com:8000

Page Options