crawlee

Index

Autoscaling

• AutoscaledPool
• Snapshotter
• SystemStatus

Browser management

• BrowserController
• BrowserPlugin
• BrowserPool
• PlaywrightBrowserController
• PlaywrightBrowserPlugin
• PlaywrightPersistentBrowser

Configuration

• ConcurrencySettings
• Configuration
• ProxyConfiguration
• ServiceLocator

Crawlers

• AbstractHttpCrawler
• AdaptivePlaywrightCrawler
• BasicCrawler
• BeautifulSoupCrawler
• HttpCrawler
• ParselCrawler
• PlaywrightCrawler

Crawling contexts

• AdaptivePlaywrightCrawlingContext
• AdaptivePlaywrightPreNavCrawlingContext
• BasicCrawlingContext
• BeautifulSoupCrawlingContext
• HttpCrawlingContext
• HttpCrawlingResult
• ParsedHttpCrawlingContext
• ParselCrawlingContext
• PlaywrightCrawlingContext
• PlaywrightPreNavCrawlingContext

Errors

• ContextPipelineFinalizationError
• ContextPipelineInitializationError
• ContextPipelineInterruptedError
• HttpClientStatusCodeError
• HttpStatusCodeError
• ProxyError
• RequestCollisionError
• RequestHandlerError
• ServiceConflictError
• SessionError
• UserDefinedErrorHandlerError

Event data

• Event
• EventAbortingData
• EventCrawlerStatusData
• EventExitData
• EventMigratingData
• EventPersistStateData
• EventSystemInfoData

Event managers

• EventManager
• LocalEventManager

Functions

• AddRequestsFunction
• BlockRequestsFunction
• EnqueueLinksFunction
• ExtractLinksFunction
• GetKeyValueStoreFunction
• PushDataFunction
• SendRequestFunction
• UseStateFunction

HTTP clients

• CurlImpersonateHttpClient
• HttpClient
• HttpxHttpClient
• ImpitHttpClient

HTTP parsers

• AbstractHttpParser
• BeautifulSoupParser
• NoParser
• ParselParser

Other

• BrowserforgeFingerprintGenerator
• ContextPipeline
• CrawlerInstrumentor
• DefaultRenderingTypePredictor
• FingerprintGenerator
• HeaderGenerator
• HttpHeaders
• HttpResponse
• PageSnapshot
• ProxyInfo
• RenderingTypePrediction
• RenderingTypePredictor
• RequestOptions
• Router

Request loaders

• RequestList
• RequestLoader
• RequestManager
• RequestManagerTandem
• SitemapRequestLoader

Session management

• CookieParam
• Session
• SessionCookies
• SessionPool

Statistics

• AdaptivePlaywrightCrawlerStatisticState
• FinalStatistics
• Statistics
• StatisticsState

Storage clients

• FileSystemStorageClient
• MemoryStorageClient
• SqlStorageClient
• StorageClient

Storage data

• AddRequestsResponse
• DatasetItemsListPage
• DatasetMetadata
• KeyValueStoreMetadata
• KeyValueStoreRecord
• KeyValueStoreRecordMetadata
• ProcessedRequest
• Request
• RequestQueueMetadata
• StorageMetadata
• UnprocessedRequest

Storages

• Dataset
• KeyValueStore
• RequestQueue
• Storage