FileSystemStorageOptions
Index
Properties
localDataDirectory
optionallogger
Optional logger for FileSystemStorageBackend warnings.
optionalrequestQueueAccess
How the on-disk request queues opened by this backend are expected to be accessed.
With 'single' (the default), this process asserts it is the sole consumer of every request
queue it opens: on open, any requests that a previous run left in progress (e.g. after a
crash) are reclaimed immediately, so they become fetchable again right away. This is the right
behavior for the common single-process crawl.
Use 'shared' if multiple processes share the same on-disk request queue concurrently (for
example, the "Parallel Scraping Guide"). In that mode an
in-progress request is treated as a potential live peer's lock and is only reclaimed once that
lock expires on the wall clock, so two workers won't process the same request at once.
Path to directory where the data will be saved.