Scrapy - Other Settings

AJAXCRAWL_ENABLED

It is used for enabling the large crawls.

Default value: False

AUTOTHROTTLE_DEBUG

It is enabled to see how throttling parameters are adjusted in real time, which displays stats on every received response.

Default value: False

AUTOTHROTTLE_ENABLED

It is used to enable AutoThrottle extension.

Default value: False

AUTOTHROTTLE_MAX_DELAY

It is used to set the maximum delay for download in case of high latencies.

Default value: 60.0

AUTOTHROTTLE_START_DELAY

It is used to set the initial delay for download.

Default value: 5.0

AUTOTHROTTLE_TARGET_CONCURRENCY

It defines the average number of requests for a Scrapy to send parallely to remote sites.

Default value: 1.0

CLOSESPIDER_ERRORCOUNT

It defines total number of errors that should be recieved before the spider is closed.

Default value: 0

CLOSESPIDER_ITEMCOUNT

It defines a total number of items before closing the spider.

Default value: 0

CLOSESPIDER_PAGECOUNT

It defines the maximum number of responses to crawl before spider closes.

Default value: 0

CLOSESPIDER_TIMEOUT

It defines the amount of time (in sec) for a spider to close.

Default value: 0

COMMANDS_MODULE

It is used when you want to add custom commands in your project.

Default value: ''

COMPRESSION_ENABLED

It indicates that the compression middleware is enabled.

Default value: True

COOKIES_DEBUG

If set to true, all the cookies sent in requests and received in responses are logged.

Default value: False

COOKIES_ENABLED

It indicates that cookies middleware is enabled and sent to web servers.

Default value: True

FILES_EXPIRES

It defines the delay for the file expiration.

Default value: 90 days

FILES_RESULT_FIELD

It is set when you want to use other field names for your processed files.

FILES_STORE

It is used to store the downloaded files by setting it to a valid value.

FILES_STORE_S3_ACL

It is used to modify the ACL policy for the files stored in Amazon S3 bucket.

Default value: private

FILES_URLS_FIELD

It is set when you want to use other field name for your files URLs.

HTTPCACHE_ALWAYS_STORE

Spider will cache the pages thoroughly if this setting is enabled.

Default value: False

HTTPCACHE_DBM_MODULE

It is a database module used in DBM storage backend.

Default value: 'anydbm'

HTTPCACHE_DIR

It is a directory used to enable and store the HTTP cache.

Default value: 'httpcache'

HTTPCACHE_ENABLED

It indicates that HTTP cache is enabled.

Default value: False

HTTPCACHE_EXPIRATION_SECS

It is used to set the expiration time for HTTP cache.

Default value: 0

HTTPCACHE_GZIP

This setting if set to true, all the cached data will be compressed with gzip.

Default value: False

HTTPCACHE_IGNORE_HTTP_CODES

It states that HTTP responses should not be cached with HTTP codes.

Default value: []

HTTPCACHE_IGNORE_MISSING

This setting if enabled, the requests will be ignored if not found in the cache.

Default value: False

HTTPCACHE_IGNORE_RESPONSE_CACHE_CONTROLS

It is a list containing cache controls to be ignored.

Default value: []

HTTPCACHE_IGNORE_SCHEME

It states that HTTP responses should not be cached with URI schemes.

Default value: ['file']

HTTPCACHE_POLICY

It defines a class implementing cache policy.

Default value: 'scrapy.extensions.httpcache.DummyPolicy'

HTTPCACHE_STORAGE

It is a class implementing the cache storage.

Default value: 'scrapy.extensions.httpcache.FilesystemCacheStorage'

HTTPERROR_ALLOWED_CODES

It is a list where all the responses are passed with non-200 status codes.

Default value: []

HTTPERROR_ALLOW_ALL

This setting when enabled, all the responses are passed despite of its status codes.

Default value: False

HTTPPROXY_AUTH_ENCODING

It is used to authenticate the proxy on HttpProxyMiddleware.

Default value: "latin-1"

IMAGES_EXPIRES

It defines the delay for the images expiration.

Default value: 90 days

IMAGES_MIN_HEIGHT

It is used to drop images that are too small using minimum size.

IMAGES_MIN_WIDTH

It is used to drop images that are too small using minimum size.

IMAGES_RESULT_FIELD

It is set when you want to use other field name for your processed images.

IMAGES_STORE

It is used to store the downloaded images by setting it to a valid value.

IMAGES_STORE_S3_ACL

It is used to modify the ACL policy for the images stored in Amazon S3 bucket.

Default value: private

IMAGES_THUMBS

It is set to create the thumbnails of downloaded images.

IMAGES_URLS_FIELD

It is set when you want to use other field name for your images URLs.

MAIL_FROM

The sender uses this setting to send the emails.

Default value: 'scrapy@localhost'

MAIL_HOST

It is a SMTP host used to send emails.

Default value: 'localhost'

MAIL_PASS

It is a password used to authenticate SMTP.

Default value: None

MAIL_PORT

It is a SMTP port used to send emails.

Default value: 25

MAIL_SSL

It is used to implement connection using SSL encrypted connection.

Default value: False

MAIL_TLS

When enabled, it forces connection using STARTTLS.

Default value: False

MAIL_USER

It defines a user to authenticate SMTP.

Default value: None

METAREFRESH_ENABLED

It indicates that meta refresh middleware is enabled.

Default value: True

METAREFRESH_MAXDELAY

It is a maximum delay for a meta-refresh to redirect.

Default value: 100

REDIRECT_ENABLED

It indicates that the redirect middleware is enabled.

Default value: True

REDIRECT_MAX_TIMES

It defines the maximum number of times for a request to redirect.

Default value: 20

REFERER_ENABLED

It indicates that referrer middleware is enabled.

Default value: True

RETRY_ENABLED

It indicates that the retry middleware is enabled.

Default value: True

RETRY_HTTP_CODES

It defines which HTTP codes are to be retried.

Default value: [500, 502, 503, 504, 408]

RETRY_TIMES

It defines maximum number of times for retry.

Default value: 2

TELNETCONSOLE_HOST

It defines an interface on which the telnet console must listen.

Default value: '127.0.0.1'

TELNETCONSOLE_PORT

It defines a port to be used for telnet console.

Default value: [6023, 6073]