- Extension settings
- Loading & activating extensions
- Available, enabled and disabled extensions
- Disabling an extension
- Writing your own extension
- Built-in extensions reference
- General purpose extensions
- Debugging extensions
The extensions framework provides a mechanism for inserting your own custom functionality into Scrapy.
Extensions are just regular classes that are instantiated at Scrapy startup, when extensions are initialized.
Extensions use the Scrapy settings to manage their settings, just like any other Scrapy code.
It is customary for extensions to prefix their settings with their own name, to avoid collision with existing (and future) extensions. For example, a hypothetic extension to handle Google Sitemaps would use settings like
GOOGLESITEMAP_DEPTH, and so on.
Extensions are loaded and activated at startup by instantiating a single instance of the extension class. Therefore, all the extension initialization code must be performed in the class
To make an extension available, add it to the
EXTENSIONS setting in your Scrapy settings. In
EXTENSIONS, each extension is represented by a string: the full Python path to the extension’s class name. For example:
As you can see, the
EXTENSIONS setting is a dict where the keys are the extension paths, and their values are the orders, which define the extension loading order. The
EXTENSIONS setting is merged with the
EXTENSIONS_BASE setting defined in Scrapy (and not meant to be overridden) and then sorted by order to get the final sorted list of enabled extensions.
As extensions typically do not depend on each other, their loading order is irrelevant in most cases. This is why the
EXTENSIONS_BASE setting defines all extensions with the same order (
0). However, this feature can be exploited if you need to add an extension which depends on other extensions already loaded.
Not all available extensions will be enabled. Some of them usually depend on a particular setting. For example, the HTTP Cache extension is available by default but disabled unless the
HTTPCACHE_ENABLED setting is set.
In order to disable an extension that comes enabled by default (i.e. those included in the
EXTENSIONS_BASE setting) you must set its order to
None. For example:
Each extension is a Python class. The main entry point for a Scrapy extension (this also includes middlewares and pipelines) is the
from_crawler class method which receives a
Crawler instance. Through the Crawler object you can access settings, signals, stats, and also control the crawling behaviour.
Typically, extensions connect to signals and perform tasks triggered by them.
Finally, if the
from_crawler method raises the
NotConfigured exception, the extension will be disabled. Otherwise, the extension will be enabled.
Here we will implement a simple extension to illustrate the concepts described in the previous section. This extension will log a message every time:
a spider is opened
a spider is closed
a specific number of items are scraped
The extension will be enabled through the
MYEXT_ENABLED setting and the number of items will be specified through the
Here is the code of such extension:
from scrapy import signals
from scrapy.exceptions importNotConfigured
logger = logging.getLogger(__name__)
def __init__(self, item_count):
self.item_count = item_count
def from_crawler(cls, crawler):
# first check if the extension should be enabled and raise
# NotConfigured otherwise
# get the number of items from settings
item_count = crawler.settings.getint('MYEXT_ITEMCOUNT',1000)
# instantiate the extension object
ext = cls(item_count)
# connect the extension object to signals
# return the extension object
def spider_opened(self, spider):
logger.info("opened spider %s", spider.name)
def spider_closed(self, spider):
logger.info("closed spider %s", spider.name)
def item_scraped(self, item, spider):
ifself.items_scraped %self.item_count ==0:
logger.info("scraped %d items",self.items_scraped)
Log basic stats like crawled pages and scraped items.
Enable the collection of core statistics, provided the stats collection is enabled (see Stats Collection).
Provides a telnet console for getting into a Python interpreter inside the currently running Scrapy process, which can be very useful for debugging.
This extension does not work in Windows.
Monitors the memory used by the Scrapy process that runs the spider and:
sends a notification e-mail when it exceeds a certain value
closes the spider when it exceeds a certain value
The notification e-mails can be triggered when a certain warning value is reached (
MEMUSAGE_WARNING_MB) and when the maximum value is reached (
MEMUSAGE_LIMIT_MB) which will also cause the spider to be closed and the Scrapy process to be terminated.
This extension is enabled by the
MEMUSAGE_ENABLED setting and can be configured with the following settings:
An extension for debugging memory usage. It collects information about:
objects uncollected by the Python garbage collector
objects left alive that shouldn’t. For more info, see Debugging memory leaks with trackref
To enable this extension, turn on the
MEMDEBUG_ENABLED setting. The info will be stored in the stats.
Closes a spider automatically when some conditions are met, using a specific closing reason for each condition.
The conditions for closing a spider can be configured through the following settings:
When a certain closing condition is met, requests which are currently in the downloader queue (up to
CONCURRENT_REQUESTS requests) are still processed.
An integer which specifies a number of seconds. If the spider remains open for more than that number of second, it will be automatically closed with the reason
closespider_timeout. If zero (or non set), spiders won’t be closed by timeout.
An integer which specifies a number of items. If the spider scrapes more than that amount and those items are passed by the item pipeline, the spider will be closed with the reason
closespider_itemcount. If zero (or non set), spiders won’t be closed by number of passed items.
An integer which specifies the maximum number of responses to crawl. If the spider crawls more than that, the spider will be closed with the reason
closespider_pagecount. If zero (or non set), spiders won’t be closed by number of crawled responses.
An integer which specifies the maximum number of errors to receive before closing the spider. If the spider generates more than that number of errors, it will be closed with the reason
closespider_errorcount. If zero (or non set), spiders won’t be closed by number of errors.
This simple extension can be used to send a notification e-mail every time a domain has finished scraping, including the Scrapy stats collected. The email will be sent to all recipients specified in the
engine status (using
live references (see Debugging memory leaks with trackref)
stack trace of all threads
After the stack trace and engine status is dumped, the Scrapy process continues running normally.
There are at least two ways to send Scrapy the SIGQUIT signal:
By pressing Ctrl-while a Scrapy process is running (Linux only?)
By running this command (assuming
<pid>is the process id of the Scrapy process):
kill -QUIT <pid>
For more info see Debugging in Python.
This extension only works on POSIX-compliant platforms (i.e. not Windows).