Extensions

The extensions framework provides a mechanism for inserting your owncustom functionality into Scrapy.

Extensions are just regular classes that are instantiated at Scrapy startup,when extensions are initialized.

Extension settings

Extensions use the Scrapy settings to manage theirsettings, just like any other Scrapy code.

It is customary for extensions to prefix their settings with their own name, toavoid collision with existing (and future) extensions. For example, ahypothetic extension to handle Google Sitemaps would use settings likeGOOGLESITEMAP_ENABLED, GOOGLESITEMAP_DEPTH, and so on.

Loading & activating extensions

Extensions are loaded and activated at startup by instantiating a singleinstance of the extension class. Therefore, all the extension initializationcode must be performed in the class init method.

To make an extension available, add it to the EXTENSIONS setting inyour Scrapy settings. In EXTENSIONS, each extension is representedby a string: the full Python path to the extension’s class name. For example:

EXTENSIONS = {
    'scrapy.extensions.corestats.CoreStats': 500,
    'scrapy.extensions.telnet.TelnetConsole': 500,
}

As you can see, the EXTENSIONS setting is a dict where the keys arethe extension paths, and their values are the orders, which define theextension loading order. The EXTENSIONS setting is merged with theEXTENSIONS_BASE setting defined in Scrapy (and not meant to beoverridden) and then sorted by order to get the final sorted list of enabledextensions.

As extensions typically do not depend on each other, their loading order isirrelevant in most cases. This is why the EXTENSIONS_BASE settingdefines all extensions with the same order (0). However, this feature canbe exploited if you need to add an extension which depends on other extensionsalready loaded.

Available, enabled and disabled extensions

Not all available extensions will be enabled. Some of them usually depend on aparticular setting. For example, the HTTP Cache extension is available by defaultbut disabled unless the HTTPCACHE_ENABLED setting is set.

Disabling an extension

In order to disable an extension that comes enabled by default (i.e. thoseincluded in the EXTENSIONS_BASE setting) you must set its order toNone. For example:

EXTENSIONS = {
    'scrapy.extensions.corestats.CoreStats': None,
}

Writing your own extension

Each extension is a Python class. The main entry point for a Scrapy extension(this also includes middlewares and pipelines) is the from_crawlerclass method which receives a Crawler instance. Through the Crawler objectyou can access settings, signals, stats, and also control the crawling behaviour.

Typically, extensions connect to signals and performtasks triggered by them.

Finally, if the from_crawler method raises theNotConfigured exception, the extension will bedisabled. Otherwise, the extension will be enabled.

Sample extension

Here we will implement a simple extension to illustrate the concepts describedin the previous section. This extension will log a message every time:

• a spider is opened
• a spider is closed
• a specific number of items are scraped

The extension will be enabled through the MYEXT_ENABLED setting and thenumber of items will be specified through the MYEXT_ITEMCOUNT setting.

Here is the code of such extension:

import logging
from scrapy import signals
from scrapy.exceptions import NotConfigured
 
logger = logging.getLogger(__name__)
 
class SpiderOpenCloseLogging(object):
 
    def __init__(self, item_count):
        self.item_count = item_count
        self.items_scraped = 0
 
    @classmethod
    def from_crawler(cls, crawler):
        # first check if the extension should be enabled and raise
        # NotConfigured otherwise
        if not crawler.settings.getbool('MYEXT_ENABLED'):
            raise NotConfigured
 
        # get the number of items from settings
        item_count = crawler.settings.getint('MYEXT_ITEMCOUNT', 1000)
 
        # instantiate the extension object
        ext = cls(item_count)
 
        # connect the extension object to signals
        crawler.signals.connect(ext.spider_opened, signal=signals.spider_opened)
        crawler.signals.connect(ext.spider_closed, signal=signals.spider_closed)
        crawler.signals.connect(ext.item_scraped, signal=signals.item_scraped)
 
        # return the extension object
        return ext
 
    def spider_opened(self, spider):
        logger.info("opened spider %s", spider.name)
 
    def spider_closed(self, spider):
        logger.info("closed spider %s", spider.name)
 
    def item_scraped(self, item, spider):
        self.items_scraped += 1
        if self.items_scraped % self.item_count == 0:
            logger.info("scraped %d items", self.items_scraped)

Built-in extensions reference

General purpose extensions

Log Stats extension

• class scrapy.extensions.logstats.LogStats[source]

Log basic stats like crawled pages and scraped items.

Core Stats extension

• class scrapy.extensions.corestats.CoreStats[source]

Enable the collection of core statistics, provided the stats collection isenabled (see Stats Collection).

Telnet console extension

• class scrapy.extensions.telnet.TelnetConsole[source]

Provides a telnet console for getting into a Python interpreter inside thecurrently running Scrapy process, which can be very useful for debugging.

The telnet console must be enabled by the TELNETCONSOLE_ENABLEDsetting, and the server will listen in the port specified inTELNETCONSOLE_PORT.

Memory usage extension

• class scrapy.extensions.memusage.MemoryUsage[source]

Note

This extension does not work in Windows.

Monitors the memory used by the Scrapy process that runs the spider and:

• sends a notification e-mail when it exceeds a certain value
• closes the spider when it exceeds a certain valueThe notification e-mails can be triggered when a certain warning value isreached (MEMUSAGE_WARNING_MB) and when the maximum value is reached(MEMUSAGE_LIMIT_MB) which will also cause the spider to be closedand the Scrapy process to be terminated.

This extension is enabled by the MEMUSAGE_ENABLED setting andcan be configured with the following settings:

• MEMUSAGE_LIMIT_MB
• MEMUSAGE_WARNING_MB
• MEMUSAGE_NOTIFY_MAIL
• MEMUSAGE_CHECK_INTERVAL_SECONDS

Memory debugger extension

• class scrapy.extensions.memdebug.MemoryDebugger[source]

An extension for debugging memory usage. It collects information about:

• objects uncollected by the Python garbage collector
• objects left alive that shouldn’t. For more info, see Debugging memory leaks with trackref

To enable this extension, turn on the MEMDEBUG_ENABLED setting. Theinfo will be stored in the stats.

Close spider extension

• class scrapy.extensions.closespider.CloseSpider[source]

Closes a spider automatically when some conditions are met, using a specificclosing reason for each condition.

The conditions for closing a spider can be configured through the followingsettings:

• CLOSESPIDER_TIMEOUT
• CLOSESPIDER_ITEMCOUNT
• CLOSESPIDER_PAGECOUNT
• CLOSESPIDER_ERRORCOUNT

CLOSESPIDER_TIMEOUT

Default: 0

An integer which specifies a number of seconds. If the spider remains open formore than that number of second, it will be automatically closed with thereason closespider_timeout. If zero (or non set), spiders won’t be closed bytimeout.

CLOSESPIDER_ITEMCOUNT

Default: 0

An integer which specifies a number of items. If the spider scrapes more thanthat amount and those items are passed by the item pipeline, thespider will be closed with the reason closespider_itemcount.Requests which are currently in the downloader queue (up toCONCURRENT_REQUESTS requests) are still processed.If zero (or non set), spiders won’t be closed by number of passed items.

CLOSESPIDER_PAGECOUNT

New in version 0.11.

Default: 0

An integer which specifies the maximum number of responses to crawl. If the spidercrawls more than that, the spider will be closed with the reasonclosespider_pagecount. If zero (or non set), spiders won’t be closed bynumber of crawled responses.

CLOSESPIDER_ERRORCOUNT

New in version 0.11.

Default: 0

An integer which specifies the maximum number of errors to receive beforeclosing the spider. If the spider generates more than that number of errors,it will be closed with the reason closespider_errorcount. If zero (or nonset), spiders won’t be closed by number of errors.

StatsMailer extension

• class scrapy.extensions.statsmailer.StatsMailer[source]

This simple extension can be used to send a notification e-mail every time adomain has finished scraping, including the Scrapy stats collected. The emailwill be sent to all recipients specified in the STATSMAILER_RCPTSsetting.

Debugging extensions

Stack trace dump extension

• class scrapy.extensions.debug.StackTraceDump[source]

Dumps information about the running process when a SIGQUIT or SIGUSR2signal is received. The information dumped is the following:

• engine status (using scrapy.utils.engine.get_engine_status())
• live references (see Debugging memory leaks with trackref)
• stack trace of all threadsAfter the stack trace and engine status is dumped, the Scrapy process continuesrunning normally.

This extension only works on POSIX-compliant platforms (i.e. not Windows),because the SIGQUIT and SIGUSR2 signals are not available on Windows.

There are at least two ways to send Scrapy the SIGQUIT signal:

• By pressing Ctrl-while a Scrapy process is running (Linux only?)

• By running this command (assuming <pid> is the process id of the Scrapyprocess):

kill -QUIT <pid>

Debugger extension

• class scrapy.extensions.debug.Debugger[source]

Invokes a Python debugger inside a running Scrapy process when a SIGUSR2signal is received. After the debugger is exited, the Scrapy process continuesrunning normally.

For more info see Debugging in Python.

This extension only works on POSIX-compliant platforms (i.e. not Windows).