Command line tool

New in version 0.10.

Scrapy is controlled through the scrapy command-line tool, to be referredhere as the “Scrapy tool” to differentiate it from the sub-commands, which wejust call “commands” or “Scrapy commands”.

The Scrapy tool provides several commands, for multiple purposes, and each oneaccepts a different set of arguments and options.

(The scrapy deploy command has been removed in 1.0 in favor of thestandalone scrapyd-deploy. See Deploying your project.)

Configuration settings

Scrapy will look for configuration parameters in ini-style scrapy.cfg filesin standard locations:

  • /etc/scrapy.cfg or c:\scrapy\scrapy.cfg (system-wide),
  • ~/.config/scrapy.cfg ($XDG_CONFIG_HOME) and ~/.scrapy.cfg ($HOME)for global (user-wide) settings, and
  • scrapy.cfg inside a Scrapy project’s root (see next section).Settings from these files are merged in the listed order of preference:user-defined values have higher priority than system-wide defaultsand project-wide settings will override all others, when defined.

Scrapy also understands, and can be configured through, a number of environmentvariables. Currently these are:

Default structure of Scrapy projects

Before delving into the command-line tool and its sub-commands, let’s firstunderstand the directory structure of a Scrapy project.

Though it can be modified, all Scrapy projects have the same filestructure by default, similar to this:

  1. scrapy.cfg
  2. myproject/
  3.     __init__.py
  4.     items.py
  5.     middlewares.py
  6.     pipelines.py
  7.     settings.py
  8.     spiders/
  9.         __init__.py
  10.         spider1.py
  11.         spider2.py
  12.         ...

The directory where the scrapy.cfg file resides is known as the projectroot directory. That file contains the name of the python module that definesthe project settings. Here is an example:

  1. [settings]
  2. default = myproject.settings

Sharing the root directory between projects

A project root directory, the one that contains the scrapy.cfg, may beshared by multiple Scrapy projects, each with its own settings module.

In that case, you must define one or more aliases for those settings modulesunder [settings] in your scrapy.cfg file:

  1. [settings]
  2. default = myproject1.settings
  3. project1 = myproject1.settings
  4. project2 = myproject2.settings

By default, the scrapy command-line tool will use the default settings.Use the SCRAPY_PROJECT environment variable to specify a different projectfor scrapy to use:

  1. $ scrapy settings --get BOT_NAME
  2. Project 1 Bot
  3. $ export SCRAPY_PROJECT=project2
  4. $ scrapy settings --get BOT_NAME
  5. Project 2 Bot

Using the scrapy tool

You can start by running the Scrapy tool with no arguments and it will printsome usage help and the available commands:

  1. Scrapy X.Y - no active project
  2.  
  3. Usage:
  4.   scrapy <command> [options] [args]
  5.  
  6. Available commands:
  7.   crawl         Run a spider
  8.   fetch         Fetch a URL using the Scrapy downloader
  9. [...]

The first line will print the currently active project if you’re inside aScrapy project. In this example it was run from outside a project. If run from insidea project it would have printed something like this:

  1. Scrapy X.Y - project: myproject
  2.  
  3. Usage:
  4.   scrapy <command> [options] [args]
  5.  
  6. [...]

Creating projects

The first thing you typically do with the scrapy tool is create your Scrapyproject:

  1. scrapy startproject myproject [project_dir]

That will create a Scrapy project under the project_dir directory.If project_dir wasn’t specified, project_dir will be the same as myproject.

Next, you go inside the new project directory:

  1. cd project_dir

And you’re ready to use the scrapy command to manage and control yourproject from there.

Controlling projects

You use the scrapy tool from inside your projects to control and managethem.

For example, to create a new spider:

  1. scrapy genspider mydomain mydomain.com

Some Scrapy commands (like crawl) must be run from inside a Scrapyproject. See the commands reference below for moreinformation on which commands must be run from inside projects, and which not.

Also keep in mind that some commands may have slightly different behaviourswhen running them from inside projects. For example, the fetch command will usespider-overridden behaviours (such as the user_agent attribute to overridethe user-agent) if the url being fetched is associated with some specificspider. This is intentional, as the fetch command is meant to be used tocheck how spiders are downloading pages.

Available tool commands

This section contains a list of the available built-in commands with adescription and some usage examples. Remember, you can always get more infoabout each command by running:

  1. scrapy <command> -h

And you can see all available commands with:

  1. scrapy -h

There are two kinds of commands, those that only work from inside a Scrapyproject (Project-specific commands) and those that also work without an activeScrapy project (Global commands), though they may behave slightly differentwhen running from inside a project (as they would use the project overriddensettings).

Global commands:

Project-only commands:

startproject

  • Syntax: scrapy startproject <project_name> [project_dir]
  • Requires project: no

Creates a new Scrapy project named project_name, under the project_dirdirectory.If project_dir wasn’t specified, project_dir will be the same as project_name.

Usage example:

  1. $ scrapy startproject myproject

genspider

  • Syntax: scrapy genspider [-t template] <name> <domain>
  • Requires project: no

Create a new spider in the current folder or in the current project’s spiders folder, if called from inside a project. The <name> parameter is set as the spider’s name, while <domain> is used to generate the allowed_domains and start_urls spider’s attributes.

Usage example:

  1. $ scrapy genspider -l
  2. Available templates:
  3.   basic
  4.   crawl
  5.   csvfeed
  6.   xmlfeed
  7.  
  8. $ scrapy genspider example example.com
  9. Created spider 'example' using template 'basic'
  10.  
  11. $ scrapy genspider -t crawl scrapyorg scrapy.org
  12. Created spider 'scrapyorg' using template 'crawl'

This is just a convenience shortcut command for creating spiders based onpre-defined templates, but certainly not the only way to create spiders. Youcan just create the spider source code files yourself, instead of using thiscommand.

crawl

  • Syntax: scrapy crawl <spider>
  • Requires project: yes

Start crawling using a spider.

Usage examples:

  1. $ scrapy crawl myspider
  2. [ ... myspider starts crawling ... ]

check

  • Syntax: scrapy check [-l] <spider>
  • Requires project: yes

Run contract checks.

Usage examples:

  1. $ scrapy check -l
  2. first_spider
  3.   * parse
  4.   * parse_item
  5. second_spider
  6.   * parse
  7.   * parse_item
  8.  
  9. $ scrapy check
  10. [FAILED] first_spider:parse_item
  11. >>> 'RetailPricex' field is missing
  12.  
  13. [FAILED] first_spider:parse
  14. >>> Returned 92 requests, expected 0..4

list

  • Syntax: scrapy list
  • Requires project: yes

List all available spiders in the current project. The output is one spider perline.

Usage example:

  1. $ scrapy list
  2. spider1
  3. spider2

edit

  • Syntax: scrapy edit <spider>
  • Requires project: yes

Edit the given spider using the editor defined in the EDITOR environmentvariable or (if unset) the EDITOR setting.

This command is provided only as a convenience shortcut for the most commoncase, the developer is of course free to choose any tool or IDE to write anddebug spiders.

Usage example:

  1. $ scrapy edit spider1

fetch

  • Syntax: scrapy fetch <url>
  • Requires project: no

Downloads the given URL using the Scrapy downloader and writes the contents tostandard output.

The interesting thing about this command is that it fetches the page how thespider would download it. For example, if the spider has a USER_AGENTattribute which overrides the User Agent, it will use that one.

So this command can be used to “see” how your spider would fetch a certain page.

If used outside a project, no particular per-spider behaviour would be appliedand it will just use the default Scrapy downloader settings.

Supported options:

  • —spider=SPIDER: bypass spider autodetection and force use of specific spider
  • —headers: print the response’s HTTP headers instead of the response’s body
  • —no-redirect: do not follow HTTP 3xx redirects (default is to follow them)

Usage examples:

  1. $ scrapy fetch --nolog http://www.example.com/some/page.html
  2. [ ... html content here ... ]
  3.  
  4. $ scrapy fetch --nolog --headers http://www.example.com/
  5. {'Accept-Ranges': ['bytes'],
  6.  'Age': ['1263   '],
  7.  'Connection': ['close     '],
  8.  'Content-Length': ['596'],
  9.  'Content-Type': ['text/html; charset=UTF-8'],
  10.  'Date': ['Wed, 18 Aug 2010 23:59:46 GMT'],
  11.  'Etag': ['"573c1-254-48c9c87349680"'],
  12.  'Last-Modified': ['Fri, 30 Jul 2010 15:30:18 GMT'],
  13.  'Server': ['Apache/2.2.3 (CentOS)']}

view

  • Syntax: scrapy view <url>
  • Requires project: no

Opens the given URL in a browser, as your Scrapy spider would “see” it.Sometimes spiders see pages differently from regular users, so this can be usedto check what the spider “sees” and confirm it’s what you expect.

Supported options:

  • —spider=SPIDER: bypass spider autodetection and force use of specific spider
  • —no-redirect: do not follow HTTP 3xx redirects (default is to follow them)

Usage example:

  1. $ scrapy view http://www.example.com/some/page.html
  2. [ ... browser starts ... ]

shell

  • Syntax: scrapy shell [url]
  • Requires project: no

Starts the Scrapy shell for the given URL (if given) or empty if no URL isgiven. Also supports UNIX-style local file paths, either relative with./ or ../ prefixes or absolute file paths.See Scrapy shell for more info.

Supported options:

  • —spider=SPIDER: bypass spider autodetection and force use of specific spider
  • -c code: evaluate the code in the shell, print the result and exit
  • —no-redirect: do not follow HTTP 3xx redirects (default is to follow them);this only affects the URL you may pass as argument on the command line;once you are inside the shell, fetch(url) will still follow HTTP redirects by default.

Usage example:

  1. $ scrapy shell http://www.example.com/some/page.html
  2. [ ... scrapy shell starts ... ]
  3.  
  4. $ scrapy shell --nolog http://www.example.com/ -c '(response.status, response.url)'
  5. (200, 'http://www.example.com/')
  6.  
  7. # shell follows HTTP redirects by default
  8. $ scrapy shell --nolog http://httpbin.org/redirect-to?url=http%3A%2F%2Fexample.com%2F -c '(response.status, response.url)'
  9. (200, 'http://example.com/')
  10.  
  11. # you can disable this with --no-redirect
  12. # (only for the URL passed as command line argument)
  13. $ scrapy shell --no-redirect --nolog http://httpbin.org/redirect-to?url=http%3A%2F%2Fexample.com%2F -c '(response.status, response.url)'
  14. (302, 'http://httpbin.org/redirect-to?url=http%3A%2F%2Fexample.com%2F')

parse

  • Syntax: scrapy parse <url> [options]
  • Requires project: yes

Fetches the given URL and parses it with the spider that handles it, using themethod passed with the —callback option, or parse if not given.

Supported options:

  • —spider=SPIDER: bypass spider autodetection and force use of specific spider
  • —a NAME=VALUE: set spider argument (may be repeated)
  • —callback or -c: spider method to use as callback for parsing theresponse
  • —meta or -m: additional request meta that will be passed to the callbackrequest. This must be a valid json string. Example: –meta=’{“foo” : “bar”}’
  • —cbkwargs: additional keyword arguments that will be passed to the callback.This must be a valid json string. Example: –cbkwargs=’{“foo” : “bar”}’
  • —pipelines: process items through pipelines
  • —rules or -r: use CrawlSpiderrules to discover the callback (i.e. spider method) to use for parsing theresponse
  • —noitems: don’t show scraped items
  • —nolinks: don’t show extracted links
  • —nocolour: avoid using pygments to colorize the output
  • —depth or -d: depth level for which the requests should be followedrecursively (default: 1)
  • —verbose or -v: display information for each depth level

Usage example:

  1. $ scrapy parse http://www.example.com/ -c parse_item
  2. [ ... scrapy log lines crawling example.com spider ... ]
  3.  
  4. >>> STATUS DEPTH LEVEL 1 <<<
  5. # Scraped Items  ------------------------------------------------------------
  6. [{'name': 'Example item',
  7.  'category': 'Furniture',
  8.  'length': '12 cm'}]
  9.  
  10. # Requests  -----------------------------------------------------------------
  11. []

settings

  • Syntax: scrapy settings [options]
  • Requires project: no

Get the value of a Scrapy setting.

If used inside a project it’ll show the project setting value, otherwise it’llshow the default Scrapy value for that setting.

Example usage:

  1. $ scrapy settings --get BOT_NAME
  2. scrapybot
  3. $ scrapy settings --get DOWNLOAD_DELAY
  4. 0

runspider

  • Syntax: scrapy runspider <spider_file.py>
  • Requires project: no

Run a spider self-contained in a Python file, without having to create aproject.

Example usage:

  1. $ scrapy runspider myspider.py
  2. [ ... spider starts crawling ... ]

version

  • Syntax: scrapy version [-v]
  • Requires project: no

Prints the Scrapy version. If used with -v it also prints Python, Twistedand Platform info, which is useful for bug reports.

bench

New in version 0.17.

  • Syntax: scrapy bench
  • Requires project: no

Run a quick benchmark test. Benchmarking.

Custom project commands

You can also add your custom project commands by using theCOMMANDS_MODULE setting. See the Scrapy commands inscrapy/commands for examples on how to implement your commands.

COMMANDS_MODULE

Default: '' (empty string)

A module to use for looking up custom Scrapy commands. This is used to add customcommands for your Scrapy project.

Example:

  1. COMMANDS_MODULE = 'mybot.commands'

Register commands via setup.py entry points

Note

This is an experimental feature, use with caution.

You can also add Scrapy commands from an external library by adding ascrapy.commands section in the entry points of the library setup.pyfile.

The following example adds my_command command:

  1. from setuptools import setup, find_packages
  2.  
  3. setup(name='scrapy-mymodule',
  4.   entry_points={
  5.     'scrapy.commands': [
  6.       'my_command=my_scrapy_module.commands:MyCommand',
  7.     ],
  8.   },
  9.  )