Feed exports

New in version 0.10.

One of the most frequently required features when implementing scrapers isbeing able to store the scraped data properly and, quite often, that meansgenerating an “export file” with the scraped data (commonly called “exportfeed”) to be consumed by other systems.

Scrapy provides this functionality out of the box with the Feed Exports, whichallows you to generate a feed with the scraped items, using multipleserialization formats and storage backends.

Serialization formats

For serializing the scraped data, the feed exports use the Item exporters. These formats are supported out of the box:

But you can also extend the supported format through theFEED_EXPORTERS setting.


JSON lines


  • FEED_FORMAT: csv
  • Exporter used: CsvItemExporter
  • To specify columns to export and their order useFEED_EXPORT_FIELDS. Other feed exporters can also use thisoption, but it is important for CSV because unlike many other exportformats CSV uses a fixed header.





When using the feed exports you define where to store the feed using a URI(through the FEED_URI setting). The feed exports supports multiplestorage backend types which are defined by the URI scheme.

The storages backends supported out of the box are:

Some storage backends may be unavailable if the required external libraries arenot available. For example, the S3 backend is only available if the botocorelibrary is installed.

Storage URI parameters

The storage URI can also contain parameters that get replaced when the feed isbeing created. These parameters are:

  • %(time)s - gets replaced by a timestamp when the feed is being created
  • %(name)s - gets replaced by the spider name

Any other named parameter gets replaced by the spider attribute of the samename. For example, %(site_id)s would get replaced by the spider.site_idattribute the moment the feed is being created.

Here are some examples to illustrate:

  • Store in FTP using one directory per spider:
    • ftp://user:password@ftp.example.com/scraping/feeds/%(name)s/%(time)s.json
  • Store in S3 using one directory per spider:
    • s3://mybucket/scraping/feeds/%(name)s/%(time)s.json

Storage backends

Local filesystem

The feeds are stored in the local filesystem.

  • URI scheme: file
  • Example URI: file:///tmp/export.csv
  • Required external libraries: none

Note that for the local filesystem storage (only) you can omit the scheme ifyou specify an absolute path like /tmp/export.csv. This only works on Unixsystems though.


The feeds are stored in a FTP server.

  • URI scheme: ftp
  • Example URI: ftp://user:pass@ftp.example.com/path/to/export.csv
  • Required external libraries: none

FTP supports two different connection modes: active or passive. Scrapy uses the passive connectionmode by default. To use the active connection mode instead, set theFEED_STORAGE_FTP_ACTIVE setting to True.


The feeds are stored on Amazon S3.

  • URI scheme: s3
  • Example URIs:
    • s3://mybucket/path/to/export.csv
    • s3://aws_key:aws_secret@mybucket/path/to/export.csv
  • Required external libraries: botocore

The AWS credentials can be passed as user/password in the URI, or they can bepassed through the following settings:

You can also define a custom ACL for exported feeds using this setting:

Standard output

The feeds are written to the standard output of the Scrapy process.

  • URI scheme: stdout
  • Example URI: stdout:
  • Required external libraries: none


These are the settings used for configuring the feed exports:


Default: None

The URI of the export feed. See Storage backends forsupported URI schemes.

This setting is required for enabling the feed exports.

Changed in version 2.0: Added pathlib.Path support.


The serialization format to be used for the feed. SeeSerialization formats for possible values.


Default: None

The encoding to be used for the feed.

If unset or set to None (default) it uses UTF-8 for everything except JSON output,which uses safe numeric encoding (\uXXXX sequences) for historic reasons.

Use utf-8 if you want UTF-8 for JSON too.


Default: None

A list of fields to export, optional.Example: FEED_EXPORT_FIELDS = ["foo", "bar", "baz"].

Use FEED_EXPORT_FIELDS option to define fields to export and their order.

When FEED_EXPORT_FIELDS is empty or None (default), Scrapy uses fieldsdefined in dicts or Item subclasses a spider is yielding.

If an exporter requires a fixed set of fields (this is the case forCSV export format) and FEED_EXPORT_FIELDSis empty or None, then Scrapy tries to infer field names from theexported data - currently it uses field names from the first item.


Default: 0

Amount of spaces used to indent the output on each level. If FEED_EXPORT_INDENTis a non-negative integer, then array elements and object members will be pretty-printedwith that indent level. An indent level of 0 (the default), or negative,will put each item on a new line. None selects the most compact representation.

Currently implemented only by JsonItemExporterand XmlItemExporter, i.e. when you are exportingto .json or .xml.


Default: False

Whether to export empty feeds (i.e. feeds with no items).


Default: {}

A dict containing additional feed storage backends supported by your project.The keys are URI schemes and the values are paths to storage classes.


Default: False

Whether to use the active connection mode when exporting feeds to an FTP server(True) or use the passive connection mode instead (False, default).

For information about FTP connection modes, see What is the difference betweenactive and passive FTP?.


Default: '' (empty string)

A string containing a custom ACL for feeds exported to Amazon S3 by your project.

For a complete list of available values, access the Canned ACL section on Amazon S3 docs.



  1. {
  2. '': 'scrapy.extensions.feedexport.FileFeedStorage',
  3. 'file': 'scrapy.extensions.feedexport.FileFeedStorage',
  4. 'stdout': 'scrapy.extensions.feedexport.StdoutFeedStorage',
  5. 's3': 'scrapy.extensions.feedexport.S3FeedStorage',
  6. 'ftp': 'scrapy.extensions.feedexport.FTPFeedStorage',
  7. }

A dict containing the built-in feed storage backends supported by Scrapy. Youcan disable any of these backends by assigning None to their URI scheme inFEED_STORAGES. E.g., to disable the built-in FTP storage backend(without replacement), place this in your settings.py:

  2. 'ftp': None,
  3. }


Default: {}

A dict containing additional exporters supported by your project. The keys areserialization formats and the values are paths to Item exporter classes.



  1. {
  2. 'json': 'scrapy.exporters.JsonItemExporter',
  3. 'jsonlines': 'scrapy.exporters.JsonLinesItemExporter',
  4. 'jl': 'scrapy.exporters.JsonLinesItemExporter',
  5. 'csv': 'scrapy.exporters.CsvItemExporter',
  6. 'xml': 'scrapy.exporters.XmlItemExporter',
  7. 'marshal': 'scrapy.exporters.MarshalItemExporter',
  8. 'pickle': 'scrapy.exporters.PickleItemExporter',
  9. }

A dict containing the built-in feed exporters supported by Scrapy. You candisable any of these exporters by assigning None to their serializationformat in FEED_EXPORTERS. E.g., to disable the built-in CSV exporter(without replacement), place this in your settings.py:

  2. 'csv': None,
  3. }