Requests and Responses

Scrapy uses Request and Response objects for crawling websites.

Typically, Request objects are generated in the spiders and passacross the system until they reach the Downloader, which executes the requestand returns a Response object which travels back to the spider thatissued the request.

Both Request and Response classes have subclasses which addfunctionality not required in the base classes. These are describedbelow in Request subclasses andResponse subclasses.

Request objects

class scrapy.http.Request(url[, callback, method='GET', headers, body, cookies, meta, encoding='utf-8', priority=0, dont_filter=False, errback])

A Request object represents an HTTP request, which is usuallygenerated in the Spider and executed by the Downloader, and thus generatinga Response.

参数:

- url (string) – the URL of this request - callback (callable) – the function that will be called with the response of thisrequest (once its downloaded) as its first parameter. For more informationsee Passing additional data to callback functions below.If a Request doesn’t specify a callback, the spider’sparse() method will be used.Note that if exceptions are raised during processing, errback is called instead. - method (string) – the HTTP method of this request. Defaults to 'GET'. - meta (dict) – the initial values for the Request.meta attribute. Ifgiven, the dict passed in this parameter will be shallow copied. - body (str or unicode) – the request body. If a unicode is passed, then it’s encoded tostr using the encoding passed (which defaults to utf-8). Ifbody is not given,, an empty string is stored. Regardless of thetype of this argument, the final value stored will be a str (neverunicode or None). - headers (dict) – the headers of this request. The dict values can be strings(for single valued headers) or lists (for multi-valued headers). IfNone is passed as value, the HTTP header will not be sent at all. - cookies (dict or list) – the request cookies. These can be sent in two forms. - Using a dict: requestwith_cookies = Request(url="http://www.example.com", cookies={'currency': 'USD', 'country': 'UY'}) - Using a list of dicts: request_with_cookies = Request(url="http://www.example.com", cookies=[{'name': 'currency', 'value': 'USD', 'domain': 'example.com', 'path': '/currency'}]) The latter form allows for customizing the domain and pathattributes of the cookie. This is only useful if the cookies are savedfor later requests. When some site returns cookies (in a response) those are stored in thecookies for that domain and will be sent again in future requests. That’sthe typical behaviour of any regular web browser. However, if, for somereason, you want to avoid merging with existing cookies you can instructScrapy to do so by setting the dont_merge_cookies key in theRequest.meta. Example of request without merging cookies: request_with_cookies = Request(url="http://www.example.com", cookies={'currency': 'USD', 'country': 'UY'}, meta={'dont_merge_cookies': True}) For more info see CookiesMiddleware. - encoding (_string) – the encoding of this request (defaults to 'utf-8').This encoding will be used to percent-encode the URL and to convert thebody to str (if given as unicode). - priority (int) – the priority of this request (defaults to 0).The priority is used by the scheduler to define the order used to processrequests. Requests with a higher priority value will execute earlier.Negative values are allowed in order to indicate relatively low-priority. - dont_filter (boolean) – indicates that this request should not be filtered bythe scheduler. This is used when you want to perform an identicalrequest multiple times, to ignore the duplicates filter. Use it withcare, or you will get into crawling loops. Default to False. - errback (callable) – a function that will be called if any exception wasraised while processing the request. This includes pages that failedwith 404 HTTP errors and such. It receives a Twisted Failure instanceas first parameter.

url

A string containing the URL of this request. Keep in mind that thisattribute contains the escaped URL, so it can differ from the URL passed inthe constructor.

This attribute is read-only. To change the URL of a Request usereplace().

method

A string representing the HTTP method in the request. This is guaranteed tobe uppercase. Example: "GET", "POST", "PUT", etc

headers

A dictionary-like object which contains the request headers.

body

A str that contains the request body.

This attribute is read-only. To change the body of a Request usereplace().

meta

A dict that contains arbitrary metadata for this request. This dict isempty for new Requests, and is usually populated by different Scrapycomponents (extensions, middlewares, etc). So the data contained in thisdict depends on the extensions you have enabled.

See Request.meta special keys for a list of special meta keysrecognized by Scrapy.

This dict is shallow copied when the request is cloned using thecopy() or replace() methods, and can also be accessed, in yourspider, from the response.meta attribute.

copy()

Return a new Request which is a copy of this Request. See also:Passing additional data to callback functions.

replace([url, method, headers, body, cookies, meta, encoding, dont_filter, callback, errback])

Return a Request object with the same members, except for those membersgiven new values by whichever keyword arguments are specified. Theattribute Request.meta is copied by default (unless a new valueis given in the meta argument). See alsoPassing additional data to callback functions.

Passing additional data to callback functions

The callback of a request is a function that will be called when the responseof that request is downloaded. The callback function will be called with thedownloaded Response object as its first argument.

Example:

def parse_page1(self, response):
    return scrapy.Request("http://www.example.com/some_page.html",
                          callback=self.parse_page2)
 
def parse_page2(self, response):
    # this would log http://www.example.com/some_page.html
    self.logger.info("Visited %s", response.url)

In some cases you may be interested in passing arguments to those callbackfunctions so you can receive the arguments later, in the second callback. Youcan use the Request.meta attribute for that.

Here’s an example of how to pass an item using this mechanism, to populatedifferent fields from different pages:

def parse_page1(self, response):
    item = MyItem()
    item['main_url'] = response.url
    request = scrapy.Request("http://www.example.com/some_page.html",
                             callback=self.parse_page2)
    request.meta['item'] = item
    return request
 
def parse_page2(self, response):
    item = response.meta['item']
    item['other_url'] = response.url
    return item

Request.meta special keys

The Request.meta attribute can contain any arbitrary data, but thereare some special keys recognized by Scrapy and its built-in extensions.

Those are:

• dont_redirect
• dont_retry
• handle_httpstatus_list
• handle_httpstatus_all
• dont_merge_cookies (see cookies parameter of Request constructor)
• cookiejardont_cache
• redirect_urls
• bindaddress
• dont_obey_robotstxt
• download_timeout
• download_maxsize
• proxy

bindaddress

The IP of the outgoing IP address to use for the performing the request.

download_timeout

The amount of time (in secs) that the downloader will wait before timing out.See also: DOWNLOAD_TIMEOUT.

Request subclasses

Here is the list of built-in Request subclasses. You can also subclassit to implement your own custom functionality.

FormRequest objects

The FormRequest class extends the base Request with functionality fordealing with HTML forms. It uses lxml.html forms to pre-populate formfields with form data from Response objects.

class scrapy.http.FormRequest(url[, formdata, …])

The FormRequest class adds a new argument to the constructor. Theremaining arguments are the same as for the Request class and arenot documented here.

参数:	formdata (dict or iterable of tuples) – is a dictionary (or iterable of (key, value) tuples)containing HTML Form data which will be url-encoded and assigned to thebody of the request.

The FormRequest objects support the following class method inaddition to the standard Request methods:

classmethod fromresponse(_response[, formname=None, formnumber=0, formdata=None, formxpath=None, clickdata=None, dont_click=False, …])

Returns a new FormRequest object with its form field valuespre-populated with those found in the HTML <form> element containedin the given response. For an example see使用FormRequest.from_response()方法模拟用户登录.

The policy is to automatically simulate a click, by default, on any formcontrol that looks clickable, like a <input type="submit">. Eventhough this is quite convenient, and often the desired behaviour,sometimes it can cause problems which could be hard to debug. Forexample, when working with forms that are filled and/or submitted usingjavascript, the default from_response() behaviour may not be themost appropriate. To disable this behaviour you can set thedontclick argument to True. Also, if you want to change thecontrol clicked (instead of disabling it) you can also use theclickdata argument.

参数:

- response (Response object) – the response containing a HTML form which will be usedto pre-populate the form fields - formname (_string) – if given, the form with name attribute set to this value will be used. - formxpath (string) – if given, the first form that matches the xpath will be used. - formnumber (integer) – the number of form to use, when the response containsmultiple forms. The first one (and also the default) is 0. - formdata (dict) – fields to override in the form data. If a field wasalready present in the response <form> element, its value isoverridden by the one passed in this parameter. - clickdata (dict) – attributes to lookup the control clicked. If it’s notgiven, the form data will be submitted simulating a click on thefirst clickable element. In addition to html attributes, the controlcan be identified by its zero-based index relative to othersubmittable inputs inside the form, via the nr attribute. - dont_click (boolean) – If True, the form data will be submitted withoutclicking in any element.

The other parameters of this class method are passed directly to theFormRequest constructor.


0.10.3 新版功能: The formname parameter.



0.17 新版功能: The formxpath parameter.

return [FormRequest(url="http://www.example.com/post/action",
                    formdata={'name': 'John Doe', 'age': '27'},
                    callback=self.after_post)]

import scrapy
 
class LoginSpider(scrapy.Spider):
    name = 'example.com'
    start_urls = ['http://www.example.com/users/login.php']
 
    def parse(self, response):
        return scrapy.FormRequest.from_response(
            response,
            formdata={'username': 'john', 'password': 'secret'},
            callback=self.after_login
        )
 
    def after_login(self, response):
        # check login succeed before going on
        if "authentication failed" in response.body:
            self.logger.error("Login failed")
            return
 
        # continue scraping with authenticated session...