process_request(request, spider)
This method is called when each request is downloaded through the middleware.

process_request() must return one of them: None, Response, Request or raise IgnoreRequest.

If it returns to None, Scrapy will continue to process the request and execute the corresponding methods of other middleware until the appropriate download handler is called and the request is executed (its response is downloaded).

If it returns the Response object, Scrapy will not call any other process_request() or process_exception() method, or download the function accordingly; it will return the response. The process_response() method of the installed middleware is invoked when each response returns.

If it returns the Request object, Scrapy stops calling the process_request method and reschedules the returned request. When the newly returned request is executed, the corresponding middleware chain will be invoked according to the downloaded response.

If its raise has an IgnoreRequest exception, the process_exception() method of the installed download middleware will be invoked. If no method handles the exception, the request's errback(Request.errback) method is called. If no code handles the thrown exception, it is ignored and not recorded (unlike other exceptions).

Parameters:	
request (Request object) - Processing request
 spider (Spider object) - The spider corresponding to the request

process_response(request, response, spider)
process_request() must return one of the following: a Response object, a Request object, or an IgnoreRequest exception for raise.

If it returns a Response (which can be the same as the incoming response or a completely new object), the response will be processed by the process_response() method of other middleware in the chain.

If it returns a Request object, the middleware chain stops and the returned request is rescheduled for download. Processing is similar to what process_request() does when it returns request.

If it throws an IgnoreRequest exception, the request's errback(Request.errback) is called. If no code handles the thrown exception, it is ignored and not recorded (unlike other exceptions).

Parameters:	
request (Request object) - the corresponding request of response
 response (Response Object) - Responsed to be processed
 spider (Spider object) - spider corresponding to response
process_exception(request, exception, spider)
Scrapy calls process_exception() when an exception (including IgnoreRequest exception) is thrown by the download handler or process_request() download middleware.

process_exception() should return one of the following: None, a Response object, or a Request object.

If it returns to None, Scrapy will continue to handle the exception and then call the process_exception() method of the other middleware installed until all the middleware has been invoked, then the default exception handling is invoked.

If it returns a Response object, the process_response() method of the installed middleware chain is called. Scrapy will not call the process_exception() method of any other middleware.

If it returns a Request object, the returned request will be re-invoked for download. This stops the process_exception() method execution of the middleware as it returns a response.

Parameters:	
request (request object) - request that produces exceptions
 exception (Exception object) - Exception thrown
 spider (Spider object) - spider corresponding to request