我先尝试用scrapy登录一个网站，登录成功后，
定向到后缀名为.jsp文件都有模板，如
Crawled (200)
但是当我定向到
DEBUG: Retrying (failed 1 times): [
<twisted.python.failure.Failure >]这样后面为.do结尾的文件就总是retry
然后失败
然后我用谷歌浏览器的开发者工具抓包，发现它只有一个页面跳转，并没有其他页面
Remote Address:202.117.112.64:80
Request URL:http://example.com/jxpgXsAction.do?oper=listWj
Request Method:GET
Status Code:200 OK
Request Headersview source
Accept:text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,/;q=0.8
Accept-Encoding:gzip, deflate, sdch
Accept-Language:zh-CN,zh;q=0.8
Connection:keep-alive
Cookie:
User-Agent:Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.111 Safari/537.36
Query String Parametersview sourceview URL encoded
oper:listWj
Response Headersview source
Connection:Keep-Alive
Content-Type:text/html; charset=GBK
Date:Thu, 02 Jul 2015 13:51:09 GMT
Keep-Alive:timeout=8, max=498
Server:Apache
Transfer-Encoding:chunked
难道是因为这个文件根本就没有模板全都是js生成的数据显示所以scrapy抓取不到吗？
这是我spider跳转的代码
def menu(self, response):
print 'Menu arrive'
# print response.body
return [scrapy.Request('http://example.com/jxpgXsAction.do',
meta={'cookiejar': response.meta['cookiejar'],
},
callback=self.evalu,
dont_filter=True)]