在scrapy的settings.py文件中,USER_AGENT
像这样:
USER_AGENT = 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.101 Safari/537.36'
如果要多个USER_AGENT
切换使用,应该怎么做呢?
用Faker 见 http://imchenkun.com
看这个库 https://github.com/cnu/scrapy...
从源代码来看主要的实现是
def process_request(self, request, spider):
user_agent = random.choice(self.user_agent_list)
if user_agent:
request.headers.setdefault('User-Agent', user_agent)