1、下载设置代理ip
2、正常访问设置代理ip
下载设置代理ip
1、设置setting
DOWNLOADER_MIDDLEWARES = {
'enduction.middlewares.EnductionDownloaderMiddleware': 543,
}
2、编写代码
def process_request(self, request, spider):
request.meta["proxy"]=http:// +地址+端口号
正常访问设置代理ip
重写spider
def start_requests(self):
for url in self.start_urls:
yield scrapy.Request(url,meta={'proxy':http:// +地址+端口号},callback=self.parse)
