通过这几个月以来对爬虫的基础库的研究和使用之后，个人觉得已经可以进一步拓展技能深度，学习当今流行的开源爬虫框架。当然，前期的调研工作需要做好，即了解下目前市场上的主流爬虫框架。
经过初步搜索，市面上流行的主要就Scrapy和Pyspider这两个框架，考虑到框架自身的知识深度以及将来分布式爬虫的开发与研究，我决定从scrapy入手，毕竟其具有高度的可定制性和可拓展性。
众所周知，scrapy主要由5个核心组件构成分别是 Engine，Scheduler，Downloader，Spider和Item Pipeline，此外还有两大中间件：Downloader middleware和Spider middlewares。
此处就不涉及scrapy的具体机制了，强烈推荐大家一篇解析scrapy代码的好文章，作者从架构概览，运行入口，核心组件初始化和核心抓取流程这四个方向入手，向我们详细解说了scrapy的运行机制。网址如下：
http://kaito-kidd.com/2016/11/01/scrapy-code-analyze-architecture/
好了，直奔主题，本次我们分享的是起点小说网的爬虫。
item定义如下：
import scrapy
class QidianItem(scrapy.Item):
# define the fields for your item here like:
# name = scrapy.Field()
author = scrapy.Field()
book_id = scrapy.Field()
book_type = scrapy.Field()
book_sub_type = scrapy.Field()
book_name = scrapy.Field()
book_url = scrapy.Field()
total_words = scrapy.Field()
click_count = scrapy.Field()
recommand_count = scrapy.Field()
book_status = scrapy.Field()
rank_score = scrapy.Field()
rank_ppl_involved = scrapy.Field()
last_upload_date = scrapy.Field()
Spider:
# -*- coding: utf-8 -*-
import scrapy
from scrapy import Selector
from qidian.items import QidianItem
from urllib.parse import urlencode
import json
class QidianSpider(scrapy.Spider):
name = 'Qidian'
allowed_domain=['www.qidian.com',
'book.qidian.com']
def start_requests(self):
urls = 'https://www.qidian.com/all?'
# print('request start')
# url = 'https://www.qidian.com/all?chanId=21&orderId=&page=1&style=2&pageSize=50&siteid=1&pubflag=0&hiddenField=0'
request_body= self.settings.getdict('DEFAULT_PARAM')
for chanid in self.settings.getdict('CHANIDLIST').values():
request_body['chanId']= chanid
for page in range(1,20):
request_body['page']=page
url = 'https://www.qidian.com/all?'+urlencode(request_body)
yield scrapy.Request(url,dont_filter=True)
def parse(self,response):
bot = Selector(response)
csrfToken = self.get_cookies('_csrfToken',response)
contents = bot.xpath('//tbody/tr')
for content in contents:
item = QidianItem()
item['book_type'] = content.xpath('td[1]/a[1]/text()').extract_first()
item['book_sub_type'] = content.xpath('td[1]/a[2]/text()').extract_first()
item['book_name']= content.xpath('td[2]/a[1]/text()').extract_first()
item['book_url'] = 'https:' + content.xpath('td[2]/a[1]/@href').extract_first()
item['total_words'] = content.xpath('td[4]/span/text()').extract_first()
item['author'] = content.xpath('td[5]/a/text()').extract_first()
item['last_upload_date'] = content.xpath('td[6]/text()').extract_first()
yield scrapy.Request(item['book_url'],meta={'item':item,'csrfToken':csrfToken},
callback=self.parse_detail)
def parse_detail(self,response):
bot = Selector(response)
item = response.meta['item']
item['book_status'] = bot.xpath("//div/p/span[@class='blue'][1]/text()").extract_first()
click_content=bot.xpath('normalize-space(//div[2]/p[3])').extract_first()
item['click_count'] = click_content.split('|')[1].split('·')[0]
item['recommand_count'] = click_content.split('|')[2].split('·')[0]
item['book_id'] = item['book_url'].split('/')[-1]
detail_params = self.settings.getdict('DEFAULT_COMEMENT_PARM')
detail_params['_csrfToken'] = response.meta['csrfToken']
detail_params['bookId']= item['book_id']
next_url = self.settings.get('COMMENTS_URL')+urlencode(detail_params)
yield scrapy.Request(next_url,meta={'item':item},callback=self.parse_rank)
def get_cookies(self,cookie_key,response):
cookie_list = response.headers.getlist('Set-Cookie')[0].decode('utf-8').split(';')
for cookie in cookie_list:
if cookie_key in cookie:
return cookie.strip().split('=')[-1]
self.logger.info('ERORR:Cookie not found!! KEY is {}'.format(cookie_key))
return ''
def parse_rank(self,response):
jsondata = json.loads(response.text)
item = response.meta['item']
item['rank_score'] = jsondata['data']['rate']
item['rank_ppl_involved'] = jsondata['data']['userCount']
yield item
还有其他代码，涉及到数据库的写入和简单的数据清洗，由于篇幅限制，此处不再贴出，请大家移步我的github，里面有具体的代码，网址如下：
https://github.com/xiaxia47/qidian
文章虽然已告一段落，但是这个爬虫还有需要改进的地方，比如说middleware中貌似random-user-agent的功能并没有被调用，还有ajax异步接口这边的middleware也没有起作用，另外爬虫的效率有待提升。接下来一段时间我的目标就是继续优化这个起点爬虫，使之更高效，更简洁，同时往分布式发展。
最后以我最爱的一句短语结束,极致优雅，极致简洁！
