基于scrapy实现的简单蜘蛛采集程序

本文实例讲述了基于scrapy实现的简单蜘蛛采集程序。分享给大家供大家参考。具体如下：

# Standard Python library imports
# 3rd party imports
from scrapy.contrib.spiders import CrawlSpider, Rule
from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor
from scrapy.selector import HtmlXPathSelector
# My imports
from poetry_analysis.items import PoetryAnalysisItem
HTML_FILE_NAME = r'.+\.html'
class PoetryParser(object):
 """
 Provides common parsing method for poems formatted this one specific way.
 """
 date_pattern = r'(\d{2} \w{3,9} \d{4})'

def parse_poem(self, response):
   hxs = HtmlXPathSelector(response)
   item = PoetryAnalysisItem()
   # All poetry text is in pre tags
   text = hxs.select('//pre/text()').extract()
   item['text'] = ''.join(text)
   item['url'] = response.url
   # head/title contains title - a poem by author
   title_text = hxs.select('//head/title/text()').extract()[0]
   item['title'], item['author'] = title_text.split(' - ')
   item['author'] = item['author'].replace('a poem by', '')
   for key in ['title', 'author']:
     item[key] = item[key].strip()
   item['date'] = hxs.select("//p[@class='small']/text()").re(date_pattern)
   return item
class PoetrySpider(CrawlSpider, PoetryParser):
 name = 'example.com_poetry'
 allowed_domains = ['www.example.com']
 root_path = 'someuser/poetry/'
 start_urls = ['http://www.example.com/someuser/poetry/recent/',
        'http://www.example.com/someuser/poetry/less_recent/']
 rules = [Rule(SgmlLinkExtractor(allow=[start_urls[0] + HTML_FILE_NAME]),
                 callback='parse_poem'),
      Rule(SgmlLinkExtractor(allow=[start_urls[1] + HTML_FILE_NAME]),
                 callback='parse_poem')]

希望本文所述对大家的Python程序设计有所帮助。