scrapy自定义pipeline类实现将采集数据保存到mongodb的方法

本文实例讲述了scrapy自定义pipeline类实现将采集数据保存到mongodb的方法。分享给大家供大家参考。具体如下：

# Standard Python library imports
# 3rd party modules
import pymongo
from scrapy import log
from scrapy.conf import settings
from scrapy.exceptions import DropItem
class MongoDBPipeline(object):
 def __init__(self):
   self.server = settings['MONGODB_SERVER']
   self.port = settings['MONGODB_PORT']
   self.db = settings['MONGODB_DB']
   self.col = settings['MONGODB_COLLECTION']
   connection = pymongo.Connection(self.server, self.port)
   db = connection[self.db]
   self.collection = db[self.col]
 def process_item(self, item, spider):
   err_msg = ''
   for field, data in item.items():
     if not data:
       err_msg += 'Missing ％s of poem from ％s\n' ％ (field, item['url'])
   if err_msg:
     raise DropItem(err_msg)
   self.collection.insert(dict(item))
   log.msg('Item written to MongoDB database ％s/％s' ％ (self.db, self.col),
       level=log.DEBUG, spider=spider)
   return item

希望本文所述对大家的python程序设计有所帮助。