python通过urllib2爬网页上种子下载示例
时间:2022-04-12 10:51:58
通过urllib2、re模块抓种子
思路
1.用程序登录论坛(如果需要登录才能访问的版块)
2.访问指定版块
3.遍历帖子(先取指定页,再遍历页面所有帖子的url)
4.循环访问所有帖子url,从帖子页面代码中取种子下载地址(通过正则表达式或第三方页面解析库)
5.访问种子页面下载种子
import urllib
import urllib2
import cookielib
import re
import sys
import os
# site is website address | fid is part id
site = "http://xxx.yyy.zzz/"
source = "thread0806.php?fid=x&search=&page="
btSave = "./clyzwm/"
if os.path.isdir(btSave):
print btSave + " existing"
else:
os.mkdir(btSave)
logfile = "./clyzwm/down.log"
errorfile = "./clyzwm/error.log"
sucfile = "./clyzwm/sucess.log"
headers = {'User-Agent' : 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36',
'Referer' : 'http://xxx.yyy.zzz/'}
def btDown(url, dirPath):
logger(logfile, "download file : " + url)
try:
#pageCode = urllib2.urlopen(url).read()
#print pageCode
btStep1 = re.findall('http://[\w]+\.[\w]+\.[\w]{0,4}/[\w]{2,6}\.php\?[\w]{2,6}=([\w]+)', url, re.I)
#print btStep1
if len(btStep1)>0:
ref = btStep1[0]
downsite = ""
downData = {}
if len(ref)>20:
downsite = re.findall('http://www.[\w]+\.[\w]+/', url)[0]
downsite = downsite + "download.php"
reff = re.findall('input\stype=\"hidden\"\sname=\"reff\"\svalue=\"([\w=]+)\"', urllib2.urlopen(url).read(), re.I)[0]
downData = {'ref': ref, 'reff':reff, 'submit':'download'}
else:
downsite = "http://www.downhh.com/download.php"
downData = {'ref': ref, 'rulesubmit':'download'}
#print "bt site - " + downsite + "\n downData:"
#print downData
downData = urllib.urlencode(downData)
downReq = urllib2.Request(downsite, downData)
downReq.add_header('User-Agent','Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36')
downPost = urllib2.urlopen(downReq)
stream = downPost.read(-1)
if (len(stream) > 1000):
downPost.close()
name = btStep1[0]+ ".torrent"
fw = open(dirPath + name, 'w')
fw.write(stream)
fw.close()
logger(sucfile, url+"\n")
else:
logger(errorfile, url+"\n")
except urllib2.URLError, e:
print e.reason
def logger(logfile, msg):
print msg
fw = open(logfile, 'a')
fw.write(msg)
fw.close()
for i in range(1, 1000):
logger(logfile, "\n\n\n@ page " + str(i) + " ...")
part = site + source + str(i)
content = urllib2.urlopen(part).read()
content = content.decode('gbk').encode('utf8')
#print content
pages = re.findall('<a\s+href=\"(htm_data/[\d]+/[\d]+/[\d]+\.html).*?<\/a>', content,re.I)
#print pages
for page in pages:
page = site + page;
#logger(logfile, "\n# visiting " + page + " ...")
pageCode = urllib2.urlopen(page).read()
#print pageCode
zzJump = re.findall('http://www.viidii.info/\?http://[\w]+/[\w]+\?[\w]{2,6}=[\w]+' ,pageCode)
#zzJump = re.findall('http://www.viidii.info/\?http://[\w/\?=]*', pageCode)
if len(zzJump) > 0:
zzJump = zzJump[0]
#print "- jump page - " + zzJump
pageCode = urllib2.urlopen(page).read()
zzPage = re.findall('http://[\w]+\.[\w]+\.[\w]+/link[\w]?\.php\?[\w]{2,6}=[\w]+' ,pageCode)
if len(zzPage) > 0:
zzPage = zzPage[0]
logger(logfile, "\n- zhongzi page -" + zzPage)
btDown(zzPage, btSave)
else:
logger(logfile, "\n. NOT FOUND .")
else:
logger(logfile, "\n... NOT FOUND ...")
zzPage = re.findall('http://[\w]+\.[\w]+\.[\w]+/link[\w]?\.php\?ref=[\w]+' ,pageCode)
![](/images/zang.png)
![](/images/jiucuo.png)
猜你喜欢
理解SQL SERVER中的逻辑读,预读和物理读
mysql 安装使用小记
ASP CacheControl 属性
基于Python的自媒体小助手---登录页面的实现代码
![](https://img.aspxhome.com/file/2023/5/109915_0s.png)
一文详解如何用GPU来运行Python代码
![](https://img.aspxhome.com/file/2023/1/99151_0s.png)
Python实现字典的key和values的交换
Oracle SQL中实现indexOf和lastIndexOf功能的思路及代码
Python实现8个概率分布公式的方法详解
![](https://img.aspxhome.com/file/2023/4/79654_0s.png)
Python中用于去除空格的三个函数的使用小结
关于IDEA git 只有Commit没有Push的问题
![](https://img.aspxhome.com/file/2023/7/63437_0s.png)
OraclePL/SQL单行函数和组函数详解
详解Python字典的操作
![](https://img.aspxhome.com/file/2023/3/64643_0s.png)
python itertools包内置无限迭代器
python标准库random模块处理随机数
Django初步使用Celery处理耗时任务和定时任务问题
![](https://img.aspxhome.com/file/2023/0/88010_0s.png)
Pyqt5 实现窗口缩放,控件在窗口内自动伸缩的操作
![](https://img.aspxhome.com/file/2023/7/91997_0s.jpg)