Python多线程下载文件的方法
作者:speedmancs 时间:2021-12-26 05:21:32
本文实例讲述了Python多线程下载文件的方法。分享给大家供大家参考。具体实现方法如下:
import httplib
import urllib2
import time
from threading import Thread
from Queue import Queue
from time import sleep
proxy = 'your proxy';
opener = urllib2.build_opener( urllib2.ProxyHandler({'http':proxy}) )
urllib2.install_opener( opener )
ids = {};
for i in range(1,110):
try:
listUrl = "http://www.someweb.net/sort/list_8_%d.shtml" % (i);
print listUrl;
page = urllib2.urlopen(listUrl).read();
speUrl = "http://www.someweb.net/soft/";
speUrlLen = len(speUrl);
idx = page.find(speUrl,0);
while idx!=-1:
dotIdx = page.find(".",idx + speUrlLen);
if dotIdx != -1:
id = page[idx + speUrlLen:dotIdx];
ids[id] = 1;
idx = page.find("http://www.someweb.net/soft/",idx + speUrlLen);
except:
pass;
q = Queue()
NUM = 5
failedId = [];
def do_somthing_using(id):
try:
url = "http://www.someweb.net/download.php?softid=%s&type=dx" % (id);
h2 = httplib.HTTPConnection("your proxy", "you port");
h2.request("HEAD", url);
resp = h2.getresponse();
header = resp.getheaders();
location = header[3][1];
sContent = urllib2.urlopen(location).read();
savePath = "C:\\someweb\\%s.rar" % (id);
file=open(savePath,'wb');
file.write(sContent);
file.close();
print savePath + " saved";
except:
pass;
def working():
while True:
arguments = q.get()
do_somthing_using(arguments)
sleep(1)
q.task_done()
for i in range(NUM):
t = Thread(target=working)
t.setDaemon(True)
t.start()
for id in ids:
q.put(id)
q.join()
希望本文所述对大家的Python程序设计有所帮助。
标签:Python,多线程,下载
![](/images/zang.png)
![](/images/jiucuo.png)
猜你喜欢
Go语言递归函数的具体实现
2023-08-05 02:35:32
PHP 数组和字符串互相转换实现方法
2023-06-19 15:04:17
IE6局部调用PNG32合并图片
2009-03-11 21:24:00
![](https://img.aspxhome.com/file/UploadPic/20093/11/01-12s.jpg)
详解php实现页面静态化原理
2023-10-25 10:58:10
设计与用户体验
2009-05-06 13:36:00
![](https://img.aspxhome.com/file/UploadPic/20095/6/youshang02-88s.png)
YUI学习笔记(3)
2009-01-21 16:24:00
对mysql表进行优化、分析、检查和修复的说明
2010-04-22 16:28:00
动态提示的select下拉框
2007-12-02 14:54:00
优化MySQL数据库性能的八大“妙手”
2007-11-18 14:49:00
用CSS设计多种文本框与按钮样式风格
2007-08-10 13:11:00
![](https://img.aspxhome.com/file/UploadPic/20072/20072311391742s.gif)
uniqueidentifier转换成varchar数据类型的sql语句
2011-09-30 11:17:48
利用Python提取PDF文本的简单方法实例
2023-05-09 05:37:34
![](https://img.aspxhome.com/file/2023/9/66409_0s.jpg)
asp 各种进制转换函数
2008-06-24 12:35:00
10个简化PHP开发的工具
2023-07-14 18:02:05
![](https://img.aspxhome.com/file/2023/3/55293_0s.png)
Python实现自动驾驶训练模型
2023-07-28 18:45:29
![](https://img.aspxhome.com/file/2023/1/61481_0s.png)
python Zmail模块简介与使用示例
2023-07-23 23:42:37
Javascript中的isNaN函数使用说明
2023-08-27 10:10:02
Windows 2003服务器上传文件受限制的解决方法
2011-02-14 11:29:00
isset和empty的区别
2023-11-20 12:24:08
SQL Server 2008数据挖掘查询任务
2009-03-16 16:50:00