python爬虫URL重试机制的实现方法(python2.7以及python3.5)
作者:ZJL-阿友 时间:2023-08-06 07:51:43
应用场景:
状态不是200的URL重试多次
代码比较简单还有部分注释
python2.7实现:
# -*-coding:utf-8-*-
"""
ayou
"""
import requests
def url_retry(url,num_retries=3):
print("access!")
try:
request = requests.get(url,timeout=60)
#raise_for_status(),如果不是200会抛出HTTPError错误
request.raise_for_status()
html = request.content
except requests.HTTPError as e:
html=None
if num_retries>0:
#如果不是200就重试,每次递减重试次数
return url_retry(url,num_retries-1)
#如果url不存在会抛出ConnectionError错误,这个情况不做重试
except requests.exceptions.ConnectionError as e:
return
return html
url_retry("http://httpbin.org/status/404")
python3.5实现:
# -*-coding:utf-8-*-
"""
ayou
"""
import asyncio
import aiohttp
async def print_page(url,num_retries=3):
async with aiohttp.ClientSession() as session:
try:
async with session.get(url,timeout=60) as response:
print("access!")
#raise_for_status(),如果不是200会抛出HttpProcessingError错误
response.raise_for_status()
body = await response.text()
except aiohttp.errors.HttpProcessingError as e:
body = None
if num_retries > 0:
#如果不是200就重试,每次递减重试次数
return await print_page(url, num_retries - 1)
#不存在URL会抛出ClientResponseError错误
except aiohttp.errors.ClientResponseError as e:
return e
session.close()
print(body)
return body
def main():
#这是一个不存在URL
# url = 'http://httpbin.org/status/404111'
#这是一个404的URL
url = 'http://httpbin.org/status/404'
loop = asyncio.get_event_loop()
loop.run_until_complete(print_page(url))
loop.close()
if __name__ == '__main__':
main()
爬虫URL重试机制封装成修饰器(python2.7以及python3.5以上)
python2.7版本:
# -*-coding:utf-8-*-
"""
ayou
"""
import requests
#定义一个重试修饰器,默认重试一次
def retry(num_retries=1):
#用来接收函数
def wrapper(func):
#用来接收函数的参数
def wrapper(*args,**kwargs):
#为了方便看抛出什么错误定义一个错误变量
last_exception =None
#循环执行包装的函数
for _ in range(num_retries):
try:
#如果没有错误就返回包装的函数,这样跳出循环
return func(*args, **kwargs)
except Exception as e:
#捕捉到错误不要return,不然就不会循环了
last_exception = e
#如果要看抛出错误就可以抛出
# raise last_exception
return wrapper
return wrapper
if __name__=="__main__":
@retry(5)
def url_retry(url):
request = requests.get(url, timeout=60)
print("access!")
request.raise_for_status()
html = request.content
print(html)
return html
url_retry("http://httpbin.org/status/404")
# url_retry("http://httpbin.org/status/404111")
# url_retry("http://www.baidu.com")
python3.5以上版本:
# -*-coding:utf-8-*-
"""
ayou
"""
import aiohttp,asyncio
#定义一个重试修饰器,默认重试一次
def retry(num_retries=1):
#用来接收函数
def wrapper(func):
#用来接收函数的参数
def wrapper(*args,**kwargs):
#为了方便看抛出什么错误定义一个错误变量
last_exception =None
#循环执行包装的函数
for _ in range(num_retries):
try:
#如果没有错误就返回包装的函数,这样跳出循环
return func(*args, **kwargs)
except Exception as e:
#捕捉到错误不要return,不然就不会循环了
last_exception = e
#如果要看抛出错误就可以抛出
# raise last_exception
return wrapper
return wrapper
async def print_page(url):
async with aiohttp.ClientSession() as session:
async with session.get(url,timeout=60) as response:
print("access!")
#raise_for_status(),如果不是200会抛出HttpProcessingError错误
response.raise_for_status()
body = await response.text()
session.close()
print(body)
return body
@retry(5)
def loop_get():
# url = "http://www.baidu.com"
# url = 'http://httpbin.org/status/404111'
url = 'http://httpbin.org/status/404'
loop = asyncio.get_event_loop()
loop.run_until_complete(print_page(url))
loop.close()
if __name__ == '__main__':
loop_get()
来源:https://blog.csdn.net/u013055678/article/details/54290481
标签:python,爬虫,重试,机制
![](/images/zang.png)
![](/images/jiucuo.png)
猜你喜欢
通过Python绘制中国结的示例代码
2021-05-29 15:15:21
![](https://img.aspxhome.com/file/2023/4/102784_0s.png)
vue-router命名视图的使用讲解
2024-04-09 10:59:52
![](https://img.aspxhome.com/file/2023/1/139721_0s.png)
three.js实现炫酷的全景3D重力感应
2024-04-25 13:12:37
![](https://img.aspxhome.com/file/2023/8/135538_0s.jpg)
JavaScript多种页面刷新方法小结
2024-04-22 12:53:09
Python PIL读取的图像发生自动旋转的实现方法
2022-05-01 20:29:26
python安装cx_Oracle模块常见问题与解决方法
2021-04-24 13:00:27
Python DataFrame.groupby()聚合函数,分组级运算
2022-02-08 07:23:09
Python文件简单操作及openpyxl操作excel文件详解
2023-04-26 14:23:22
教你用压缩技术给SQL Server备份文件瘦身
2009-03-05 14:59:00
Django外键(ForeignKey)操作以及related_name的作用详解
2023-03-15 22:48:09
![](https://img.aspxhome.com/file/2023/6/125156_0s.png)
图片预载ImageLoader 1.1 Release
2008-11-04 20:04:00
2009年五一劳动节logo欣赏
2009-05-02 18:26:00
![](https://img.aspxhome.com/file/UploadPic/20095/2/baidu-090501-18s.jpg)
总结分析Python的5个硬核函数
2022-04-12 11:05:42
![](https://img.aspxhome.com/file/2023/5/91925_0s.png)
python从PDF中提取数据的示例
2021-10-15 21:46:05
![](https://img.aspxhome.com/file/2023/2/99462_0s.png)
详解python3 GUI刷屏器(附源码)
2022-02-02 12:34:15
![](https://img.aspxhome.com/file/2023/9/77719_0s.jpg)
使用pycharm和pylint检查python代码规范操作
2023-06-06 08:02:38
![](https://img.aspxhome.com/file/2023/3/65453_0s.jpg)
Python实现类似比特币的加密货币区块链的创建与交易实例
2022-03-19 15:49:49
![](https://img.aspxhome.com/file/2023/0/93890_0s.jpg)
Python的Flask框架及Nginx实现静态文件访问限制功能
2023-08-13 03:13:38
学习 YUI3 中的沙箱机制
2010-04-12 12:52:00
sql语句之数据操作介绍
2024-01-22 18:09:39