Python爬虫设置ip代理过程解析
作者:知人知面 时间:2021-05-27 15:08:30
1、get方式:如何为爬虫添加ip代理,设置Request header(请求头)
import urllib
import urllib.request
import urllib.parse
import random
import time
from fake_useragent import UserAgent
ua = UserAgent()
url = "http://www.baidu.com"
########################################################
'''
设置ip代理
iplist = [ '127.0.0.1:80'] #可自行上网找一些代理
proxy_support = urllib.request.ProxyHandler({'http':random.choice(iplist)}) #也可以设置为https,要看你的代理支不支持
opener = urllib.request.build_opener(proxy_support)
'''
########################################################
'''无ip代理'''
opener = urllib.request.build_opener()
'''f12查看请求头添加即可,不一定都需要全添加↓↓↓'''
opener.addheaders = [('Host', 'newtab.firefoxchina.cn'),
('User-Agent',ua.random),
('Accept-Encoding','deflate, br'),
('Accept', 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8'),
('Accept-Language', 'zh-CN,zh;q=0.8,zh-TW;q=0.7,zh-HK;q=0.5,en-US;q=0.3,en;q=0.2'),
('Connection', 'keep-alive'),
('Upgrade-Insecure-Requests',1),
('Cookie', '__gads=ID=138080209be66bf8:T=1592037395:S=ALNI_Ma-g9wHmfxFL4GCy9veAjJrJRsNmg; Hm_lvt_dd4738b5fb302cb062ef19107df5d2e4=1592449208,1592471447,1592471736,1594001802; uid=rBADnV7m04mi8wRJK3xYAg=='),
]
urllib.request.install_opener(opener)
while True:
try:
response = urllib.request.urlopen(url)
break
except Exception as e:
print("错误信息:" + str(e))
time.sleep(3)
html = response.read().decode("utf-8")
print(html)
2、post方式添加载荷(此处是打比方),修改urllib.request.install_opener(opener)以下的代码即可
urllib.request.install_opener(opener)
# data = {} #当页面提交数据是有载荷但是载荷内容为空时,必须以data = {}传参,不然无法获取网页数据
data = {'_csrf':'请把',
'collection-name':'载荷的参数',
'description':'以这种形式',
'_csrf':'装载'
}
data = urllib.parse.urlencode(data).encode('utf-8')
req = urllib.request.Request(url,data)
while True:
try:
response = urllib.request.urlopen(req)
break
except Exception as e:
print("错误信息:" + str(e))
time.sleep(3)
html = response.read().decode("utf-8")
来源:https://www.cnblogs.com/zrzm/p/13332371.html
标签:Python,爬虫,ip,代理
![](/images/zang.png)
![](/images/jiucuo.png)
猜你喜欢
浅谈Python 中的复数问题
2023-04-22 17:02:11
![](https://img.aspxhome.com/file/2023/1/75181_0s.png)
PHP composer更新指定依赖包过程详细讲解
2023-05-27 18:05:34
python PyQt5 爬虫实现代码
2022-10-19 20:24:42
![](https://img.aspxhome.com/file/2023/3/86623_0s.jpg)
页面制作的重要性
2007-10-30 13:14:00
![](https://img.aspxhome.com/file/UploadPic/200710/30/2007103013184371s.gif)
Python安装Graphviz 超详细图文教程
2023-02-27 16:11:18
![](https://img.aspxhome.com/file/2023/1/75891_0s.jpg)
Python模块glob函数示例详解教程
2023-10-27 15:49:36
![](https://img.aspxhome.com/file/2023/3/85173_0s.png)
python实现简易的学生信息管理系统
2021-08-24 12:07:09
Python实现滑雪小游戏
2023-08-10 15:30:03
![](https://img.aspxhome.com/file/2023/7/66397_0s.gif)
IE6 升级要不要?
2009-07-23 20:44:00
![](https://img.aspxhome.com/file/UploadPic/20097/23/123541f77cbg214-50s.jpg)
在HTML中,常见的URL有多种表示方式:
2009-07-28 12:18:00
Python基于Django实现验证码登录功能
2023-06-25 03:21:13
![](https://img.aspxhome.com/file/2023/8/72238_0s.png)
Python+pyecharts绘制双动态曲线教程详解
2023-03-04 09:19:48
![](https://img.aspxhome.com/file/2023/4/99234_0s.png)
python自动化测试中装饰器@ddt与@data源码深入解析
2021-10-08 16:07:14
![](https://img.aspxhome.com/file/2023/1/72951_0s.png)
Windows下anaconda安装第三方包的方法小结(tensorflow、gensim为例)
2021-09-14 00:22:11
![](https://img.aspxhome.com/file/2023/0/95540_0s.png)
PHP使用星号隐藏用户名,手机和邮箱的实现方法
2023-11-05 10:41:56
accept-charset与Header P3P
2009-04-01 18:43:00
python 进程池pool使用详解
2022-02-13 03:42:52
![](https://img.aspxhome.com/file/2023/8/105228_0s.gif)
python从入门到精通(DAY 3)
2023-11-03 08:23:18
![](https://img.aspxhome.com/file/2023/8/63598_0s.png)
Pytorch结合PyG实现MLP过程详解
2022-05-01 21:51:55
python二分查找算法的递归实现方法
2023-05-12 23:22:48