java网络爬虫连接超时解决实例代码
作者:sober_qianyang 时间:2022-02-02 06:57:28
本文研究的主要是java网络爬虫连接超时的问题,具体如下。
在网络爬虫中,经常会遇到如下报错。即连接超时。针对此问题,一般解决思路为:将连接时间、请求时间设置长一下。如果出现连接超时的情况,则在重新请求【设置重新请求次数】。
Exception in thread "main" java.net.ConnectException: Connection timed out: connect
下面的代码便是使用httpclient解决连接超时的样例程序。直接上程序。
package daili;
import java.io.IOException;
import java.net.URI;
import org.apache.http.HttpRequest;
import org.apache.http.HttpResponse;
import org.apache.http.client.ClientProtocolException;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.client.params.CookiePolicy;
import org.apache.http.client.protocol.ClientContext;
import org.apache.http.impl.client.BasicCookieStore;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.DefaultHttpClient;
import org.apache.http.impl.client.DefaultHttpRequestRetryHandler;
import org.apache.http.impl.client.HttpClients;
import org.apache.http.impl.cookie.BasicClientCookie2;
import org.apache.http.params.HttpConnectionParams;
import org.apache.http.params.HttpParams;
import org.apache.http.protocol.BasicHttpContext;
import org.apache.http.protocol.ExecutionContext;
import org.apache.http.protocol.HttpContext;
import org.apache.http.util.EntityUtils;
/*
* author:合肥工业大学 管院学院 钱洋
*1563178220@qq.com
*/
public class Test1 {
public static void main(String[] args) throws ClientProtocolException, IOException, InterruptedException {
getRawHTML("http://club.autohome.com.cn/bbs/forum-c-2098-1.html#pvareaid=103447");
}
public static String getRawHTML ( String url ) throws ClientProtocolException, IOException, InterruptedException{
//初始化
DefaultHttpClient httpclient = new DefaultHttpClient();
httpclient.getParams().setParameter("http.protocol.cookie-policy",
CookiePolicy.BROWSER_COMPATIBILITY);
//设置参数
HttpParams params = httpclient.getParams();
//连接时间
HttpConnectionParams.setConnectionTimeout(params, 6000);
HttpConnectionParams.setSoTimeout(params, 6000*20);
//超时重新请求次数
DefaultHttpRequestRetryHandler dhr = new DefaultHttpRequestRetryHandler(5,true);
HttpContext localContext = new BasicHttpContext();
HttpRequest request2 = (HttpRequest) localContext.getAttribute(
ExecutionContext.HTTP_REQUEST);
httpclient.setHttpRequestRetryHandler(dhr);
BasicCookieStore cookieStore = new BasicCookieStore();
BasicClientCookie2 cookie = new BasicClientCookie2("Content-Type","text/html;charset=UTF-8");
BasicClientCookie2 cookie1 = new BasicClientCookie2("User-Agent","Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36");
cookieStore.addCookie(cookie);
cookieStore.addCookie(cookie1);
localContext.setAttribute(ClientContext.COOKIE_STORE, cookieStore);
HttpGet request = new HttpGet();
request.setURI(URI.create(url));
HttpResponse response = null;
String rawHTML = "";
response = httpclient.execute(request,localContext);
int StatusCode = response.getStatusLine().getStatusCode();
//获取响应状态码
System.out.println(StatusCode);
if(StatusCode == 200){
//状态码200表示响应成功
//获取实体内容
rawHTML = EntityUtils.toString (response.getEntity());
System.out.println(rawHTML);
//输出实体内容
EntityUtils.consume(response.getEntity());
//消耗实体
} else {
//关闭HttpEntity的流实体
EntityUtils.consume(response.getEntity());
//消耗实体
Thread.sleep(20*60*1000);
//如果报错先休息30分钟
}
httpclient.close();
System.out.println(rawHTML);
return rawHTML;
}
}
结果:
来源:http://blog.csdn.net/qy20115549/article/details/54973952
标签:java,爬虫,超时
0
投稿
猜你喜欢
Java实现经典游戏黄金矿工的示例代码
2022-01-09 16:35:41
Java反射概念与使用实例代码
2022-02-24 23:55:40
Unity Shader实现玻璃材质效果
2021-10-27 05:25:29
深入理解Java设计模式之外观模式
2023-05-02 09:48:00
SpringBoot实现本地存储文件上传及提供HTTP访问服务的方法
2022-09-14 19:09:12
SpringBoot整合Shiro实现登录认证的方法
2022-03-23 01:12:19
Java IO之File 类详解
2023-08-07 20:02:12
深入讲解Java Maven配置
2022-07-01 05:09:21
详解Spring boot使用Redis集群替换mybatis二级缓存
2023-03-09 17:17:38
Java语言求解完美数代码分析
2023-01-28 10:17:58
Java全面分析面向对象之继承
2023-11-23 11:55:59
Jsoup解析html实现招聘信息查询功能
2022-01-08 19:38:07
Entity Framework配置关系
2023-10-15 09:45:30
windows定时器配置执行java jar文件的方法详解
2023-04-17 16:40:21
flutter Bloc 实现原理示例解析
2023-07-18 08:00:39
Java Set集合的遍历及实现类的比较
2023-11-05 16:08:10
详解基于spring多数据源动态调用及其事务处理
2023-06-23 14:37:25
Spring Data Jpa 复杂查询方式总结(多表关联及自定义分页)
2023-11-21 10:01:27
Java实现马踏棋盘算法
2023-03-05 04:30:46
JavaWeb实现文件上传与下载的方法
2023-12-23 04:42:56