Java爬虫实现Jsoup利用dom方法遍历Document对象
作者:Cyril_KI 时间:2023-06-15 07:52:36
先给出网页地址:
https://wall.alphacoders.com/featured.php?lang=Chinese
主要步骤:
利用Jsoup的connect方法获取Document对象
String html = "https://wall.alphacoders.com/featured.php?lang=Chinese";
Document doc = Jsoup.connect(html).get();
内容过长,就不再显示。
我们以这部分为例:
<ul class="nav nav-pills">
<li><a href="https://alphacoders.com/site/about-us" rel="external nofollow" rel="external nofollow" >About Us</a></li>
<li><a href="https://alphacoders.com/site/faq" rel="external nofollow" rel="external nofollow" >FAQ</a></li>
<li><a href="https://alphacoders.com/site/privacy" rel="external nofollow" rel="external nofollow" >Privacy Policy</a></li>
<li><a href="https://alphacoders.com/site/tos" rel="external nofollow" rel="external nofollow" >Terms Of Service</a></li>
<li><a href="https://alphacoders.com/site/acceptable_use" rel="external nofollow" rel="external nofollow" >Acceptable Use</a></li>
<li><a href="https://alphacoders.com/site/etiquette" rel="external nofollow" rel="external nofollow" >Etiquette</a></li>
<li><a href="https://alphacoders.com/site/advertising" rel="external nofollow" rel="external nofollow" >Advertise With Us</a></li>
<li><a id="change_consent">Change Consent</a></li>
</ul>
我们先找到所有的ul:
Elements elements = doc.getElementsByTag("ul");
输出如下:
<ul class="nav navbar-nav center">
<li> <a title="Submit Wallpapers" href="https://alphacoders.com/site/submit-wallpaper" rel="external nofollow" ><i class="el el-circle-arrow-up"></i> 提交</a> </li>
<li> <a href="https://alphacoders.com/contest" rel="external nofollow" ><i class="el el-gift"></i> 精美奖品</a> </li>
</ul>
<ul class="nav navbar-nav navbar-right center">
<li> <a href="language.php?lang=Chinese" rel="external nofollow" > <img src="https://static.alphacoders.com/wa/Chinese-flag.png" alt="Chinese-flag"> 中文 </a> </li>
<li> <a href="https://alphacoders.com/users/login" rel="external nofollow" ><i class="el el-user"></i> 登录</a> </li>
<li> <a href="https://alphacoders.com/users/register" rel="external nofollow" ><i class="el el-edit"></i> 注册</a> </li>
</ul>
<ul class="pagination">
<li class="active"><a id="prev_page" href="#" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" >< 上一页</a></li>
<li class="active"><a>1</a></li>
<li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&page=2" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" >2</a></li>
<li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&page=3" rel="external nofollow" rel="external nofollow" rel="external nofollow" >3</a></li>
<li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&page=4" rel="external nofollow" rel="external nofollow" rel="external nofollow" >4</a></li>
<li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&page=5" rel="external nofollow" rel="external nofollow" rel="external nofollow" >5</a></li>
<li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&page=6" rel="external nofollow" rel="external nofollow" rel="external nofollow" >6</a></li>
<li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&page=7" rel="external nofollow" rel="external nofollow" rel="external nofollow" >7</a></li>
<li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&page=8" rel="external nofollow" >8</a></li>
<li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&page=9" rel="external nofollow" >9</a></li>
<li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&page=10" rel="external nofollow" >10</a></li>
<li><a>...</a></li>
<li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&page=319" rel="external nofollow" rel="external nofollow" rel="external nofollow" >319</a></li>
<li><a id="next_page" href="https://wall.alphacoders.com/featured.php?lang=Chinese&page=2" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" >下一页 ></a></li>
</ul>
<ul class="pagination">
<li class="active"><a href="#" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" >< 上一页</a></li>
<li class="active"><a>1</a></li>
<li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&page=2" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" >2</a></li>
<li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&page=3" rel="external nofollow" rel="external nofollow" rel="external nofollow" >3</a></li>
<li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&page=4" rel="external nofollow" rel="external nofollow" rel="external nofollow" >4</a></li>
<li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&page=5" rel="external nofollow" rel="external nofollow" rel="external nofollow" >5</a></li>
<li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&page=6" rel="external nofollow" rel="external nofollow" rel="external nofollow" >6</a></li>
<li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&page=7" rel="external nofollow" rel="external nofollow" rel="external nofollow" >7</a></li>
<li><a>...</a></li>
<li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&page=319" rel="external nofollow" rel="external nofollow" rel="external nofollow" >319</a></li>
<li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&page=2" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" >下一页 ></a></li>
</ul>
<ul class="pagination">
<li class="active"><a href="#" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" ><< </a></li>
<li class="active"><a href="#" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" >< 上一页</a></li>
<li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&page=2" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" >下一页 ></a></li>
<li><a title="末页 (319)" href="https://wall.alphacoders.com/featured.php?lang=Chinese&page=319" rel="external nofollow" rel="external nofollow" rel="external nofollow" > >></a></li>
</ul>
<ul class="pagination">
<li class="active"><a href="#" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" >1</a></li>
<li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&page=2" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" >2</a></li>
<li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&page=3" rel="external nofollow" rel="external nofollow" rel="external nofollow" >3</a></li>
<li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&page=4" rel="external nofollow" rel="external nofollow" rel="external nofollow" >4</a></li>
<li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&page=5" rel="external nofollow" rel="external nofollow" rel="external nofollow" >5</a></li>
<li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&page=6" rel="external nofollow" rel="external nofollow" rel="external nofollow" >6</a></li>
<li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&page=7" rel="external nofollow" rel="external nofollow" rel="external nofollow" >7</a></li>
</ul>
<ul class="nav nav-pills">
<li><a href="https://alphacoders.com/site/about-us" rel="external nofollow" rel="external nofollow" >About Us</a></li>
<li><a href="https://alphacoders.com/site/faq" rel="external nofollow" rel="external nofollow" >FAQ</a></li>
<li><a href="https://alphacoders.com/site/privacy" rel="external nofollow" rel="external nofollow" >Privacy Policy</a></li>
<li><a href="https://alphacoders.com/site/tos" rel="external nofollow" rel="external nofollow" >Terms Of Service</a></li>
<li><a href="https://alphacoders.com/site/acceptable_use" rel="external nofollow" rel="external nofollow" >Acceptable Use</a></li>
<li><a href="https://alphacoders.com/site/etiquette" rel="external nofollow" rel="external nofollow" >Etiquette</a></li>
<li><a href="https://alphacoders.com/site/advertising" rel="external nofollow" rel="external nofollow" >Advertise With Us</a></li>
<li><a id="change_consent">Change Consent</a></li>
</ul>
可以发现class为"nav nav-pills"的只有一个,我们找到它:
Elements elements = doc.getElementsByTag("ul");
//System.out.println(elements);
Element tempElement = null;
for(Element element : elements) {
if (element.className().equals("nav nav-pills")) {
tempElement = element;
//System.out.println(element.className());
break;
}
}
循环遍历这个ul,输出其中每一个li里每一个a的href和rel属性:
Elements li = tempElement.getElementsByTag("li");
for(Element element : li) {
Elements element2 = element.getElementsByTag("a");
for(Element element3 : element2) {
String hrefString = element3.attr("href");
String relString = element3.attr("rel");
if(hrefString != "" && relString != "") {
System.out.println("href=" + hrefString + " rel="external nofollow" rel="external nofollow" " + "rel=" + relString);
}
}
}
最终结果:
href=https://alphacoders.com/site/about-us rel=nofollow
href=https://alphacoders.com/site/faq rel=nofollow
href=https://alphacoders.com/site/privacy rel=nofollow
href=https://alphacoders.com/site/tos rel=nofollow
href=https://alphacoders.com/site/acceptable_use rel=nofollow
href=https://alphacoders.com/site/etiquette rel=nofollow
href=https://alphacoders.com/site/advertising rel=nofollow
完整代码:
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
import org.jsoup.select.Elements;
import java.io.IOException;
import org.jsoup.Jsoup;
/**
* @ClassName: Jsoup_Test
* @description:
* @author: KI
* @Date: 2020年8月17日 下午8:15:14
*/
public class Jsoup_Test {
public static void main(String[] args) throws IOException {
// TODO 自动生成的方法存根
String html = "https://wall.alphacoders.com/featured.php?lang=Chinese";
Document doc = Jsoup.connect(html).get();
System.out.println(doc);
Elements elements = doc.getElementsByTag("ul");
//System.out.println(elements);
Element tempElement = null;
for(Element element : elements) {
if (element.className().equals("nav nav-pills")) {
tempElement = element;
//System.out.println(element.className());
break;
}
}
System.out.println(tempElement);
Elements li = tempElement.getElementsByTag("li");
for(Element element : li) {
Elements element2 = element.getElementsByTag("a");
for(Element element3 : element2) {
String hrefString = element3.attr("href");
String relString = element3.attr("rel");
if(hrefString != "" && relString != "") {
System.out.println("href=" + hrefString + " rel="external nofollow" rel="external nofollow" " + "rel=" + relString);
}
}
}
}
}
来源:https://blog.csdn.net/Cyril_KI/article/details/108073933
标签:Java,Jsoup,遍历,Document对象
![](/images/zang.png)
![](/images/jiucuo.png)
猜你喜欢
springboot logback调整mybatis日志级别无效的解决
2023-05-03 05:36:21
Java开发实现人机猜拳游戏
2023-10-17 16:01:35
![](https://img.aspxhome.com/file/2023/9/61149_0s.jpg)
java实现图片滑动验证(包含前端代码)
2022-03-21 12:13:52
![](https://img.aspxhome.com/file/2023/8/60538_0s.jpg)
Android 如何获取手机总内存和可用内存等信息
2023-07-27 13:11:42
用SpringBoot+Vue+uniapp小程序实现在线房屋装修管理系统
2023-11-12 04:10:48
![](https://img.aspxhome.com/file/2023/3/58883_0s.png)
Java ArrayList实现删除指定位置的元素
2023-11-25 05:34:13
![](https://img.aspxhome.com/file/2023/0/60160_0s.png)
流读取导致StringBuilder.toString()乱码的问题及解决
2022-12-20 13:34:14
关于@ConditionalOnProperty的作用及用法说明
2023-11-24 02:39:19
idea设置JVM运行参数的几种方式
2023-05-26 14:57:33
![](https://img.aspxhome.com/file/2023/3/63803_0s.png)
Java 处理超大数类型之BigInteger案例详解
2021-06-20 15:36:38
java读写oracle的blob字段示例
2023-12-22 16:19:00
java 数据结构单链表的实现
2022-07-24 09:45:33
java连接SQL Server数据库的超详细教程
2023-04-05 21:46:22
![](https://img.aspxhome.com/file/2023/5/63095_0s.png)
java基础之包装类的介绍及使用
2023-01-10 17:45:29
![](https://img.aspxhome.com/file/2023/5/61255_0s.png)
Java链表(Linked List)基本原理与实现方法入门示例
2021-10-12 05:49:14
![](https://img.aspxhome.com/file/2023/4/60824_0s.png)
Android使用xUtils3.0实现文件上传
2023-08-04 19:50:02
Java 实现微信和支付宝支付功能
2023-03-08 23:18:04
java多线程加锁以及Condition类的使用实例解析
2023-08-07 07:25:30
![](https://img.aspxhome.com/file/2023/3/57713_0s.png)
springboot整合Quartz实现动态配置定时任务的方法
2023-03-08 22:13:10
Java实现简单的递归操作方法实例
2021-11-01 14:37:32
![](https://img.aspxhome.com/file/2023/6/65306_0s.png)