Python 使用tf-idf算法计算文档关键字权重并生成词云的方法
作者:虚坏叔叔 时间:2022-03-28 13:09:18
Python 使用tf-idf算法计算文档关键字权重,并生成词云
1. 根据tf-idf计算一个文档的关键词或者短语:
代码如下:
注意需要安装pip install sklean
;
from re import split
from jieba.posseg import dt
from sklearn.feature_extraction.text import TfidfVectorizer
from collections import Counter
from time import time
import jieba
#pip install sklean
FLAGS = set('a an b f i j l n nr nrfg nrt ns nt nz s t v vi vn z eng'.split())
def cut(text):
for sentence in split('[^a-zA-Z0-9\u4e00-\u9fa5]+', text.strip()):
for w in dt.cut(sentence):
if len(w.word) > 2 and w.flag in FLAGS:
yield w.word
class TFIDF:
def __init__(self, idf):
self.idf = idf
@classmethod
def train(cls, texts):
model = TfidfVectorizer(tokenizer=cut)
model.fit(texts)
idf = {w: model.idf_[i] for w, i in model.vocabulary_.items()}
return cls(idf)
def get_idf(self, word):
return self.idf.get(word, max(self.idf.values()))
def extract(self, text, top_n=10):
counter = Counter()
for w in cut(text):
counter[w] += self.get_idf(w)
#return [i[0:2] for i in counter.most_common(top_n)]
return [i[0] for i in counter.most_common(top_n)]
if __name__ == '__main__':
t0 = time()
with open('./nlp-homework.txt', encoding='utf-8')as f:
_texts = f.read().strip().split('\n')
# print(_texts)
tfidf = TFIDF.train(_texts)
# print(_texts)
for _text in _texts:
seq_list=jieba.cut(_text,cut_all=True) #全模式
# seq_list=jieba.cut(_text,cut_all=False) #精确模式
# seq_list=jieba.cut_for_search(_text,) #搜索引擎模式
# print(list(seq_list))
print(tfidf.extract(_text))
with open('./resultciyun.txt','a+', encoding='utf-8') as g:
for i in tfidf.extract(_text):
g.write(str(i) + " ")
print(time() - t0)
2. 生成词云:
代码如下:
注意需要安装
pip install wordcloud
;以及为了保证中文字体正常显示,需要下载
SimSun.ttf
字体,并且将这个字体包也放在和程序相同的目录下;
from wordcloud import WordCloud
filename = "resultciyun.txt"
with open(filename) as f:
resultciyun = f.read()
wordcloud = WordCloud(font_path="simsun.ttf").generate(resultciyun)
# %pylab inline
import matplotlib.pyplot as plt
plt.imshow(wordcloud, interpolation='bilinear')
plt.axis("off")
plt.show()
3 最后词云的图片
来源:https://blog.csdn.net/biggbang/article/details/129553871
标签:Python,tf-idf算法,关键字权重,生成词云
![](/images/zang.png)
![](/images/jiucuo.png)
猜你喜欢
使用sp_xml_preparedocument处理XML文档的方法
2011-09-30 11:56:15
有关asp的系统变量ServerVariables (“HTTP_USER_AGENT“)?
2009-10-29 12:12:00
Python 动态绑定属性和方法
2021-03-02 21:25:34
Python实现非正太分布的异常值检测方式
2021-11-22 12:27:57
![](https://img.aspxhome.com/file/2023/4/76054_0s.jpg)
python实现实时监控文件的方法
2021-08-18 21:17:06
javascript 时间脚本收集
2013-07-17 19:52:50
python+matplotlib演示电偶极子实例代码
2021-07-09 12:52:35
![](https://img.aspxhome.com/file/2023/3/77673_0s.jpg)
python实现控制台打印的方法
2021-12-18 12:21:04
八种获取当前日期的js代码
2007-09-07 10:26:00
python实现聚类算法原理
2022-07-23 23:42:33
![](https://img.aspxhome.com/file/2023/8/80628_0s.gif)
QQ聊天窗口链接提示效果代码
2008-12-16 12:59:00
instanceof 内部机制探析
2009-09-25 13:09:00
asp连接mysql的问题(端口)
2009-08-21 13:23:00
Python matplotlib绘制散点图配置(万能模板案例)
2023-08-02 23:04:18
![](https://img.aspxhome.com/file/2023/9/61989_0s.png)
可输入的select代码
2009-12-26 18:25:00
Pytest mark使用实例及原理解析
2021-02-27 14:30:11
Python中类的初始化特殊方法
2021-05-26 14:50:01
asp日期转换成汉字格式程序
2008-07-08 18:19:00
一个奇怪的CSS现象
2010-02-10 12:28:00
![](https://img.aspxhome.com/file/UploadPic/20102/10/y200921122655-79s.jpg)
使用ACCESS做网络版程序的四种解决方案
2009-01-14 16:22:00