5.RDD操作综合实例
作者:互联网
一、词频统计
A. 分步骤实现
- 1.准备文件
-
- 下载小说或长篇新闻稿
![](https://www.icode9.com/i/l/?n=22&i=blog/2763869/202203/2763869-20220329151457006-1333995375.png)
-
- 上传到hdfs上
![](https://www.icode9.com/i/l/?n=22&i=blog/2763869/202203/2763869-20220329151426070-1608416271.png)
- 2.读文件创建RDD
![](https://www.icode9.com/i/l/?n=22&i=blog/2763869/202203/2763869-20220329151626576-965063847.png)
- 3.分词
![](https://www.icode9.com/i/l/?n=22&i=blog/2763869/202203/2763869-20220329232054097-1401913952.png)
- 4.标点符号re.split(pattern,str),flatMap(),
先导入re然后用re.split分词(\W+会匹配所有非单词字符,(\W+)会返回这些,但我们不需要返回,所以这里用\W+即可)
![](https://www.icode9.com/i/l/?n=22&i=blog/2763869/202203/2763869-20220329232441036-1074074644.png)
再分词一遍清除去标点后产生的新的无用数据
![](https://www.icode9.com/i/l/?n=22&i=blog/2763869/202203/2763869-20220329232524149-887740934.png)
-
- 排除大小写lower(),map()
![](https://www.icode9.com/i/l/?n=22&i=blog/2763869/202203/2763869-20220329232754724-1338302666.png)
-
- 停用词,可网盘下载stopwords.txt,filter(),
![](https://www.icode9.com/i/l/?n=22&i=blog/2763869/202203/2763869-20220329183043899-1607824733.png)
将停用词文件分词储存到变量中
![](https://www.icode9.com/i/l/?n=22&i=blog/2763869/202203/2763869-20220329233406066-1309436307.png)
筛选出不在停用表中的词
![](https://www.icode9.com/i/l/?n=22&i=blog/2763869/202203/2763869-20220329233708177-649542830.png)
筛选前后对比
![](https://www.icode9.com/i/l/?n=22&i=blog/2763869/202203/2763869-20220329233748001-189365494.png)
-
- 长度小于2的词filter()
![](https://www.icode9.com/i/l/?n=22&i=blog/2763869/202203/2763869-20220329233951177-1609076491.png)
筛选前后对比(因为许多小于2的词在停用表中已经被筛选过了一次了)
![](https://www.icode9.com/i/l/?n=22&i=blog/2763869/202203/2763869-20220329234030216-2004754637.png)
- 5.统计词频
先把单词映射成键值对
![](https://www.icode9.com/i/l/?n=22&i=blog/2763869/202203/2763869-20220329234240163-1059469042.png)
将key相同的values合并起来,做词频的统计
![](https://www.icode9.com/i/l/?n=22&i=blog/2763869/202203/2763869-20220329234603129-73087744.png)
- 6.按词频排序
用sortBy进行词频的排序(对values值进行降序排序,并且指定区块数量为1,以方便查文件)
![](https://www.icode9.com/i/l/?n=22&i=blog/2763869/202203/2763869-20220329235030877-1153362366.png)
- 7.输出到文件
分别输出到本地文件和分布式文件上
![](https://www.icode9.com/i/l/?n=22&i=blog/2763869/202203/2763869-20220329235301978-1595959785.png)
![](https://www.icode9.com/i/l/?n=22&i=blog/2763869/202203/2763869-20220329235330748-1107539318.png)
- 8.查看结果
查看文件前15条结果是否正确即可
![](https://www.icode9.com/i/l/?n=22&i=blog/2763869/202203/2763869-20220329235522955-1918124892.png)
![](https://www.icode9.com/i/l/?n=22&i=blog/2763869/202203/2763869-20220329235640849-1107677857.png)
B. 一句话实现:文件入文件出
通过代码结合实现一句话完成词频排序并保存文件
![](https://www.icode9.com/i/l/?n=22&i=blog/2763869/202203/2763869-20220330000959018-274973310.png)
查看文件(由于没有去除停用表内的词,所以结果与前面不同)
![](https://www.icode9.com/i/l/?n=22&i=blog/2763869/202203/2763869-20220330001217802-705943452.png)
二、求Top值
- 网盘下载payment.txt文件,通过RDD操作实现选出最大支付额的用户。
标签:文件,re,RDD,实例,词频,操作,排序,分词 来源: https://www.cnblogs.com/Yeav/p/16073702.html