Java执行hadoop的基本操作实例代码
作者:lqh 时间:2022-11-13 12:08:04
Java执行hadoop的基本操作实例代码
向HDFS上传本地文件
public static void uploadInputFile(String localFile) throws IOException{
Configuration conf = new Configuration();
String hdfsPath = "hdfs://localhost:9000/";
String hdfsInput = "hdfs://localhost:9000/user/hadoop/input";
FileSystem fs = FileSystem.get(URI.create(hdfsPath), conf);
fs.copyFromLocalFile(new Path(localFile), new Path(hdfsInput));
fs.close();
System.out.println("已经上传文件到input文件夹啦");
}
将output文件下载到本地
public static void getOutput(String outputfile) throws IOException{
String remoteFile = "hdfs://localhost:9000/user/hadoop/output/part-r-00000";
Path path = new Path(remoteFile);
Configuration conf = new Configuration();
String hdfsPath = "hdfs://localhost:9000/";
FileSystem fs = FileSystem.get(URI.create(hdfsPath),conf);
fs.copyToLocalFile(path, new Path(outputfile));
System.out.println("已经将输出文件保留到本地文件");
fs.close();
}
删除hdfs中的文件
public static void deleteOutput() throws IOException{
Configuration conf = new Configuration();
String hdfsOutput = "hdfs://localhost:9000/user/hadoop/output";
String hdfsPath = "hdfs://localhost:9000/";
Path path = new Path(hdfsOutput);
FileSystem fs = FileSystem.get(URI.create(hdfsPath), conf);
fs.deleteOnExit(path);
fs.close();
System.out.println("output文件已经删除");
}
执行mapReduce程序
创建Mapper类和Reducer类
public static class TokenizerMapper extends Mapper<Object, Text, Text, IntWritable>{
private final static IntWritable one = new IntWritable(1);
private Text word = new Text();
public void map(Object key, Text value, Context context) throws IOException, InterruptedException{
String line = value.toString();
line = line.replace("\\", "");
String regex = "性别:</span><span class=\"pt_detail\">(.*?)</span>";
Pattern pattern = Pattern.compile(regex);
Matcher matcher = pattern.matcher(line);
while(matcher.find()){
String term = matcher.group(1);
word.set(term);
context.write(word, one);
}
}
}
public static class IntSumReducer extends Reducer<Text, IntWritable, Text, IntWritable>{
private IntWritable result = new IntWritable();
public void reduce(Text key, Iterable<IntWritable> values, Context context) throws IOException, InterruptedException{
int sum = 0;
for(IntWritable val :values){
sum+= val.get();
}
result.set(sum);
context.write(key, result);
}
}
执行mapReduce程序
public static void runMapReduce(String[] args) throws Exception {
Configuration conf = new Configuration();
String[] otherArgs = new GenericOptionsParser(conf, args).getRemainingArgs();
if(otherArgs.length != 2){
System.err.println("Usage: wordcount<in> <out>");
System.exit(2);
}
Job job = new Job(conf, "word count");
job.setJarByClass(WordCount.class);
job.setMapperClass(TokenizerMapper.class);
job.setCombinerClass(IntSumReducer.class);
job.setReducerClass(IntSumReducer.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(IntWritable.class);
FileInputFormat.addInputPath(job, new Path(otherArgs[0]));
FileOutputFormat.setOutputPath(job, new Path(otherArgs[1]));
System.out.println("mapReduce 执行完毕!");
System.exit(job.waitForCompletion(true)?0:1);
}
感谢阅读,希望能帮助到大家,谢谢大家对本站的支持!
来源:http://blog.csdn.net/qq_30843221/article/details/54429792
标签:Java,hadoop
0
投稿
猜你喜欢
将JavaDoc注释生成API文档的操作
2023-06-16 18:24:06
Android开发之OkHttpUtils的具体使用方法
2021-12-07 18:07:47
Android对图片Drawable实现变色示例代码
2021-11-25 15:49:47
Java设计模式之共享模式/享元模式(Flyweight模式)介绍
2023-06-15 04:35:10
适用于WebForm Mvc的Pager分页组件C#实现
2022-05-11 22:11:34
C#对DataTable中的某列进行分组
2021-12-10 23:06:45
C# 执行CMD命令并接收返回结果的操作方式
2023-09-03 10:36:56
Java创建型设计模式之建造者模式详解
2023-10-09 17:25:13
Mybatis日志参数快速替换占位符工具的详细步骤
2023-09-30 21:29:36
解决SpringBoot加载application.properties配置文件的坑
2023-02-05 19:55:27
Nacos 动态服务发现、配置和服务管理平台初体验
2022-09-10 23:56:53
详解Android内存优化策略
2022-01-03 13:52:10
SpringBoot集成mybatis实例
2023-03-09 16:57:01
在Spring Boot中实现HTTP缓存的方法
2023-10-06 14:18:16
Kotlin 使用Lambda来设置回调的操作
2021-07-22 03:08:04
Java由浅入深带你了解什么是包package
2022-04-17 02:33:39
使用java技术抓取网站上彩票双色球信息详解
2022-01-04 14:47:51
Java Fluent Mybatis 项目工程化与常规操作详解流程篇 下
2021-07-19 11:33:11
Java 使用 HttpClient 发送 GET请求和 POST请求
2023-07-23 07:56:13
如何使用Android注解处理器
2023-08-16 16:49:05