我正在编写一个非常简单的mapreduce作业来测试运行了多少线程并发,代码如下:
System.out.println("--42-- thread num in main is : " + Thread.activeCount());
JobConf jconf2 = new JobConf(TestPermission.class);
System.out.println("--44-- thread num in main is : " + Thread.activeCount());
jconf2.setMapOutputKeyClass(IntWritable.class);
jconf2.setMapOutputvalueClass(NullWritable.class);
System.out.println("--48-- thread num in main is : " + Thread.activeCount());
jconf2.setMapperClass(Map1.class);
System.out.println("--55-- thread num in main is : " + Thread.activeCount());
jconf2.setInputFormat(TextInputFormat.class);
System.out.println("--58-- thread num in main is : " + Thread.activeCount());
jconf2.setOutputFormat(NothingOutputFormat.class);
System.out.println("--60-- thread num in main is : " + Thread.activeCount());
jconf2.set("mapred.map.tasks",String.valueOf(1));
System.out.println("--63-- thread num in main is : " + Thread.activeCount());
jconf2.set("mapred.reduce.tasks",String.valueOf(0));
System.out.println("--65-- thread num in main is : " + Thread.activeCount());
jconf2.set("mapreduce.reduce.shuffle.parallelcopies","0");
System.out.println("--68-- thread num in main is : " + Thread.activeCount());
jconf2.set("mapreduce.tasktracker.http.threads","0");
System.out.println("--70-- thread num in main is : " + Thread.activeCount());
jconf2.set("mapreduce.jobtracker.jobinit.threads","0");
System.out.println("--72-- thread num in main is : " + Thread.activeCount());
TextInputFormat.setInputPaths(jconf2,new Path(args[0]));
System.out.println("--76-- thread num in main is : " + Thread.activeCount());
NothingOutputFormat.setOutputPath(jconf2,new Path(args[1]));
System.out.println("--78-- thread num in main is : " + Thread.activeCount());
Path output_path = new Path(args[1]);
FileSystem fs = output_path.getFileSystem(jconf2);
System.out.println("--82-- thread num in main is : " + Thread.activeCount());
FsPermission fper = new FsPermission(
Fsaction.ALL,//user action
Fsaction.ALL,//group action
Fsaction.ALL);
giveAllAthority(fs,output_path,fper);
System.out.println("---90---thread num in main is : " + Thread.activeCount());
JobClient.runJob(jconf2);
System.out.println("--93-- thread num in main is : " + Thread.activeCount());
我运行作业时,结果如下:
--42-- thread num in main is : 1
--44-- thread num in main is : 1
--48-- thread num in main is : 1
--55-- thread num in main is : 1
--58-- thread num in main is : 1
--60-- thread num in main is : 1
--63-- thread num in main is : 1
--65-- thread num in main is : 1
--68-- thread num in main is : 1
--70-- thread num in main is : 1
--72-- thread num in main is : 1
--76-- thread num in main is : 3
--78-- thread num in main is : 3
--82-- thread num in main is : 3
---90---thread num in main is : 6
............
--93-- thread num in main is : 13
我很困惑,为什么活动线程在第76/90/93行增加。我的工作不需要洗牌,因此也不会减少。因此,如果不需要这些线程,我想知道如何关闭它们。