如何减少MapReduce任务中不必要的线程

我正在编写一个非常简单的mapreduce作业来测试运行了多少线程并发,代码如下:

    System.out.println("--42-- thread num in main is : " + Thread.activeCount());
    JobConf jconf2 = new JobConf(TestPermission.class);
    System.out.println("--44-- thread num in main is : " + Thread.activeCount());

    jconf2.setMapOutputKeyClass(IntWritable.class);
    jconf2.setMapOutputvalueClass(NullWritable.class);
    System.out.println("--48-- thread num in main is : " + Thread.activeCount());

    jconf2.setMapperClass(Map1.class);
    System.out.println("--55-- thread num in main is : " + Thread.activeCount());

    jconf2.setInputFormat(TextInputFormat.class);
    System.out.println("--58-- thread num in main is : " + Thread.activeCount());
    jconf2.setOutputFormat(NothingOutputFormat.class);
    System.out.println("--60-- thread num in main is : " + Thread.activeCount());

    jconf2.set("mapred.map.tasks",String.valueOf(1));
    System.out.println("--63-- thread num in main is : " + Thread.activeCount());
    jconf2.set("mapred.reduce.tasks",String.valueOf(0));
    System.out.println("--65-- thread num in main is : " + Thread.activeCount());

    jconf2.set("mapreduce.reduce.shuffle.parallelcopies","0");
    System.out.println("--68-- thread num in main is : " + Thread.activeCount());
    jconf2.set("mapreduce.tasktracker.http.threads","0");
    System.out.println("--70-- thread num in main is : " + Thread.activeCount());
    jconf2.set("mapreduce.jobtracker.jobinit.threads","0");
    System.out.println("--72-- thread num in main is : " + Thread.activeCount());


    TextInputFormat.setInputPaths(jconf2,new Path(args[0]));
    System.out.println("--76-- thread num in main is : " + Thread.activeCount());
    NothingOutputFormat.setOutputPath(jconf2,new Path(args[1]));
    System.out.println("--78-- thread num in main is : " + Thread.activeCount());

    Path output_path = new Path(args[1]);
    FileSystem fs = output_path.getFileSystem(jconf2);
    System.out.println("--82-- thread num in main is : " + Thread.activeCount());
    FsPermission fper = new FsPermission(
            Fsaction.ALL,//user action
            Fsaction.ALL,//group action
            Fsaction.ALL);

    giveAllAthority(fs,output_path,fper);

    System.out.println("---90---thread num in main is : " + Thread.activeCount());

    JobClient.runJob(jconf2);
    System.out.println("--93-- thread num in main is : " + Thread.activeCount());

我运行作业时,结果如下:

--42-- thread num in main is : 1
--44-- thread num in main is : 1
--48-- thread num in main is : 1
--55-- thread num in main is : 1
--58-- thread num in main is : 1
--60-- thread num in main is : 1
--63-- thread num in main is : 1
--65-- thread num in main is : 1
--68-- thread num in main is : 1
--70-- thread num in main is : 1
--72-- thread num in main is : 1
--76-- thread num in main is : 3
--78-- thread num in main is : 3
--82-- thread num in main is : 3
---90---thread num in main is : 6

    ............

--93-- thread num in main is : 13

我很困惑,为什么活动线程在第76/90/93行增加。我的工作不需要洗牌,因此也不会减少。因此,如果不需要这些线程,我想知道如何关闭它们。

a10440377 回答:如何减少MapReduce任务中不必要的线程

暂时没有好的解决方案,如果你有好的解决方案,请发邮件至:iooj@foxmail.com
本文链接:https://www.f2er.com/3145939.html

大家都在问