Datanode 未在 Windows 10 for Hadoop 3.1.3

我正在尝试在 Windows 10 上为 hadoop 3.1.3 启动 datanode 和 namenode,并且我将所需的 winutils.exe 和 hadoop.dll 保留在 bin 文件夹以及系统 32 文件夹中。但我仍然低于数据节点的例外:


2021-04-09 13:45:24,464 INFO checker.ThrottledAsyncChecker: Scheduling a check for [DISK]file:/c:/Sankha/Study/hadoop-3.1.3/hadoop-3.1.3/data/datanode
2021-04-09 13:45:24,691 WARN checker.StoragelocationChecker: Exception checking Storagelocation [DISK]file:/c:/Sankha/Study/hadoop-3.1.3/hadoop-3.1.3/data/datanode
java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.nativeio.NativeIO$POSIX.stat(Ljava/lang/String;)Lorg/apache/hadoop/io/nativeio/NativeIO$POSIX$Stat;
        at org.apache.hadoop.io.nativeio.NativeIO$POSIX.stat(Native Method)
        at org.apache.hadoop.io.nativeio.NativeIO$POSIX.getStat(NativeIO.java:455)
        at org.apache.hadoop.fs.RawLocalFileSystem$DeprecatedRawLocalFileStatus.loadPermissionInfoByNativeIO(RawLocalFileSystem.java:796)
        at org.apache.hadoop.fs.RawLocalFileSystem$DeprecatedRawLocalFileStatus.loadPermissionInfo(RawLocalFileSystem.java:710)
        at org.apache.hadoop.fs.RawLocalFileSystem$DeprecatedRawLocalFileStatus.getPermission(RawLocalFileSystem.java:678)
        at org.apache.hadoop.util.DiskChecker.mkdirsWithExistsAndPermissionCheck(DiskChecker.java:233)
        at org.apache.hadoop.util.DiskChecker.checkDirInternal(DiskChecker.java:141)
        at org.apache.hadoop.util.DiskChecker.checkDir(DiskChecker.java:116)
        at org.apache.hadoop.hdfs.server.datanode.Storagelocation.check(Storagelocation.java:239)
        at org.apache.hadoop.hdfs.server.datanode.Storagelocation.check(Storagelocation.java:52)
        at org.apache.hadoop.hdfs.server.datanode.checker.ThrottledAsyncChecker$1.call(ThrottledAsyncChecker.java:142)
        at com.google.common.util.concurrent.TrustedListenableFuturetask$TrustedFutureInterruptibletask.runInterruptibly(TrustedListenableFuturetask.java:125)
        at com.google.common.util.concurrent.Interruptibletask.run(Interruptibletask.java:57)
        at com.google.common.util.concurrent.TrustedListenableFuturetask.run(TrustedListenableFuturetask.java:78)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
2021-04-09 13:45:24,706 ERROR datanode.DataNode: Exception in secureMain
org.apache.hadoop.util.DiskChecker$DiskErrorException: Too many failed volumes - current valid volumes: 0,volumes configured: 1,volumes failed: 1,volume failures tolerated: 0
        at org.apache.hadoop.hdfs.server.datanode.checker.StoragelocationChecker.check(StoragelocationChecker.java:231)
        at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:2799)
        at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:2714)
        at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:2756)
        at org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:2900)
        at org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:2924)
2021-04-09 13:45:24,765 INFO util.ExitUtil: Exiting with status 1: org.apache.hadoop.util.DiskChecker$DiskErrorException: Too many failed volumes - current valid volumes: 0,volume failures tolerated: 0
2021-04-09 13:45:24,816 INFO datanode.DataNode: SHUTDOWN_MSG:

我知道同一问题有不同的线程。我去那里尝试了所有类似的方法

以管理员身份运行, 获取正确的 winutils.exe 和 hadoop.dll 并放置在 hadoop/bin 和 windows 的 system32 文件夹中。 但没有任何结果。

下面是我的配置xml:

yarn-site.xml
--------------------
<configuration>

<!-- Site specific YARN configuration properties -->
    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>
    <property>
        <name>yarn.nodemanager.auxservices.mapreduce.shuffle.class</name>
        <value>org.apache.hadoop.mapred.ShuffleHandler</value>
    </property>
</configuration>

mapred-site.xml
-------------------
<configuration>
    <property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>
</configuration>

hdfs-site.xml
------------------
<configuration>
    <property>
        <name>dfs.replication</name>
        <value>1</value>
    </property>
    <property>
        <name>dfs.namenode.name.dir</name>
        <value>/Study/hadoop-3.1.3/hadoop-3.1.3/data/namenode</value>
    </property>
    <property>
        <name>dfs.datanode.data.dir</name>
        <value>/Study/hadoop-3.1.3/hadoop-3.1.3/data/datanode</value>
    </property> 
    <property>
        <name>dfs.datanode.failed.volumes.tolerated</name>
        <value>0</value>
    </property> 
</configuration>

core-site.xml
-----------------
<configuration>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://localhost:9000</value>
    </property>
</configuration>
zhao_xunze 回答:Datanode 未在 Windows 10 for Hadoop 3.1.3

暂时没有好的解决方案,如果你有好的解决方案,请发邮件至:iooj@foxmail.com
本文链接:https://www.f2er.com/609094.html

大家都在问