[报错] spark开发环境 操作hdfs 报错

报错1 

22/05/0305:48:53 WARN DFSClient: Failed to connect to /10.0.24.10:9866 for block, add to deadNodes and continue. org.apache.hadoop.net.ConnectTimeoutException: 60000 millis timeout while waiting for channel to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending remote=/10.0.24.10:9866]
org.apache.hadoop.net.ConnectTimeoutException: 60000 millis timeout while waiting for channel to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending remote=/10.0.24.10:9866]

查看报错信息, 是连接 datanode: 9866 时报错,为什么会使用 内网ip呢？有没有方法可以避免呢？

解决:

在resource目录添加 hdfs-site.xml

    <!-- datanode 通信是否使用域名,默认为false，改为true -->
    <property>
        <name>dfs.client.use.datanode.hostname</name>
        <value>true</value>
        <description>Whether datanodes should use datanode hostnames whenconnecting to other datanodes for data transfer.
        </description>
    </property>

报错2

 Exception in thread "main" org.apache.hadoop.security.AccessControlException: Permission denied: user=wst, access=WRITE, inode="/":root:supergroup:drwxr-xr-x

查看报错信息,是往 hdfs写数据时,没有权限,因为使用的是 开发环境的user=wst提交的

  //TODO 5.对hdfs 的操作
  object hdfs {
    def main(args: Array[String]): Unit = {
　　　　// 设置 hadoop用户名
      System.setProperty("HADOOP_USER_NAME", "root")

标签：hdfs,10.0,hadoop,报错,datanode,spark,9866
来源： https://www.cnblogs.com/bajiaotai/p/16217571.html