在一些场景下,需要打开Spark集群的HDFS端口
云HBase答疑
(钉钉号),开通HDFS( 由于hdfs的开放可能造成用户的恶意攻击,引起集群不稳定甚至造成破坏。因此此功能暂时不直接开放给用户,当用户特别需要的情况下,我们通过云HBase答疑后台开通,随后客户使用完成,再关闭)添加以下两个hdfs配置到hadoop客户端conf目录(实例id需要替换为实际实例id);
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://${实例id}</value>
</property>
</configuration>
<configuration>
<property>
<name>dfs.nameservices</name>
<value>${实例id}</value>
</property>
<property>
<name>dfs.client.failover.proxy.provider.${实例id}</name>
<value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
</property>
<property>
<name>dfs.ha.automatic-failover.enabled.${实例id}</name>
<value>true</value>
</property>
<property>
<name>dfs.namenode.http-address.${实例id}.nn1</name>
<value>${实例id}-master1-001.spark.rds.aliyuncs.com:50070</value>
</property>
<property>
<name>dfs.namenode.http-address.${实例id}.nn2</name>
<value>${实例id}-master2-001.spark.rds.aliyuncs.com:50070</value>
</property>
<property>
<name>dfs.ha.namenodes.${实例id}</name>
<value>nn1,nn2</value>
</property>
<property>
<name>dfs.namenode.rpc-address.${实例id}.nn1</name>
<value>${实例id}-master1-001.spark.rds.aliyuncs.com:8020</value>
</property>
<property>
<name>dfs.namenode.rpc-address.${实例id}.nn2</name>
<value>${实例id}-master2-001.spark.rds.aliyuncs.com:8020</value>
</property>
</configuration>
实例ID查看:
添加conf路径到hadoop客户端classpath中
hadoop-env.sh添加或修改行:
export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:"hadoop conf路径"
添加HADOOP_CONF_DIR变量到系统中
export HADOOP_CONF_DIR="hadoop conf路径"
将该配置加入到系统默认加载的配置文件中,如${HOME}/.bashrc或${HOME}/.bash_profile,然后source一下配置文件
验证hdfs端口能否正常访问
echo "hdfs port test" >/tmp/test
hadoop dfs -put /tmp/test /
hadoop dfs -cat /test
在文档使用中是否遇到以下问题
更多建议
匿名提交