E-MapReduce客戶端安裝和卸載工具，實現Gateway功能

作者：陳雲亮，系深圳某數據治理廠商，有10餘年IT從業經驗，在數據倉庫，傳統數據庫和大數據領域有豐富的實戰經驗。
在阿里雲某私有云的項目中，遇到了需要在E-MapReduce集群節點外使用客戶端的需求，而目前在集群管理中只提供了通過Gateway新建一臺ECS來滿足。但是，由於客戶方ECS資源已經申請，所以無法使用Gateway，基於此，通過對E-MapReduce的官方文檔和社群資源學習，深入集群瞭解，特開發一套部署客戶端程序。
（PS：在華為FIHD大數據平臺，星環和CDH等大數據平臺的管理界面都提供了下載客戶端安裝包並能實現一鍵安裝部署。希望下個版本能有這樣的功能。）
E-MapReduce客戶端安裝和卸載工具，主要用於集群外的ECS（已經存在）中需要使用hadoop，hive等客戶端命令工具。
目前雲平臺提供的gateway只能重新創建ECS進行部署，而不支持在已經創建的Ecs中部署。
主要涉及的程序：

[root@iZ1cr01rx0b6j12dir0qaxZ chyl20]# ls -ltr
total 12
-rw-r--r-- 1 root root   13 Oct 10 14:38 pwd.txt
-rwxr-xr-x 1 root root 1195 Oct 11 10:32 uninstall_emr_client.sh
-rwxr-xr-x 1 root root 3337 Oct 11 10:32 install_emr_client.sh

其中install_emr_client.sh的程序內容：

#!/usr/bin/bash
if [ $# != 2 ]
then
   echo "Usage: sh $0 master_ip master_password_file(store root password plaintext)"
   echo "such as: sh $0 172.30.XXX.XXX /opt/chyl20/pwd.txt"
   exit 1;
fi
masterip=${1}
masterpwdfile=${2}
client_dir="/opt/emr_client/"
if ! type sshpass >/dev/null 2>&1; then
   yum install -y sshpass
fi
if ! type java >/dev/null 2>&1; then
   yum install -y java-1.8.0-openjdk
fi
[ -d ${client_dir} ] && echo "Please uninstall the installed client!" && exit 1

echo "Start to create file path..."
mkdir -p ${client_dir}
mkdir -p /etc/ecm
echo "Start to copy package from ${masterip} to local client(${client_dir})..."
echo " -copying hadoop"
sshpass -f ${masterpwdfile} scp -r -o 'StrictHostKeyChecking no' root@${masterip}:/usr/lib/hadoop-current ${client_dir}
echo " -copying zookeeper"
sshpass -f ${masterpwdfile} scp -r root@${masterip}:/usr/lib/zookeeper-current ${client_dir}
echo " -copying hive"
sshpass -f ${masterpwdfile} scp -r root@${masterip}:/usr/lib/hive-current ${client_dir}
echo " -copying spark"
sshpass -f ${masterpwdfile} scp -r root@${masterip}:/usr/lib/spark-current ${client_dir}
echo " -extra-jars"
sshpass -f ${masterpwdfile} scp -r root@${masterip}:/opt/apps/extra-jars ${client_dir}

echo "Start to link /usr/lib/\${app}-current to ${client_dir}\${app}..."
[ -L /usr/lib/hadoop-current ] && unlink /usr/lib/hadoop-current
ln -s ${client_dir}hadoop-current  /usr/lib/hadoop-current

[ -L /usr/lib/zookeeper-current ] && unlink /usr/lib/zookeeper-current
ln -s ${client_dir}zookeeper-current  /usr/lib/zookeeper-current

[ -L /usr/lib/hive-current ] && unlink /usr/lib/hive-current
ln -s ${client_dir}hive-current  /usr/lib/hive-current

[ -L /usr/lib/spark-current ] && unlink /usr/lib/spark-current
ln -s ${client_dir}spark-current /usr/lib/spark-current

echo "Start to copy conf from ${masterip} to local client(/etc/ecm)..."
sshpass -f ${masterpwdfile} scp -r root@${masterip}:/etc/ecm/hadoop-conf  /etc/ecm/hadoop-conf
sshpass -f ${masterpwdfile} scp -r root@${masterip}:/etc/ecm/zookeeper-conf  /etc/ecm/zookeeper-conf
sshpass -f ${masterpwdfile} scp -r root@${masterip}:/etc/ecm/hive-conf /etc/ecm/hive-conf
sshpass -f ${masterpwdfile} scp -r root@${masterip}:/etc/ecm/spark-conf /etc/ecm/spark-conf

echo "Start to copy environment from ${masterip} to local client(/etc/profile.d)..."
sshpass -f ${masterpwdfile} scp root@${masterip}:/etc/profile.d/hdfs.sh /etc/profile.d/
sshpass -f ${masterpwdfile} scp root@${masterip}:/etc/profile.d/zookeeper.sh /etc/profile.d/
sshpass -f ${masterpwdfile} scp root@${masterip}:/etc/profile.d/yarn.sh /etc/profile.d/
sshpass -f ${masterpwdfile} scp root@${masterip}:/etc/profile.d/hive.sh /etc/profile.d/
sshpass -f ${masterpwdfile} scp root@${masterip}:/etc/profile.d/spark.sh /etc/profile.d/
if [ -L /usr/lib/jvm/java ]
then
   unlink /usr/lib/jvm/java
fi
echo "" >>/etc/profile.d/hdfs.sh
echo export JAVA_HOME=/usr/lib/jvm/jre-1.8.0 >>/etc/profile.d/hdfs.sh

echo "Start to copy host info from ${masterip} to local client(/etc/hosts)"
sshpass -f ${masterpwdfile} scp root@${masterip}:/etc/hosts /etc/hosts_bak
cat /etc/hosts_bak | grep emr | grep cluster >>/etc/hosts
! egrep "^hadoop" /etc/group >& /dev/null && groupadd hadoop
! egrep "^hadoop" /etc/passwd >& /dev/null && useradd -g hadoop hadoop
echo "Install EMR client successfully!"

uninstall_emr_client.sh的程序內容：

#!/usr/bin/bash
if [ $# != 0 ]
then
   echo "Usage: sh $0 "
   exit 1;
fi
client_dir="/opt/emr_client/"

echo "Start to delete path..."
[ -d ${client_dir} ] && rm -rf ${client_dir}

echo "Start to delete link /usr/lib/\${app}-current..."
[ -L /usr/lib/hadoop-current ] && unlink /usr/lib/hadoop-current

[ -L /usr/lib/zookeeper-current ] && unlink /usr/lib/zookeeper-current

[ -L /usr/lib/hive-current ] && unlink /usr/lib/hive-current

[ -L /usr/lib/spark-current ] && unlink /usr/lib/spark-current

echo "Start to delete conf..."
[ -d /etc/ecm ] && rm -rf /etc/ecm

echo "Start to delete environment..."
[ -f /etc/profile.d/hdfs.sh ] && rm -rf /etc/profile.d/hdfs.sh
[ -f /etc/profile.d/zookeeper.sh ] && rm -rf /etc/profile.d/zookeeper.sh
[ -f /etc/profile.d/yarn.sh ] && rm -rf /etc/profile.d/yarn.sh
[ -f /etc/profile.d/hive.sh ] && rm -rf /etc/profile.d/hive.sh
[ -f /etc/profile.d/spark.sh ] && rm -rf /etc/profile.d/spark.sh

echo "Start to remove host info..."
sed -i "/emr/d" /etc/hosts

echo "Start to remove user..."
egrep "^hadoop" /etc/passwd >& /dev/null && userdel -r hadoop
egrep "^hadoop" /etc/group >& /dev/null && groupdel hadoop

echo "Uninstall EMR client successfully!"

pwd.txt中存放密碼明文即可。

創建後給程序授權755
安裝執行：

sh install_emr_client.sh 172.30.1*2.1*3 /opt/chyl20/pwd.txt

卸載執行：

sh uninstall_emr_client.sh

安裝後重新連接會話，或重新打開一個PTY
驗證：
運行hive

beeline -u jdbc:hive2://emr-header-1:10000 -e "show tables; " -n hadoop -p hadoop

運行hadoop

hadoop  jar /usr/lib/hadoop-current/share/hadoop/mapreduce/hadoop-mapreduce-examples*.jar pi 10 10

運行spark

spark-submit --name spark-pi --class org.apache.spark.examples.SparkPi --master yarn /usr/lib/spark-current/examples/jars/spark-examples_2.11-2.4.3.jar