1. Cluster distribution file
Application scenario
We often need to copy the newly created and modified files to the same directory of all nodes. It is not so friendly to execute the scp command once.
Basic knowledge
(a) rsync command original copy:
[root@bigdata801 hadoop-3.3.1]# rsync -av /opt/module/hadoop-3.3.1/ bigdata802:/opt/module/hadoop-3.3.1/
(b) Expected script:
xsync Name of the file to synchronize
(c) It is expected that the script can be used in any path (the script is placed in the path where the global environment variable is declared)
[root@bigdata801 bin]# echo $PATH /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/opt/module/jdk1.8.0_181/bin:/opt/module/hadoop-3.3.1/bin:/opt/module/hadoop-3.3.1/sbin:/root/bin:/opt/module/jdk1.8.0_181/bin:/opt/module/hadoop-3.3.1/bin:/opt/module/hadoop-3.3.1/sbin:/opt/module/jdk1.8.0_181/bin:/opt/module/hadoop-3.3.1/bin:/opt/module/hadoop-3.3.1/sbin:/opt/module/jdk1.8.0_181/bin:/opt/module/hadoop-3.3.1/bin:/opt/module/hadoop-3.3.1/sbin
Script implementation
(a) Create an xsync file in the / root/bin directory
[root@bigdata801 ~]# cd /root/ [root@bigdata801 ~]# mkdir bin [root@bigdata801 ~]# cd bin/ [root@bigdata801 bin]# vim xsync
Write the following code in this file
#!/bin/bash
#1. Number of judgment parameters
if [ $# -lt 1 ]
then
echo Not Enough Arguement!
exit;
fi
#2. Traverse all machines in the cluster
for host in bigdata801 bigdata802 bigdata803 bigdata804
do
echo ==================== $host ====================
#3. Traverse all directories and send them one by one
for file in $@
do
#4. Judge whether the file exists
if [ -e $file ]
then
#5. Get parent directory
pdir=$(cd -P $(dirname $file); pwd)
#6. Get the name of the current file
fname=$(basename $file)
ssh $host "mkdir -p $pdir"
rsync -av $pdir/$fname $host:$pdir
else
echo $file does not exists!
fi
done
done
The modified script xsync has execution permission
[root@bigdata801 bin]# chmod 777 xsync [root@bigdata801 bin]# ll Total consumption 4 -rwxrwxrwx 1 root root 753 8 June 22-20:45 xsync
Test script
Distribute the newly created bin to other machines
[root@bigdata801 ~]# xsync bin/
The execution process is as follows
[root@bigdata801 ~]# xsync bin/ ==================== bigdata801 ==================== sending incremental file list sent 75 bytes received 17 bytes 184.00 bytes/sec total size is 753 speedup is 8.18 ==================== bigdata802 ==================== sending incremental file list bin/ bin/xsync sent 878 bytes received 39 bytes 611.33 bytes/sec total size is 753 speedup is 0.82 ==================== bigdata803 ==================== sending incremental file list bin/ bin/xsync sent 878 bytes received 39 bytes 1,834.00 bytes/sec total size is 753 speedup is 0.82 ==================== bigdata804 ==================== sending incremental file list bin/ bin/xsync sent 878 bytes received 39 bytes 611.33 bytes/sec total size is 753 speedup is 0.82 [root@bigdata801 ~]#
Check whether the file is successfully distributed on bigdata802
[root@bigdata802 hadoop-3.3.1]# cd /root/[root@bigdata802 ~]# ll Total consumption 4-rw-------. 1 root root 1460 2 June 28-19:08 anaconda-ks.cfgdrwxr-xr-x. 2 root root 19 8 June 22-20:47 bin[root@bigdata802 ~]#
2. View the status of all nodes on a node (JPS)
Application scenario
After starting the cluster, we need to check whether the cluster is started successfully. The key step is to enter JSP commands on each service to view the corresponding services on the server. However, it is too troublesome to enter them on each machine every time, so write the following script.
Script implementation
Create the jpsall file under / root/bin
[root@bigdata801 bin]# cd /root/bin/[root@bigdata801 bin]# vim jpsall
The script is as follows
#!/bin/bash #Execute the JPS command to query the node status on each server echo ============================== query the status of each server ============= for host in bigdata801 bigdata802 bigdata803 bigdata804do echo =================== $host =========== SSH $host '/ opt / module / jdk1.8.0_ 181 / bin / jps'doneecho ===================== end of querying server status===============
Grant script execution permission
[root@bigdata801 bin]# chmod +x jpsall
Distribution / root/bin / directory
Ensure that custom scripts can be used on every machine
[root@bigdata801 bin]# xsync /root/bin/
test
[root@bigdata801 bin]# jpsall ===============Query the status of each server============================== bigdata801 ===============5712 JobHistoryServer5202 DataNode5542 NodeManager5047 NameNode5967 Jps=============== bigdata802 ===============3156 NodeManager3461 Jps3064 SecondaryNameNode2943 DataNode=============== bigdata803 ===============3488 ResourceManager3281 DataNode3651 NodeManager4139 Jps=============== bigdata804 ===============2576 DataNode2699 NodeManager2957 Jps===============End of query server status===============[root@bigdata801 bin]#
3. Execute unified commands on multiple machines
Application scenario
After we modify the environment variable, we need to reload the source. It is not so friendly for each node to execute
Script implementation
Create the call file in the / root/bin folder
[root@bigdata801 bin]# pwd/root/bin[root@bigdata801 bin]# vim callall
The script is as follows
#!/bin/bash#Execute the same command in batches on all machines in the cluster if(($#==0))then echo Please enter the command you want to operate! exitfiecho The commands to execute are: $*#Loop through this command for ((I = 801; I < = 804; I + +) do echo ----------------------- bigdata $I ----------------------- SSH bigdata $I $* done
Grant executable rights
[root@bigdata801 bin]# chmod 777 callall
Distribute other machines
[root@bigdata801 bin]# xsync /root/bin/==================== bigdata801 ====================sending incremental file listsent 150 bytes received 17 bytes 334.00 bytes/sectotal size is 2,556 speedup is 15.31==================== bigdata802 ====================sending incremental file listbin/bin/callallsent 489 bytes received 39 bytes 1,056.00 bytes/sectotal size is 2,556 speedup is 4.84==================== bigdata803 ====================sending incremental file listbin/bin/callallsent 489 bytes received 39 bytes 352.00 bytes/sectotal size is 2,556 speedup is 4.84==================== bigdata804 ====================sending incremental file listbin/bin/callallsent 489 bytes received 39 bytes 1,056.00 bytes/sectotal size is 2,556 speedup is 4.84[root@bigdata801 bin]#
test
[root@bigdata801 bin]# callall ls -l The command to execute is ls -l---------------------bigdata801-----------------Total consumption 4-rw-------. 1 root root 1460 2 June 28-19:08 anaconda-ks.cfgdrwxr-xr-x 2 root root 67 8 June 29-14:11 bin---------------------bigdata802-----------------Total consumption 4-rw-------. 1 root root 1460 2 June 28-19:08 anaconda-ks.cfgdrwxr-xr-x. 2 root root 67 8 June 29-14:11 bin---------------------bigdata803-----------------Total consumption 4-rw-------. 1 root root 1460 2 June 28-19:08 anaconda-ks.cfgdrwxr-xr-x. 2 root root 67 8 June 29-14:11 bin---------------------bigdata804-----------------Total consumption 4-rw-------. 1 root root 1460 2 June 28-19:08 anaconda-ks.cfgdrwxr-xr-x. 2 root root 67 8 June 29-14:11 bin[root@bigdata801 bin]#
be careful
If you enter callall, an error will be reported bash: ll: command not found
This is because the "ll" command is not the basic command of linux. It is an alias of "ls -l". Some versions do not directly support the output of the "ll" command.
[root@bigdata801 bin]# callall ll The command to execute is ll---------------------bigdata801-----------------bash: ll: Command not found---------------------bigdata802-----------------bash: ll: Command not found---------------------bigdata803-----------------bash: ll: Command not found---------------------bigdata804-----------------bash: ll: Command not found[root@bigdata801 bin]#