Transfer file command under linux

lrzsz

Introduction: lrzsz is a program that can be uploaded and downloaded in linux instead of ftp.

Installation: yum-y install lrzsz

Use:

• Upload: rz

• Download: sz filename

Example:

[root@server1 ~]# yum install -y lrzsz.x86_64 
.................
[root@server1 ~]# echo "September 9, 2002, 10.43 p.m." > test1
[root@server1 ~]# rz	# Upload to linux
# Select files to upload under windows
[root@server1 ~]# sz test1 	#Download Files to windows

scp

• Upload files between hosts

[root@server1 ~]# scp test1 root@192.168.174.20:/root/		
The authenticity of host '192.168.174.20 (192.168.174.20)' can't be established.
ECDSA key fingerprint is SHA256:l5y8+0D0KcBmCK2edW3ZSY7RyIDgBdlLOE10JAlVtKk.
ECDSA key fingerprint is MD5:60:21:02:4d:b8:ce:cd:bc:19:33:0d:58:d6:4c:bd:fe.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '192.168.174.20' (ECDSA) to the list of known hosts.
root@192.168.174.20's password: 
test1                                                    100%   49    26.6KB/s   00:00 

rsync

rsync is a data mirroring backup tool under class unix, remote sync.

The basic features of Rsync are as follows:

• You can mirror the entire directory tree and file system.
• It is very easy to keep the permissions, time, hard and soft links of the original file.
• Installation without special privileges; Optimized process, high file transfer efficiency;
• You can use rcp, ssh, etc. to transfer files, or you can use a direct socket connection.
• Support anonymous transmission;
• The main feature of rsync is incremental transmission, which transmits only the changed parts.

rsync principle

rsync is an efficient algorithm for synchronizing files under linux, which is used to synchronize updates to files and directories of two computers, and to properly use different blocks in the lookup file to reduce data transmission. The main feature of rsync is incremental transmission, which transfers only the changed parts.

Incremental synchronization algorithm

If we now need to synchronize two files and just want to transfer different parts, then we need to diff the files on both sides, but these two problems are not possible on two different machines. If we do diff, we need to transfer one file to another machine to do diff, but then we transfer the whole file, which is like us onlyThe intent of transferring different parts is opposite. So we want to find a way to make the files on both sides invisible, but we still know what the difference is. This is rsync's algorithm.

Install Deployment Server

• Install xineted service

[root@server1 ~]# yum install rsync.x86_64 -y
[root@server1 ~]# yum install xinetd -y		# Used to deploy rsync

• Modify xineted profile

[root@server1 ~]# vim /etc/xinetd.d/rsync
service rsync
{
disable = no
socket_type = stream
wait = no
user = root
server = /usr/bin/rsync
server_args = --daemon
port = 873
log_on_failure = USERID
}

• Modify rsync profile

[root@server1 ~]# vim /etc/rsyncd.conf
[test]
path = /test
uid = root
gid = root
max connections = 2
timeout = 300
read only = false
auth users = root
secrets file = /etc/rsync.passwd
strict modes = yes
use chroot = yes

• Prepare password file

[root@server1 ~]# vim /etc/rsync.passwd
root:200988
[root@server1 ~]# chmod 600 /etc/rsync.passwd

• Start Services

[root@server1 ~]# systemctl start xinetd.service 
[root@server1 ~]# ss -tanl | grep 873
LISTEN     0      64          :::873                     :::*

• prepare documents

[root@server1 ~]# mkdir /test
[root@server1 ~]# touch /test/123

Six working modes of rsync

• Install rsync on server2

[root@server2 ~]# yum install rsync.x86_64 -y

Parameters for the rysnc tool

rsync Related parameters

-v: display rsync Process details. Can be used"-vvvv"Get more details.
-P: Displays progress information for file transfer.(Actually"-P"="--partial --progress"，Among them"--progress"Is what shows progress information). 
-n --dry-run  : Only test the transport, not the actual transport. Often sum"-vvvv"Use together to view rsync How it works.
-a --archive  : Archive mode, which means to transfer files recursively and maintain file attributes. Equivalent to"-rtopgDl". 
-r --recursive: Recurse to the directory.
-t --times: keep mtime Attributes. It is strongly recommended that you add"-t"，Otherwise target file mtime Will be set to system time, causing the next update
          : Check Out mtime Different results in invalid incremental transmission.
-o --owner: keep owner attribute(Owner). 
-g --group: keep group attribute(Genus group). 
-p --perms: keep perms attribute(Permissions, excluding special permissions). 
-D        : yes"--device --specials"A combination of options that also copies device files and special files.
-l --links: If the file is a soft link file, copy the soft link itself instead of the object it points to.
-z        : Compression during transmission improves efficiency.
-R --relative: Use relative paths. This means sending the full path specified on the command line instead of the file name at the end of the path to the server, including their properties. See the example below for usage.
--size-only : The default algorithm is to check the file size and mtime Different files, use this option to check file size only.
-u --update : Only at source mtime Existing file than the target mtime Copy only when new. Note that this option is judged by the receiver and does not affect the deletion behavior.
-d --dirs   : Copy the directory itself in a non-recursive manner. By default, if the source is"dir1/file1"，No copy dir1 Directory, use this option to copy dir1 But do not copy file1. 
--max-size  : limit rsync Maximum file size to transfer. You can use a unit suffix or a decimal value(For example:"--max-size=1.5m")
--min-size  : limit rsync Minimum file size to transfer. This can be used to prohibit the transfer of small files or those garbage files.
--exclude   : Specify exclusion rules to exclude files that do not need to be transferred.
--delete    : with SRC Mainly, for DEST Synchronize. Delete more and supplement less."--delete"Executed on the receiving side, so it is in
            : exclude/include Rules will not be enforced until they become effective.
-b --backup : Make a backup of the files that already exist on the target. The backup file name is used by default"~"Make a suffix.
--backup-dir: Specify the path to save the backup files. When not specified, the default and backup files are saved in the same directory.
-e          : Specify the remote to use shell Program, default is ssh. 
--port      : Connect daemon Port number used, defaulting to port 873.
--password-file: daemon Password file in mode, which allows non-interactive reading of passwords. Note that this is not remote shell Authenticated password, but rsync Password for module authentication.
-W --whole-file: rsync Incremental transfer is no longer used, but full transfer. This option is more efficient than incremental transfer when network bandwidth is higher than disk bandwidth.
--existing  : Requires that only files that already exist on the target end be updated, and files that do not exist on the target end are not transferred. Note that when using relative paths, files that do not exist on the upper directory will not be transferred.
--ignore-existing: Requires that only files that do not exist on the target end be updated. and"--existing"This combination has special features, as illustrated below.
--remove-source-files: Requires deletion of files that have been successfully transferred from the source.
-h, --help display help information

• Mode 1, see what data sources are available on the server side

[root@server2 ~]# rsync --list-only root@192.168.174.10::		# Note Host Firewall
test  

• Mode 2, copy local files to local, start copy from local when neither src nor dest contains a colon (cp command)

[root@server2 ~]# mkdir /backup
[root@server2 ~]# ls
anaconda-ks.cfg  test1
[root@server2 ~]# rsync test1 /backup/
[root@server2 ~]# ls /backup/
test1

• Mode 3, copy local files to remote, start copy to remote when dest contains a colon (scp command)

[root@server2 ~]# rsync test1 root@192.168.174.10:/test
The authenticity of host '192.168.174.10 (192.168.174.10)' can't be established.
ECDSA key fingerprint is SHA256:l5y8+0D0KcBmCK2edW3ZSY7RyIDgBdlLOE10JAlVtKk.
ECDSA key fingerprint is MD5:60:21:02:4d:b8:ce:cd:bc:19:33:0d:58:d6:4c:bd:fe.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '192.168.174.10' (ECDSA) to the list of known hosts.
root@192.168.174.10's password: 
[root@server2 ~]# 

[root@server1 test]# ll /test/test1 
-rw-r--r-- 1 root root 49 9 September 23:42 /test/test1

• Mode 4, copy remote files locally, and start remote copies locally when the src contains a colon (scp command)

[root@server2 ~]# rsync root@192.168.174.10:/test/123 /backup/
root@192.168.80.128's password:
[root@server2 ~]# ll /backup/123
-rw-r--r-- 1 root root 0 3 February 2411:07 /backup/123

• Mode 5, Copy from remote server to local, enabled when src uses two colons followed by modules set on the server side

[root@server2 ~]# rsync -r root@192.168.174.10::test /backup
Password: 			# Password exists in server1/etc/rsync.passwd
[root@server2 ~]# ls /backup/
test1

• Mode 6, from local copy to remote server

[root@server2 ~]# touch test2
[root@server2 ~]# rsync -r test2 root@192.168.174.10::test
Password: 
[root@server1 test]# ls /test
test1  test2

Case: Timed Backup

Client Requirements

1. The directory where the client prepares to store the backup beforehand, with the following directory rules: /backup/host IP time
2. Client packages backups locally (system profile, application configuration, etc.) to copy to/backup/host IP time
3. The client finally pushes the backed up data to the backup server
4. Clients execute the script at 1 a.m. every day
5. Client servers keep data for the last 7 days locally to avoid wasting disk space

Server side requirements

1. The server deploys rsync to receive backup data pushed by the client
2. The server needs to check every day whether the data pushed by the client is complete
3. The server needs to notify the administrator of the results of the daily checks
4. The server only keeps 6 months of backup data and deletes all the rest

Client Preparation

• Create directory

[root@server2 ~]# mkdir /backup

• Install expect tool

[root@server2 ~]# yum install expect* -y

• Prepare expect script

[root@server2 ~]# vim rsync.exp
#!/usr/bin/expect
set mulu [lindex $argv 0]
set timeout 10
spawn rsync -avzr /backup/$mulu root@192.168.80.10::backup_server
expect Password
send "123456\r"
expect eof

• Prepare backup script

[root@server2 ~]# vim beifen.sh
#!/bin/bash

# Directory to prepare compressed files
mulu=`ip a | grep global|awk -F'[ /]+' '{print $3}'`_`date +%F`
echo $mulu
mkdir -pv /backup/$mulu &> /dev/null

# Package data to be sent
tar zcf /backup/$mulu/conf.tar.gz /etc/passwd /etc/vimrc &> /dev/null
touch /backup/$mulu

# send data
#rsync -avzr /backup/$mulu root@192.168.80.10::backup_server
expect rsync.exp $mulu

# Keep data within seven days
find /backup -mtime +7 -delete

Server side preparation

• Install rsync

[root@server1 ~]# yum install rsync.x86_64 -y

• Modify Profile

[root@server1 ~]# cat /etc/rsyncd.conf
[backup_server]
path = /backup
uid = root
gid = root
max connections = 2
timeout = 300
read only = false
auth users = root
secrets file = /etc/rsync.passwd
strict modes = yes
use chroot = yes

• Create directory

[root@server1 ~]# mkdir /backup -pv

• Prepare password file

[root@server1 ~]# cat /etc/rsync.passwd
root:123456
[root@server1 ~]# chmod 600 /etc/rsync.passwd

• Start rsync

[root@server1 ~]# systemctl start rsyncd.service
[root@server1 ~]# ss -tanl
State       Recv-Q Send-Q Local Address:Port               Peer Address:Port              
LISTEN      0      5                 *:873                           *:*                  
LISTEN      0      128               *:22                            *:*                  
LISTEN      0      100       127.0.0.1:25                            *:*                  
LISTEN      0      5                :::873                          :::*                  
LISTEN      0      128              :::22                           :::*                  
LISTEN      0      100             ::1:25                           :::*

• Validation Server

[root@server2 ~]# rsync --list-only root@192.168.174.10::
backup_server

• Simulate one month's data to verify results

[root@server2 ~]# for i in {1..30};do date -s 2021/08/$i; /root/beifen.sh ;
done
[root@server2 backup]# ll
 Total dosage 0
drwxr-xr-x 2 root root 25 8 February 2300:00 192.168.80.20_2021-08-23
drwxr-xr-x 2 root root 25 8 February 2400:00 192.168.80.20_2021-08-24
drwxr-xr-x 2 root root 25 8 February 2500:00 192.168.80.20_2021-08-25
drwxr-xr-x 2 root root 25 8 February 2600:00 192.168.80.20_2021-08-26
drwxr-xr-x 2 root root 25 8 February 2700:00 192.168.80.20_2021-08-27
drwxr-xr-x 2 root root 25 8 February 2800:00 192.168.80.20_2021-08-28
drwxr-xr-x 2 root root 25 8 February 2900:00 192.168.80.20_2021-08-29
drwxr-xr-x 2 root root 25 8 Month 30 00:00 192.168.80.20_2021-08-30
[root@server1 backup]# ll
 Total dosage 0
drwxr-xr-x 2 root root 25 8 January 100:00 192.168.80.20_2021-08-01
drwxr-xr-x 2 root root 25 8 February 200:00 192.168.80.20_2021-08-02
drwxr-xr-x 2 root root 25 8 March 3 2021 192.168.80.20_2021-08-03
drwxr-xr-x 2 root root 25 8 April 4 2021 192.168.80.20_2021-08-04
drwxr-xr-x 2 root root 25 8 May 2021 192.168.80.20_2021-08-05
drwxr-xr-x 2 root root 25 8 June 6 2021 192.168.80.20_2021-08-06
drwxr-xr-x 2 root root 25 8 July 2021 192.168.80.20_2021-08-07
.................................
drwxr-xr-x 2 root root 25 8 February 29, 2021 192.168.80.20_2021-08-29
drwxr-xr-x 2 root root 25 8 Month 30 2021 192.168.80.20_2021-08-30