Section 21 Rsync Data Synchronization Tool

Label (Space Separation): Linux Practical Teaching Notes - Chen Siqi

This teaching note is a summary of my study and work career, which is the first draft (there are many imperfections), the original work, allowed to be reproduced, when reproduced, please be sure to indicate the original source of the article, author information and this statement in the form of hyperlinks. Otherwise, legal liability will be pursued. http://www.cnblogs.com/chensiqiqi/

1.1 Introduction to Rsync

1.1.1 What is Rsync?

Introduction to 1.1.2 Rsync

• Rsync is called Remote synchronization in English. It can be seen from the name of the software that Rsync has the function of making data between local and remote hosts copy synchronous image quickly and backup remotely. This function is similar to the scp command of ssh band, but better than the function of scp command. scp is a full copy every time, while Rsync can copy incrementally. Of course, Rsync can also replicate data in full and incrementally between different partitions or directories of the local host, which is similar to the cp command, but also better than the cp command. cp is a full copy every time, while Rsync can replicate incrementally.

• An rsync is equivalent to scp, cp, rm, but better than each of their commands.

• When backing up data synchronously, by default, Rsync only synchronizes files or directories whose size or last modification time changes through its unique "quick check" algorithm. Of course, Rsync can also synchronize changes in attributes such as ownership and privileges, but it needs to specify appropriate parameters, and even can synchronize only the changed content of a file, so it can be real-life. Fast synchronization backup data.

• CentOS 5, rsync2.x comparison method, all files are compared once, and then synchronized.

• CentOS 6, Rsync 3. x comparison method, while comparing differences, while synchronizing the differences.

Characteristics of 1.3 Rsync

Rsync has the following characteristics:

• Support for copying special files such as linked files, devices, etc.
• Can we exclude (tar? find?) Specifies the function of file or directory synchronization, which is equivalent to the exclusion function of the packing command tar
• It can keep the privileges of original files or directories, time, hard and soft links, ownership, group and other attributes unchanged - p
• Incremental synchronization can be achieved, which can only synchronize the changed data, so the data transmission efficiency is very high (tar-N)
• You can use rcp, rsh, ssh and other ways to cooperate with the transfer of files (rsync itself does not encrypt data)
• You can transfer files and data (server and client) through socket (process mode)
• Support anonymous or authenticated (without system users) process mode transmission, which can facilitate and secure data backup and mirroring

1.1.4 Rsync Enterprise Work Scenario Description

1.1.4.1 Data Synchronization (Timing Task + Backup Data) between two servers, i.e. crond+rsync

Resume project experience:
Propose and implement the data backup solution of the whole network server 200x.03-200x.09

1) Aiming at the chaotic situation and leader of company's important data backup, the solution of backing up the whole network data is put forward.
2) Packing backup locally, then rsync and inotify application backup the whole network data to a fixed storage server, and then check the backup results by script on the storage server and alarm the administrator.
3) regularly back up the data of IDC computer room to the internal server of the company to prevent data loss caused by earthquake and fire in the computer room.

1.1.4.2 Real-time Synchronization (Solving the Single Point Problem of Storage Servers, etc.)

Using rsync and inotify function to synchronize real-time data, according to the change of directory on the storage server, the changed data can be synchronized to the backup server in real time through inotify or sersync combined with rsync command, and the data synchronization between two computers can also be realized through drbd scheme and dual-write scheme.

1.2 Way of Rsync Work

In order to facilitate students to learn, I have made a division from the actual use of functions. Generally speaking, Rsync uses three main ways to transmit data. They are:

• Data transmission between individual hosts locally (in this case similar to the function of the cp command)
• Transfer data by means of rcp, ssh and other channels (at this time similar to the function of scp command)
• Transfer data in the form of a socket (an important function of rsync itself)

The above several ways of working of rsync can be obtained through the help of man rsync or by looking at official manuals:

NAME
       rsync -- a fast, versatile, remote (and local) file-copying tool

SYNOPSIS
       Local:  rsync [OPTION...] SRC... [DEST]

       Access via remote shell:
         Pull: rsync [OPTION...] [USER@]HOST:SRC... [DEST]
         Push: rsync [OPTION...] SRC... [USER@]HOST:DEST

       Access via rsync daemon:
         Pull: rsync [OPTION...] [USER@]HOST::SRC... [DEST]
               rsync [OPTION...] rsync://[USER@]HOST[:PORT]/SRC... [DEST]
         Push: rsync [OPTION...] SRC... [USER@]HOST::DEST
               rsync [OPTION...] SRC... rsync://[USER@]HOST[:PORT]/DEST

       Usages with just one SRC arg and no DEST arg will list the source files
       instead of copying.

1.2.1 local-only mode

The syntax of Rsync local transport mode is as follows:
rsync [OPTION...] SRC...[DEST]
Grammatical Description:
1) Rsync is a synchronous command;
2) [OPTION] is the parameter option for synchronization
3) SRC is the source, i.e. the partition, file or directory to be copied.
4) [DEST] partition for purpose, file or directory, etc.

Direct local synchronization: equivalent to cp
rsync /etc/hosts /tmp/

Example 1-1 Example 1: Synchronize the hosts file of the system to the / opt directory

[root@chen ~]# rsync /etc/hosts /opt
[root@chen ~]# cat /opt/hosts 
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.14.200 mirrors.aliyum.com
192.168.197.133 www.test.com

Example 1-2 Example 2: Copy the opt directory to / mnt

[root@chen ~]# rsync -avz /opt /mnt #Equivalent to cp-ap/opt/mnt
sending incremental file list
opt/
opt/hosts
opt/rh/

sent 224 bytes  received 39 bytes  526.00 bytes/sec
total size is 221  speedup is 0.84
[root@chen ~]# ll /mnt
total 8
drwxr-xr-x. 3 root root 4096 Mar  5 19:54 opt
-rw-r--r--. 1 root root    5 Dec 25 11:19 test.txt

Delete function, equivalent to rm command

[root@chen ~]# mkdir /old
[root@chen ~]# rsync -avz --delete /old/ /tmp/
sending incremental file list
./
deleting pear/temp/
deleting pear/
deleting old/
deleting .ICE-unix/
deleting user_passwd

sent 29 bytes  received 15 bytes  88.00 bytes/sec
total size is 0  speedup is 0.00
[root@chen ~]# ll /tmp/
total 0

1.2.2 rsync Command Common Parameter Options Description:

- v,--verbose detailed mode output, transmission progress and other information
- z, - compress transmission is compressed to improve transmission efficiency, - compress-level=NUM can be compressed by level.
- a, - Archive archive mode, which means that files are transmitted recursively and all file attributes are preserved, equal to - rtopgD1 (letter 1)
==================================

1.2.3 Case: A DBA makes data synchronization, and its bandwidth is full, which makes it impossible for users to access the website.

rsync -avz dbfile 10.0.0.41:/backup #No bandwidth restrictions
rsync -avz --bwlimit=100 dbfile 172.16.1.41:／backup   #Limited bandwidth

1.2.4 Keep directories and file attributes in sync

Here - avzP is equivalent to - vzrtopgDIP (and more DI functions). The commonly used parameter options in production environment are - avzP or - vzrtopgP. If it is put into the script, it can also remove - VHE - P. Here -- programs can be replaced by - P.

Production parameters: - avz or - vzrtopg

1.2.5 Use rsync to backup and transfer data locally

Example 1: Testing local Rsync synchronization, rsync-avz/opt/tmp

[root@chen ~]# cd /opt #Entry directory
[root@chen opt]# mkdir chensiqi  #Create directory
[root@chen opt]# touch chensiqi/test.txt #create a file
[root@chen opt]# chmod -R 700 chensiqi #Recursive authorization 700
[root@chen opt]# ls -l #View directory permissions 700
total 4
drwx------. 2 root root 4096 Mar  5 22:18 chensiqi
[root@chen opt]# ls -l chensiqi/  #View File Permission 700
total 0
-rwx------. 1 root root 0 Mar  5 22:18 test.txt
[root@chen opt]# rsync -avz /opt/ /tmp/  #Perform local synchronization through rsync
sending incremental file list
./
chensiqi/
chensiqi/test.txt

sent 116 bytes  received 38 bytes  308.00 bytes/sec
total size is 0  speedup is 0.00
[root@chen opt]# tree /tmp  #Catalog files are fully synchronized
/tmp
`-- chensiqi
    `-- test.txt

1 directory, 1 file
[root@chen opt]# ll /tmp/  #Folder permissions 700, consistent
total 4
drwx------. 2 root root 4096 Mar  5 22:18 chensiqi
[root@chen opt]# ll /tmp/chensiqi/ #Document permissions 700 are consistent
total 0
-rwx------. 1 root root 0 Mar  5 22:18 test.txt

The example above demonstrates synchronizing files in the local / opt directory (excluding opt itself) to / tmp, where - avz is the parameter that keeps the relevant properties of the directory or file.

When data transmission is needed between different local directories, especially incremental transmission, this case command can replace commands such as cp to improve the efficiency of copy for you.

Example 2: Back up all content under / etc (including / etc directory itself) to / tmp directory

[root@chen ~]# rsync -avz /etc /tmp/
sending incremental file list
etc/
etc/.pwd.lock
etc/DIR_COLORS
etc/DIR_COLORS.256color
etc/DIR_COLORS.lightbgcolor
etc/adjtime
etc/aliases
etc/aliases.db
etc/anacrontab
//The following output is omitted...

[root@chen ~]# ll /tmp  #Synchronous completion
total 4 
drwxr-xr-x. 79 root root 4096 Mar  5 19:25 etc

The first time you run a command, it takes longer to scan and synchronize all files and directories. If backed up again, it will make a quick comparison and ignore the passed files, which will be faster, as follows:

[root@chen ~]# rsync -avz /etc /tmp/
sending incremental file list

sent 39813 bytes  received 196 bytes  80018.00 bytes/sec
total size is 27542875  speedup is 688.42

We can see that the synchronization is completed immediately, and there is very little data to be transmitted. Because rsync compares all files and directories, only files or directories with changes (content, modification time, etc.) are synchronized. If you change to the cp command, you will also re-execute the complete copy, wasting system resources and time.
Of course, local backup synchronization not only backs up directories, but also synchronizes individual files, devices and so on. I believe you are smart enough to think about it, so you don't have to spend much ink here.

1.3 Data transmission between different hosts via ssh channel

Example 1: Push: Push the current host content to a remote host

rsync -avzP -e 'ssh -p 22'/etc/ root@192.168.197.129:/tmp/

[root@chensiqi ~]# rsync -avzP -e 'ssh -p 22' /etc/  root@192.168.197.129:/tmp/ #Start synchronization
//Ignore the above....
yum/version-groups.conf
         444 100%    1.14kB/s    0:00:00 (xfer#985, to-check=6/1558)
yum/pluginconf.d/
yum/pluginconf.d/fastestmirror.conf
         279 100%    0.72kB/s    0:00:00 (xfer#986, to-check=2/1558)
yum/pluginconf.d/security.conf
          17 100%    0.04kB/s    0:00:00 (xfer#987, to-check=1/1558)
yum/protected.d/
yum/vars/
yum/vars/infra
           6 100%    0.02kB/s    0:00:00 (xfer#988, to-check=0/1558)

sent 9847758 bytes  received 20677 bytes  1518220.77 bytes/sec
total size is 27542879  speedup is 2.79

#Command specification
-e 'ssh -p 22' Represented by ssh The way to push through port 22, if you don't write the default port 22


[root@chensiqi ~]# ssh root@chensiqi2 "ls -l /tmp" #View synchronization results
root@chensiqi2's password: 
total 1668
drwxr-xr-x.  5 root root   4096 Dec 24 09:26 ConsoleKit
-rw-r--r--.  1 root root   4439 Apr 12  2016 DIR_COLORS
-rw-r--r--.  1 root root   5139 Apr 12  2016 DIR_COLORS.256color
-rw-r--r--.  1 root root   4113 Apr 12  2016 DIR_COLORS.lightbgcolor
drwxr-xr-x.  3 root root   4096 May 12  2016 NetworkManager
drwxr-xr-x.  4 root root   4096 Dec 24 09:26 X11
Some of the following are omitted...

#Instructions:
ssh root@chensiqi2 It means that ssh Connect in a way that passes through root Account to log in to host name chensiqi2 This host.
ssh root@chensiqi2 + Command, you can feedback the results of the command.
chensiqi2 It's one of the current hosts. hosts Insinuate/etc/hosts Add: IP Address host names correspond to mappings. When the host name is entered, the system automatically passes through hosts Resolve the correspondence IP Address. for example ssh root@chensiqi2  <==> ssh root@192.168.197.129

Example 2: Draw remote host content to the current host

rsync -avzP -e 'ssh -p 22' root@chensiqi2:/opt /tmp

Key grammatical descriptions:
1) - avz is equivalent to - vzrtopgDI, indicating that the file and directory attributes remain unchanged when synchronized.
2) - P shows the process of synchronization, which can be replaced by - programs.
3) - e'ssh-p 22'means that data is transmitted through SSH channels, which can be omitted.
4)root@chensiqi2 :/ opt Remote Host System User, Address, Path
5)/tmp Local Path

Practice demonstration: pull data to local / tmp directory from 192.168.197.129 / opt directory (including directory itself) by root user

[root@chensiqi ~]# rsync -avzP -e 'ssh -p 22' root@192.168.197.129:/opt /tmp/
root@192.168.197.129's password: 
receiving incremental file list
opt/
opt/chensiqi
           0 100%    0.00kB/s    0:00:00 (xfer#1, to-check=1/3)
opt/rh/

sent 38 bytes  received 122 bytes  29.09 bytes/sec
total size is 0  speedup is 0.00
[root@chensiqi ~]# ll /tmp
total 4
drwxr-xr-x. 3 root root 4096 Mar  6  2017 opt

You can also remove - e'ssh-p 22'(default port 22)

[root@chensiqi ~]# rsync -avzP root@192.168.197.129:/opt /tmp/
root@192.168.197.129's password: 
receiving incremental file list

sent 13 bytes  received 80 bytes  37.20 bytes/sec
total size is 0  speedup is 0.00

You can also use the mapped host name: (/etc/hosts)

[root@chensiqi ~]# tail -1 /etc/hosts
192.168.197.129 chensiqi2
[root@chensiqi ~]# rsync -avzP root@chensiqi2:/opt /tmp/
root@chensiqi2's password: 
Permission denied, please try again.
root@chensiqi2's password: 
receiving incremental file list
opt/
opt/chensiqi
           0 100%    0.00kB/s    0:00:00 (xfer#1, to-check=1/3)
opt/rh/

sent 38 bytes  received 122 bytes  21.33 bytes/sec
total size is 0  speedup is 0.00

1.4 Data transfer in the form of a daemon (socket)

1.4.1 Pre-deployment preparations:

1.4.2 Deployment Environment

Considering that many students have no actual production environment, this paper uses Linux host in VMWARE virtual machine environment to carry out experiments.
There is almost no difference between a real server deployment in a production environment.

Operating system:

[root@chensiqi ~]# cat /etc/redhat-release 
CentOS release 6.8 (Final)

Kernel version:

[root@chensiqi ~]# uname -r
2.6.32-642.el6.x86_64

Host network parameter settings:
| Host Name | Network Card eth0 | Usage | Code|
|--|--|--|--|--|
| chensiqi|192.168.197.133|rsync client | B-Server|
| chensiqi2|192.168.197.129|rsync server | A-Server|

1.4.3 Specific requirements

It is required to deploy Rsync services on A-Server in the way of Rsync daemon, so that all client hosts of Rsync nodes can backup local data to A-Server through rsync. The client of this example only takes B-Server and C-Server as examples.

1.5 Start deploying Rsync service-Rsync server-side A-Server operation process:

1.5.1 Configure rsyncd.conf

First, confirm whether the software is installed:

[root@chensiqi2 ~]# rpm -qa rsync
rsync-3.0.6-12.el6.x86_64

Then create the rsyncd.conf file and add the following (the file does not exist by default)

[root@chensiqi2 backup]# cat /etc/rsyncd.conf
#rsync_config_____start
#created by chensiqi 13:40 2017-3-6
##blog:http://www.cnblogs.com/chensiqiqi/
##rsyncd.conf start##

# user
uid = rsync
# group
gid = rsync
# Program Security Settings
use chroot = no
# Number of client connections
max connections = 200
# timeout
timeout = 300
# Process document location
pid file = /var/run/rsyncd.pid
# Process lock
lock file = /var/run/rsync.lock
# Location of log files
log file = /var/log/rsyncd.log
##########################################
[backup]
# Use directory
path = /backup/
# Ignore mistakes
ignore errors
# Readable and Writable (true or false)
read only = false
# Prevent remote lists (don't let the server see what's on the server remotely)
list=false
# Allow IP
hosts allow = 192.168.197.0/24
# Prohibit IP
hosts deny = 0.0.0.0/32
# Virtual user
auth users = rsync_backup
# Files that store users and passwords
secrets file = /etc/rsync.password

##rsync_config______end##

1.5.2 Creating Shared Directory and Adding rsync Program Users

[root@chensiqi2 ~]# useradd -M -s /sbin/nologin rsync  #Create rsync users
[root@chensiqi2 ~]# cat /etc/passwd | grep rsync  
rsync:x:500:500::/home/rsync:/sbin/nologin
[root@chensiqi2 ~]# cat /etc/group | grep rsync
rsync:x:500:
[root@chensiqi2 ~]# mkdir /backup #Create shared directories

1.5.3 Startup Service: rsync --daemon

[root@chensiqi2 ~]# rsync --daemon
[root@chensiqi2 ~]# netstat -antup | grep rsync
tcp        0      0 0.0.0.0:873                 0.0.0.0:*                   LISTEN      5163/rsync          
tcp        0      0 :::873                      :::*                        LISTEN      5163/rsync  

1.5.4 Change the / backup folder on A-Server to the master rsync

[root@chensiqi2 ~]# chown -R rsync /backup
[root@chensiqi2 ~]# ls -ld /backup
drwxr-xr-x. 2 rsync root 4096 3 Month 622:19 /backup

1.5.5 Create rsync Virtual Account Name and Password

[root@chensiqi2 ~]# echo "rsync_backup:123456" >/etc/rsync.password
[root@chensiqi2 ~]# cat /etc/rsync.password
rsync_backup:123456

1.5.6 Set the permission of account password file to 600 (must fail otherwise)

[root@chensiqi2 ~]# chmod 600 /etc/rsync.password 
[root@chensiqi2 ~]# ll /etc/rsync.password 
-rw-------. 1 root root 20 3 Month 622:27 /etc/rsync.password

1.5.7 Add boot start

[root@chensiqi2 ~]# echo "rsync --daemon" >> /etc/rc.local
[root@chensiqi2 ~]# tail -1 /etc/rc.local 
rsync --daemon

Be careful:
Of course, you can also use the chkconfig rsync on command, but you have to write scripts suitable for chkconfig operation.

How to restart rsync service?
pkill rsync # Close rsync service
rsync --daemon # Start the rsync service

So far, rsync server A-server has been configured.

1.6 Start deploying Rsync service--Rsync client B-Server

1.6.1 All you need to do is create a password file

[root@chensiqi ~]# rpm -qa rsync
rsync-3.0.6-12.el6.x86_64
[root@chensiqi ~]# echo "123456" > /etc/rsync.password

1.6.2 Set the password file permission to 600 (must fail otherwise)

[root@chensiqi ~]# chmod 600 /etc/rsync.password 
[root@chensiqi ~]# ls -ld /etc/rsync.password 
-rw-------. 1 root root 7 Mar  6 01:42 /etc/rsync.password

So far, rsync client B-Server has been configured.

1.6.5 Rsync Synchronization Test

1.6.5.1 Push Test 1: Push the content of client specified directory to server specified directory of rsync.

Test commands:

rsync -avz /backup/ rsync_backup@192.168.197.129::backup --password-file=/etc/rsync.password

Instructions:
- avz: keep robust attributes unchanged, - v displays synchronization information and - P displays specific synchronization process
 / backup/: The directory where the content to be pushed is located
 rsync_backup: Synchronized username for server-side rsync services (non-Linux users)
192.168.197.129:rsync server IP address
 backup: module name in rsync server configuration file
 password-file=/etc/rsync.password: Password-free operation, specify the location of the password file, if not written, will require users to enter passwords interactively. (If you want to hang on to a scheduled task, you have to be non-interactive)

Demonstration:

[root@chensiqi backup]# ls
opt.tar.gz
[root@chensiqi backup]# rsync -avzP /backup/ rsync_backup@192.168.197.129::backup --password-file=/etc/rsync.password #Synchronous test 

sending incremental file list
./
opt.tar.gz
         166 100%    0.00kB/s    0:00:00 (xfer#1, to-check=0/2)

sent 258 bytes  received 30 bytes  576.00 bytes/sec
total size is 166  speedup is 0.58

[root@chensiqi backup]# ssh root@chensiqi2 "ls -l /backup" #View synchronization results
root@chensiqi2's password: 
total 4
-rw-r--r--. 1 rsync rsync 166 Mar  6 21:02 opt.tar.gz

1.6.5.2 Push Test 2: Push any directory of client to the specified directory of rsync server

Test commands:

rsync -avzP /tmp/ rsync_backup@192.168.197.129::backup --password-file=/etc/rsync.password

Demonstration process:

[root@chensiqi backup]# rsync -avzP /tmp/ rsync_backup@192.168.197.129::backup --password-file=/etc/rsync.password 
sending incremental file list
./
opt.tar.gz2017-03-06
         162 100%    0.00kB/s    0:00:00 (xfer#1, to-check=5/8)
backup/
opt/
opt/chensiqi
           0 100%    0.00kB/s    0:00:00 (xfer#2, to-check=1/8)
opt/rh/

sent 441 bytes  received 62 bytes  1006.00 bytes/sec
total size is 162  speedup is 0.32
[root@chensiqi backup]# ssh root@chensiqi2 "ls /backup"   #Take a look at the results
root@chensiqi2's password: 
backup
opt
opt.tar.gz2017-03-06

1.6.5.3 Pull-out Test 1: Synchronize all contents of specified directories on rsync server side to client side

Test commands:

rsync -avzP rsync_backup@192.168.197.129::backup /backup/ --password-file=/etc/rsync.password

Instructions:
Compared with push, only two directories have changed their location.

Demonstration process:

[root@chensiqi backup]# ls
[root@chensiqi backup]# rsync -avzP rsync_backup@192.168.197.129::backup /backup/ --password-file=/etc/rsync.password 
receiving incremental file list
./
a
           0 100%    0.00kB/s    0:00:00 (xfer#1, to-check=1/3)
opt.tar.gz
         166 100%  162.11kB/s    0:00:00 (xfer#2, to-check=0/3)

sent 105 bytes  received 389 bytes  988.00 bytes/sec
total size is 166  speedup is 0.34
[root@chensiqi backup]# ls
a  opt.tar.gz

1.6.5.4 Pull-out Test 2: Synchronize the specified contents in the specified directory of rsync server side to the client side

Test commands:

rsync -avzP rsync_backup@192.168.197.129::backup/opt.tar.gz /backup/ --password-file=/etc/rsync.password

Demonstration process:

[root@chensiqi backup]# ls
[root@chensiqi backup]# rsync -avzP rsync_backup@192.168.197.129::backup/opt.tar.gz /backup/ --password-file=/etc/rsync.password 
receiving incremental file list
opt.tar.gz
         166 100%  162.11kB/s    0:00:00 (xfer#1, to-check=0/1)

sent 83 bytes  received 328 bytes  822.00 bytes/sec
total size is 166  speedup is 0.40
[root@chensiqi backup]# ls
opt.tar.gz

1.6.5.5 Pull-out Test 3: After excluding all contents of a directory or file specified by rsync server side, synchronize to client side

Environmental preparation
We create the following file structure under the directory specified on the rsync server side

[root@chensiqi2 backup]# ls
a  b  c  chen  d  e
[root@chensiqi2 backup]# ls chen
1  2  3  4  5

Explain:
a, b, c, d, e are files and chen are directories. There are 1,2,3,4,5 files in the directory.

Method 1: Exclusion through command line

Test commands:

rsync -avz --exclude=a --exclude=chen/3 --exclude=chen/4 rsync_backup@192.168.197.129::backup /backup/ --password-file=/etc/rsync.password 

Instructions:
- exlude = filename: excluded file

Demonstration process:

[root@chensiqi backup]# rsync -avz --exclude=a --exclude=chen/3 --exclude=chen/4 rsync_backup@192.168.197.129::backup /backup/ --password-file=/etc/rsync.password 
receiving incremental file list
./
.pwd.lock
b
c
d
e
.ICE-unix/
chen/
chen/1
chen/2
chen/5

sent 258 bytes  received 558 bytes  1632.00 bytes/sec
total size is 0  speedup is 0.00

[root@chensiqi backup]# ls
b  c  chen  d  e
[root@chensiqi backup]# ls chen
1  2  5

Method 2: Exclusion through list file

Create exclusion list file

[root@chensiqi backup]# cat /root/exclude.txt 
1
3
5
b
e

Test commands:

rsync -avz --exclude-from=/root/exclude.txt rsync_backup@192.168.197.129::backup /backup/ --password-file=/etc/rsync.password

Instructions:
-- exclude-from = absolute path to the file: refer to an exclusion list, where you only need to enter the name of the excluded file

Demonstration process:

[root@chensiqi backup]# cat /root/exclude.txt 
1
3
5
b
e
[root@chensiqi backup]# rsync -avz --exclude-from=/root/exclude.txt rsync_backup@192.168.197.129::backup /backup/ --password-file=/etc/rsync.password 
receiving incremental file list
./
a
c
d
chen/
chen/2
chen/4

sent 202 bytes  received 434 bytes  1272.00 bytes/sec
total size is 0  speedup is 0.00
[root@chensiqi backup]# ls
a  c  chen  d
[root@chensiqi backup]# ls chen
2  4

1.6.5.6 rsync synchronous pull-out test: keep rsync client specified directory content consistent with rsync server shared directory content

1) Always consistent with rsync server directory content

Test commands:

rsync -avz --delete rsync_backup@192.168.197.129::backup /backup/ --password-file=/etc/rsync.password 

Instructions:
- delete: Represents synchronized addition, deletion and alteration (file content changes, will also synchronize)

Demonstration process:

[root@chensiqi backup]# rsync -avz --delete rsync_backup@192.168.197.129::backup /backup/ --password-file=/etc/rsync.password     #Make the first synchronization
receiving incremental file list
./
a
b
c
d
e
chen/
chen/1
chen/2
chen/3
chen/4
chen/5

sent 262 bytes  received 663 bytes  1850.00 bytes/sec
total size is 8  speedup is 0.01
[root@chensiqi backup]# ls   #View synchronized files
a  b  c  chen  d  e
[root@chensiqi backup]# ssh root@chensiqi2 "rm -rf /backup/a"  #Remotely delete files a in the shared directory of the Rsync server
root@chensiqi2's password: 
[root@chensiqi backup]# rsync -avz --delete rsync_backup@192.168.197.129::backup /backup/ --password-file=/etc/rsync.password      #Second synchronization
receiving incremental file list
deleting a   #You can see a delete synchronization in the synchronization process
./

sent 69 bytes  received 278 bytes  694.00 bytes/sec
total size is 0  speedup is 0.00
[root@chensiqi backup]# ls  #Looking at the synchronization results, file a disappears.
b  c  chen  d  e
[root@chensiqi backup]# ssh root@chensiqi2 "echo 1111 >/backup/chensiqi" #Create a content file chensiqi remotely under the rsync server-side shared directory
root@chensiqi2's password: 
[root@chensiqi backup]# rsync -avz --delete rsync_backup@192.168.197.129::backup /backup/ --password-file=/etc/rsync.password   #Third synchronization
receiving incremental file list
./
chensiqi    #Added chensiqi file

sent 88 bytes  received 337 bytes  850.00 bytes/sec
total size is 5  speedup is 0.01
[root@chensiqi backup]# cat chensiqi    #View Synchronized File Contents
1111

[root@chensiqi backup]# ssh root@chensiqi2 "echo 222 >>/backup/chensiqi" #Added a line of content to the chensiqi file in the shared directory on rsync server side remotely.
root@chensiqi2's password: 
[root@chensiqi backup]# rsync -avz --delete rsync_backup@192.168.197.129::backup /backup/ --password-file=/etc/rsync.password      #Fourth synchronization
receiving incremental file list
chensiqi  #The modified files are also synchronized

sent 91 bytes  received 338 bytes  858.00 bytes/sec
total size is 9  speedup is 0.02
[root@chensiqi backup]# cat chensiqi  #View Synchronized File Content
1111
222

2) After excluding a file, synchronize with the server

Test commands:

rsync -avz --delete --exclude=c rsync_backup@192.168.197.129::backup /backup/ --password-file=/etc/rsync.password 

Instructions:
--exclude=c: Synchronization does not take into account a file named C

Demonstration process:

[root@chensiqi backup]# ls   #View the contents of the directory
b  c  chen  chensiqi  d  e
[root@chensiqi backup]# rsync -avz --delete --exclude=c rsync_backup@192.168.197.129::backup /backup/ --password-file=/etc/rsync.password   #First Synchronization
receiving incremental file list

sent 73 bytes  received 283 bytes  237.33 bytes/sec
total size is 9  speedup is 0.03
[root@chensiqi backup]# ssh root@chensiqi2 "rm -rf /backup/c"  #Remotely delete server-side c files
root@chensiqi2's password: 
[root@chensiqi backup]# rsync -avz --delete --exclude=c rsync_backup@192.168.197.129::backup /backup/ --password-file=/etc/rsync.password   #Second Synchronization
receiving incremental file list
./          #No synchronization to anything

sent 76 bytes  received 286 bytes  241.33 bytes/sec
total size is 9  speedup is 0.02
[root@chensiqi backup]# ssh root@chensiqi2 "touch /backup/c"   #Remote creation of c file
root@chensiqi2's password: 
[root@chensiqi backup]# rsync -avz --delete --exclude=c rsync_backup@192.168.197.129::backup /backup/ --password-file=/etc/rsync.password     #Third Synchronization
receiving incremental file list
./     #Or hasn't synchronized to anything yet

sent 76 bytes  received 286 bytes  241.33 bytes/sec
total size is 9  speedup is 0.02
[root@chensiqi backup]# ssh root@chensiqi2 "echo 111 >>/backup/c"  #Remotely modify c file
root@chensiqi2's password: 
[root@chensiqi backup]# rsync -avz --delete --exclude=c rsync_backup@192.168.197.129::backup /backup/ --password-file=/etc/rsync.password   #Fourth Synchronization
receiving incremental file list
                                             #Still not synchronized to anything
sent 73 bytes  received 283 bytes  712.00 bytes/sec
total size is 9  speedup is 0.03

1.6.5.7 rsync synchronous push test: Let Rsync server share directories consistently with Rsync client specified directory content.

1) consistency with rsync client directory content

Test commands:

rsync -avz --delete /backup/ rsync_backup@192.168.197.129::backup --password-file=/etc/rsync.password 

Instructions:
- delete: Represents synchronized addition, deletion and alteration (file content changes, will also synchronize)
Compared with synchronous pull-out, only the client directory is placed in front of the server.

Demonstration process:

[root@chensiqi backup]# ls
a  b  c  chen  d  e
[root@chensiqi backup]# rsync -avz --delete /backup/ rsync_backup@192.168.197.129::backup --password-file=/etc/rsync.password #First Synchronization
sending incremental file list
./
a
b
c
d
e
chen/
chen/1
chen/2
chen/3
chen/4
chen/5

sent 594 bytes  received 206 bytes  533.33 bytes/sec
total size is 0  speedup is 0.00

[root@chensiqi backup]# rm a   #Client Delete File a
rm: remove regular empty file `a'? y
[root@chensiqi backup]# rsync -avz --delete /backup/ rsync_backup@192.168.197.129::backup --password-file=/etc/rsync.password  #Second Synchronization
sending incremental file list
./
deleting a  #Synchronized the process of deleting file a

sent 225 bytes  received 13 bytes  476.00 bytes/sec
total size is 0  speedup is 0.00
[root@chensiqi backup]# touch a   #Create file a
[root@chensiqi backup]# rsync -avz --delete /backup/ rsync_backup@192.168.197.129::backup --password-file=/etc/rsync.password #Third Synchronization
sending incremental file list
./
a  #Push a file to the server

sent 271 bytes  received 32 bytes  606.00 bytes/sec
total size is 0  speedup is 0.00

[root@chensiqi backup]# echo 1111 >>a  #Modification of file a
[root@chensiqi backup]# cat a
1111
[root@chensiqi backup]# rsync -avz --delete /backup/ rsync_backup@192.168.197.129::backup --password-file=/etc/rsync.password #Fourth Synchronization
sending incremental file list
a      #Synchronized the modified file a to the server side

sent 277 bytes  received 29 bytes  612.00 bytes/sec
total size is 5  speedup is 0.02

[root@chensiqi backup]# ssh root@chensiqi2 "cat /backup/a"  #View the contents of file a in the server-side shared directory remotely
root@chensiqi2's password: 
1111

2)--exclude = file name. After excluding a file, synchronize with the server

It's exactly the same as the synchronous pull-out exclusion. It's just that the catalogue changes its location. It doesn't cost too much space here. Students test it by themselves.

1.6.6 Risk Tips for Rsync Enterprise Applications

rsync pushes enterprise work scenarios:
1) Backup -- delete risk
What is local, what is remote, there is no local remote also need to be deleted. The directory data on the server side may be lost.

rsync pulls out enterprise work scenarios:
1) Code release, download. - delete risk
What's far away, what's local (client) and what's not far away should be deleted. Local directory data may be lost.

1.6.7 rsync undifferentiated synchronous production scenario application

1.7 Advantages and disadvantages of Rsync

1.7.1 rsync advantages:

1. Incremental backup, support for socket (daemon), centralized backup (push-pull support, are client-side reference).
2. Remote SHELL channel mode can also be encrypted (SSH) transmission, socket (daemon) needs to be encrypted transmission, can use vpn services or ipsec services.

1.7.2 rsync shortcomings:

1. When a large number of small files are synchronized, the comparison time is longer. Sometimes, in the process of synchronization, the rsync process may stop and die.
2. Synchronize large files. Large files like 10G sometimes cause problems and interruptions. Before incomplete synchronization, it is a hidden file, which can be transmitted by parameters such as -- partial.
3. One-time remote copy can use scp, a large number of small files need to be packaged and copied. (important)

1.8 Necessary Thoughts for Error Elimination

• Proficiency in deployment process steps
• rsync Principle Understanding
• Learn to read logs, rsync command line output, log file / var/log/rsyncd.log

1.9 Rsync daemon service transmission data debugging ideas:

1.9.1 Rsync Server Error Removal

1. Check whether the rsync service profile path is correct and the correct default path is: / etc/rsyncd.conf
2. Check the configuration file for host allow,host deny, whether the IP segment allowed is the IP segment allowed for client access?
3. Check whether the path in the path parameter in the configuration file exists and whether the permissions are correct (normally the owner and group corresponding to the UID parameter in the configuration file)
4. Check whether the Rsync service is started. The view command is ps-ef | grep rsync. Does the port exist netstat-antup | grep 873
5. Check whether the iptables firewall and selinux are enabled to allow rsync services to pass, or consider closing them.
6. Check whether the server rsync configuration password file is 600 permissions, the password file format is correct, the correct format is: username: password, file path and secrect files parameters in the configuration file correspond.
7. If you are pushing data, check to see if the user in the configuration rsyncd.conf file has readable and writable access to the directory under the module.

1.9.2 Rsync Client Error-shooting Thought

1. Check whether the client rsync configuration password file is 600 privileges, password file format is correct, note: only need a password, and the server password is consistent.
2. Connect the rsync server ip address 873 port with telnet to check whether the service is started (whether the server firewall is blocked or not) telnet 192.168.197.129 873
3. When the client executes the command: rsync-avzP rsync_backup@192.168.197.129:: backup/backup/password-file=/etc/rsync.password
4. The details of this command should be clearly remembered, especially the double colon at 192.168.197.129::backup and the subsequent backup as the module name.

2 [Rsync Project Practice] Case Model of Data Production Architecture Scheme for Backup Network-wide Server (Mandatory)

[Business case]
There is a Web server in a company. The data is very important, but if the hard disk is broken, the data will be lost. Now the leader asks you to make a periodic and regular backup of the data on other machines. The requirements are as follows:

Specific requirements are as follows:
1) The backup directory of NFS server nfs01 and backup server backup must be / backup
2) The NFS server site directory is assumed to be (/var/www/html)
3) NFS servers only keep local backups for 7 days.
4) Check the backup results on the backup server and send them to the administrator's mailbox every day.
5) The data on the backup server is reserved every Saturday, while other backups are reserved for only 180 days.

Appendix 1: Description of common parameters of rsyncd.conf configuration file:

Appendix 2: Developing rsync service startup scripts

#!/bin/bash
#author:Mr.chen
# chkconfig:35 13 91
# description:This is Rsync service management shell script


# Source function library
. /etc/rc.d/init.d/functions


start(){
    rsync --daemon
    if [ $? -eq 0 -a `ps -ef|grep -v grep|grep rsync|wc -l` -gt 0 ];then
        action "Starting Rsync:" /bin/true
        sleep 1
    else
        action "Starting Rsync:" /bin/false
        sleep 1
    fi
}

stop(){
    pkill rsync;sleep 1;pkill rsync
    if [ `ps -ef|grep -v grep|grep "rsync --daemon"|wc -l` lt 1 ];then
        action "Stopping Rsync: `ps -ef|grep -v grep|grep rsync|wc -l`" /bin/true
        sleep 1
    else
        action "Stopping Rsync:`ps -ef|grep -v grep | grep "rsync --daemon"|wc -l`" /bin/true
        sleep 1
    fi
}

case "$1" in
    start)
        start;
        ;;
    stop)
        stop;
        ;;
    restart|reload)
        $0 stop;
        $0 start;
        ;;
    *)
        echo $"Usage: $0 {start|stop|restart|reload}"
        ;;
esac

Save it as rsyncd and place it in / etc/init.d/rsyncd

[root@chensiqi2 ~]# cp rsync /etc/init.d/rsyncd
[root@chensiqi2 ~]# chmod +x /etc/init.d/rsyncd 
[root@chensiqi2 ~]# /etc/init.d/rsyncd stop
//Terminated
[root@chensiqi2 ~]# /etc/init.d/rsyncd start
Starting Rsync:                                            [Sure?]