Section 23 Intify Event Monitoring Tool

Label (Space Separation): Linux Practical Teaching Notes - Chen Siqi

This teaching note is a summary of my study and work career, which is the first draft (there are many imperfections), the original work, allowed to be reproduced, when reproduced, please be sure to indicate the original source of the article, author information and this statement in the form of hyperlinks. Otherwise, legal liability will be pursued. http://www.cnblogs.com/chensiqiqi/

Chapter 1, the construction of NFS storage server and backup backup server.

Detailed knowledge and construction, please pay attention to:

http://www.cnblogs.com/chensiqiqi/p/6514315.html Rsync Data Synchronization Tool

http://www.cnblogs.com/chensiqiqi/p/6530859.html Enterprise NFS Network File Sharing Service

http://www.cnblogs.com/chensiqiqi/p/6531003.html [Rsync Project Practice] Backup the whole network server data

Chapter 2: Origin of rsync + inotify combination

Chapter 3 Introduction to inotify

In particular:

The following inotify configuration is a configuration process based on rsync services.

Chapter 4 Intify Implementation Preparations

The main premise is that rsync daemon service is successfully configured, and then inotify service can be configured by pushing and pulling data from rsync client.

Chapter 5 Starts Installation

Default yum source:
base + extras + updates

Extended yum source:
epel
1. Netease 163 Source
2. Aliyun epel source

Before installing inotify-tools, please confirm that your linux kernel has reached 2.6.13, and turn on the CONFIG_INOTIFY option at compile time. It can also be checked by the following commands.

5.1 See if the current system supports inotify

[root@backup ~]# uname -r
2.6.32-642.el6.x86_64
[root@backup ~]# ls -l /proc/sys/fs/inotify
//Total dosage 0
-rw-r--r-- 1 root root 0 3 Month 1105:01 max_queued_events
-rw-r--r-- 1 root root 0 3 Month 1105:01 max_user_instances
-rw-r--r-- 1 root root 0 3 Month 1105:01 max_user_watches

#Display three documentation support

Key parameters:

There are three files in the / proc/sys/fs/inotify directory, which limit the inotify mechanism.
max_user_watches: The number of files (single process) that can be monitored by setting the inotifywait or inotifywatch commands
 max_user_instances: Sets the number of processes that each user can run with the inotifywait or inotifywatch command.
max_queued_events: Sets the number of events that the inotify instance event queue can hold.

5.2 Yum installs inotify-tools:

#wget -O /etc/yum.repos.d/epel.repo http://mirrors.aliyun.com/repo/epel-6.repo
yum -y install inotify-tools
rpm -qa inotify-tools

5.3 Details of common parameters of the inotifywait command

inotifywait --help

[root@backup ~]# inotifywait --help
inotifywait 3.14
Wait for a particular event on a file or set of files.
Usage: inotifywait [ options ] file1 [ file2 ] [ file3 ] [ ... ]
Options:
    -h|--help       Show this help text.
    @<file>         Exclude the specified file from being watched.
    --exclude <pattern>
                    Exclude all events on files matching the
                    extended regular expression <pattern>.
    --excludei <pattern>
                    Like --exclude but case insensitive.
    -m|--monitor    Keep listening for events forever.  Without
                    this option, inotifywait will exit after one
                    event is received.
    -d|--daemon     Same as --monitor, except run in the background
                    logging events to a file specified by --outfile.
                    Implies --syslog.
    -r|--recursive  Watch directories recursively.
    --fromfile <file>
                    Read files to watch from <file> or `-' for stdin.
    -o|--outfile <file>
                    Print events to <file> rather than stdout.
    -s|--syslog     Send errors to syslog rather than stderr.
    -q|--quiet      Print less (only print events).
    -qq             Print nothing (not even events).
    --format <fmt>  Print using a specified printf-like format
                    string; read the man page for more details.
    --timefmt <fmt> strftime-compatible format string for use with
                    %T in --format string.
    -c|--csv        Print events in CSV format.
    -t|--timeout <seconds>
                    When listening for a single event, time out after
                    waiting for an event for <seconds> seconds.
                    If <seconds> is 0, inotifywait will never time out.
    -e|--event <event1> [ -e|--event <event2> ... ]
        Listen for specific event(s).  If omitted, all events are 
        listened for.

Exit status:
    0  -  An event you asked to watch for was received.
    1  -  An event you did not ask to watch for was received
          (usually delete_self or unmount), or some error occurred.
    2  -  The --timeout option was given and no events occurred
          in the specified interval of time.

Events:
    access      file or directory contents were read
    modify      file or directory contents were written
    attrib      file or directory attributes changed
    close_write file or directory closed, after being opened in
                writeable mode
    close_nowrite   file or directory closed, after being opened in
                read-only mode
    close       file or directory closed, regardless of read/write mode
    open        file or directory opened
    moved_to    file or directory moved to watched directory
    moved_from  file or directory moved from watched directory
    move        file or directory moved to or from watched directory
    **create**      file or directory created within watched directory
    **delete**      file or directory deleted within watched directory
    delete_self file or directory was deleted
    unmount     file system containing file or directory unmounted

Here's a list to explain the meaning of each parameter in detail

- e:--event's event Meanings

5.4 Manual Testing and Monitoring Events

Open two windows

5.4.1 Test create

Enter the following in the first window:
[root@backup ~]# ls /backup
[root@backup ~]# inotifywait -mrq --timefmt '%y %m %d %H %M' --format '%T %w%f' -e create /backup

In the second window: enter the following
[root@backup ~]# cd /backup
[root@backup backup]# touch chensiqi

Back to the first window, the following appears:
17 03 11 07 19 /backup/chensiqi

# Command description
 inotifywait: ionotify's command tool
 - M R q:-q only input short message-r, recursively monitor the entire directory including subdirectory-m for continuous monitoring
 - timefmt specifies the time format of the output 
Format: Specifies the format of the output information
 - e create: Establish the time type of monitoring and monitor the creation event.

5.4.2 Test delte

The first window enters the following information:
[root@backup ~]# inotifywait -mrq --timefmt '%y %m %d %H %M' --format '%T %w%f' -e delete /backup

The second window enters the following information:
[root@backup backup]# rm -rf chensiqi

At this point, the first window will display the following information:
17 03 11 07 29 /backup/chensiqi

# Instructions:
- e delete: Specifies the type of event to listen on. Listen for delete events

5.4.3 Test close_write

The first window enters the following information:
inotifywait -mrq --timefmt '%y %m %d %H %M' --format '%T %w%f' -e close_write /backup
 The second window enters the following information:
[root@backup backup]# touch close_write.log
[root@backup backup]# echo 111 >> close_write.log 
[root@backup backup]# rm -f close_write.log 
At this point, the first window will display the following information:
17 03 11 07 38 /backup/close_write.log
17 03 11 07 39 /backup/close_write.log

# Instructions:
- e close_write: Specifies the listening type. Close the listening file write mode.

5.4.4 Test move_to

The first window enters the following information:
[root@backup ~]# inotifywait -mrq --timefmt '%y %m %d %H %M' --format '%T %w%f' -e moved_to /backup  
The second window enters the following information:

At this point, the first window will display the following information:
[root@backup backup]# touch chensiqi
[root@backup backup]# mv chensiqi chen
[root@backup backup]# mkdir ddddd
[root@backup backup]# mv chen ddddd/

5.5 Writing inotify real-time monitoring script

#!/bin/bash

backup_Server=172.16.1.41


/usr/bin/inotifywait -mrq --format '%w%f' -e create,close_write,delete /data | while read line
do
    cd /data
    rsync -az ./ --delete rsync_backup@$backup_Server::nfsbackup --password-file=/etc/rsync.password
done

Tips:

• The reason that the script above is inefficient is that whenever the directory changes, everything in my entire directory will be pushed once. Therefore, we can make the following changes to improve efficiency

#!/bin/bash

Path=/data
backup_Server=172.16.1.41


/usr/bin/inotifywait -mrq --format '%w%f' -e create,close_write,delete /data  | while read line  
do
    if [ -f $line ];then
        rsync -az $line --delete rsync_backup@$backup_Server::nfsbackup --password-file=/etc/rsync.password       
    else
        cd $Path &&\
        rsync -az ./ --delete rsync_backup@$backup_Server::nfsbackup --password-file=/etc/rsync.password
    fi

done

The script can be added to boot up:
echo "/bin/sh /server/scripts/inotify.sh &" >> /etc/rc.local
Tips:
One - Represents running this command from the background.

5.6 Key Parameter Adjustment

Actual adjustment:

[root@nfs01 data]# cat /proc/sys/fs/inotify/max_
max_queued_events   max_user_instances  max_user_watches
[root@nfs01 data]# cat /proc/sys/fs/inotify/max_user_watches 
8192
[root@nfs01 data]# echo "50000000" > /proc/sys/fs/inotify/max_user_watches
[root@nfs01 data]# cat /proc/sys/fs/inotify/max_user_watches 
50000000
[root@nfs01 data]# cat /proc/sys/fs/inotify/max_queued_events 
16384
[root@nfs01 data]# echo "326790" > /proc/sys/fs/inotify/max_queued_events
[root@nfs01 data]# cat /proc/sys/fs/inotify/max_queued_events 
326790
[root@nfs01 data]# sysctl -p

5.7 Rsync+inotify Real-time Data Synchronization Concurrent Simple Test

10K-100K

100 concurrencies per second

[root@nfs01 data]# paste inotify_100_server.log
inotify_100_backup_server.log > inotify_100.txt
[root@nfs01 data]# cat inotify_100.txt
23:05       34227   23:05   34227
23:05       34387   23:05   34387
23:05       35027   23:05   35027
23:05       35587   23:05   35587
23:05       36473   23:05   36473
23:05       36707   23:05   36707
23:05       37587   23:05   37587 
//The following is omitted ____________

Inotify real-time concurrency:

CONCLUSION: After testing, 200 files per second are concurrent and data synchronization is almost without delay (less than 1 second).

5.8 Advantages of inotify:

1) Monitor the changes of file system events and realize real-time data synchronization through synchronization tools.

5.9 Intify shortcomings

1) If concurrency is greater than 200 files (10-100k), synchronization will be delayed.
2) The scripts we wrote before are all pushed once at a time, but they are incremental. It can also synchronize only the changing files, without changing them.
3) When events are monitored, rsync synchronization is a single process, while sersync is a multi-process synchronization. With inotify-tools, why develop sersync?

5.10 serysync has many functions: (inotify+rsync command)

1) Support for configuration file management
2) Real daemon socket
3) Timely retransmit of failed files (Timing Task Function)
4) Third party HTTP interface (e.g. updating cdn cache)
5) Default multi-process rsync synchronization

5.11 Summary of Real-time Synchronization Scheme for High Concurrent Data:

1) inotify (sersync) + rsync, which is file level.
2) drbd file system level, file system level, block-based synchronization, drawbacks: standby node data is not available
3) Synchronization function of third-party software: mysql synchronization (master-slave replication), oracle, mongodb
4) The program is written in pairs and two servers are written directly.
5) Use product business logic to solve (read-write separation, backup can not read, reader)

Explain:
The images or attachments uploaded by users are separately stored on the main NFS server.
Users read data from two NFS backup servers.
NFS master and two NFS backups are synchronized in real time by inotify+rsync.

6) NFS Cluster (1,4,5 Scheme Integration) (Dual Write Master Storage, Standby Storage inotify (sersync) +rsync