Linux Practical Teaching Notes 23:Inotify Event Monitoring Tool

Keywords: Linux inotify rsync yum EPEL

Section 23 Intify Event Monitoring Tool

Label (Space Separation): Linux Practical Teaching Notes - Chen Siqi

This teaching note is a summary of my study and work career, which is the first draft (there are many imperfections), the original work, allowed to be reproduced, when reproduced, please be sure to indicate the original source of the article, author information and this statement in the form of hyperlinks. Otherwise, legal liability will be pursued. http://www.cnblogs.com/chensiqiqi/

Chapter 1, the construction of NFS storage server and backup backup server.

Detailed knowledge and construction, please pay attention to:

http://www.cnblogs.com/chensiqiqi/p/6514315.html Rsync Data Synchronization Tool

http://www.cnblogs.com/chensiqiqi/p/6530859.html Enterprise NFS Network File Sharing Service

http://www.cnblogs.com/chensiqiqi/p/6531003.html [Rsync Project Practice] Backup the whole network server data

Chapter 2: Origin of rsync + inotify combination

Rsync (remote sync) remote synchronization tool, through Rsync can achieve incremental backup synchronization of remote server data, but Rsync itself also has bottlenecks. When synchronizing data, Rsync uses the core algorithm to compare the target files of remote server, and only performs differential synchronization. We can imagine that if the number of files on the server reaches the order of millions or even tens of millions, file comparison will be very time-consuming. And change is often a very small part of it, which is a very inefficient way. The emergence of inotify can alleviate the shortcomings of Rsync and make up for the shortcomings.

Chapter 3 Introduction to inotify

  • Inotify is a powerful, fine-grained, asynchronous file system event monitoring mechanism (software). Since 2.6.13, the linux kernel has added Inotify support. Through Inotify, various events such as adding, deleting, modifying and moving in the file system can be monitored. With this kernel interface, third-party software can monitor the changes of files in the file system, while inoti software can monitor the changes of files in the file system. Fy-tools is the software that implements such monitoring. There is also sersync developed by Zhou Yang, a native of China, in Jinshan Company.
  • Inotify is actually an event-driven mechanism, which provides a real-time response mechanism for applications to monitor file system events without the need for polling mechanisms such as cron to obtain events. Cron and other mechanisms not only can not achieve real-time, but also consume a lot of system resources. In contrast, inotify is event-driven, which can achieve real-time response to event processing, and does not consume system resources caused by polling. It is a very natural event notification interface, which is also consistent with the event mechanism of the natural world.
  • There are several kinds of software to implement inotify
    1)inotify-tools,
    2) sersync (Jinshan Zhou Yang)
    3)lsyncd

In particular:

The following inotify configuration is a configuration process based on rsync services.

Chapter 4 Intify Implementation Preparations

The main premise is that rsync daemon service is successfully configured, and then inotify service can be configured by pushing and pulling data from rsync client.

Chapter 5 Starts Installation

Default yum source:
base + extras + updates

Extended yum source:
epel
1. Netease 163 Source
2. Aliyun epel source

Before installing inotify-tools, please confirm that your linux kernel has reached 2.6.13, and turn on the CONFIG_INOTIFY option at compile time. It can also be checked by the following commands.

5.1 See if the current system supports inotify

[root@backup ~]# uname -r
2.6.32-642.el6.x86_64
[root@backup ~]# ls -l /proc/sys/fs/inotify
//Total dosage 0
-rw-r--r-- 1 root root 0 3 Month 1105:01 max_queued_events
-rw-r--r-- 1 root root 0 3 Month 1105:01 max_user_instances
-rw-r--r-- 1 root root 0 3 Month 1105:01 max_user_watches

#Display three documentation support

Key parameters:

There are three files in the / proc/sys/fs/inotify directory, which limit the inotify mechanism.
max_user_watches: The number of files (single process) that can be monitored by setting the inotifywait or inotifywatch commands
 max_user_instances: Sets the number of processes that each user can run with the inotifywait or inotifywatch command.
max_queued_events: Sets the number of events that the inotify instance event queue can hold.

5.2 Yum installs inotify-tools:

#wget -O /etc/yum.repos.d/epel.repo http://mirrors.aliyun.com/repo/epel-6.repo
yum -y install inotify-tools
rpm -qa inotify-tools

There are two tools installed, inotifywait and inotifywatch.
inotifywait: Waiting for a specific file system event (open, close, delete, etc.) to occur on the monitored file or directory, which is blocked after execution and suitable for shell scripts.
inotifywatch: Collect the statistics of the usage of the monitored file system, referring to the statistics of the number of file system events.

5.3 Details of common parameters of the inotifywait command

inotifywait --help

[root@backup ~]# inotifywait --help
inotifywait 3.14
Wait for a particular event on a file or set of files.
Usage: inotifywait [ options ] file1 [ file2 ] [ file3 ] [ ... ]
Options:
    -h|--help       Show this help text.
    @<file>         Exclude the specified file from being watched.
    --exclude <pattern>
                    Exclude all events on files matching the
                    extended regular expression <pattern>.
    --excludei <pattern>
                    Like --exclude but case insensitive.
    -m|--monitor    Keep listening for events forever.  Without
                    this option, inotifywait will exit after one
                    event is received.
    -d|--daemon     Same as --monitor, except run in the background
                    logging events to a file specified by --outfile.
                    Implies --syslog.
    -r|--recursive  Watch directories recursively.
    --fromfile <file>
                    Read files to watch from <file> or `-' for stdin.
    -o|--outfile <file>
                    Print events to <file> rather than stdout.
    -s|--syslog     Send errors to syslog rather than stderr.
    -q|--quiet      Print less (only print events).
    -qq             Print nothing (not even events).
    --format <fmt>  Print using a specified printf-like format
                    string; read the man page for more details.
    --timefmt <fmt> strftime-compatible format string for use with
                    %T in --format string.
    -c|--csv        Print events in CSV format.
    -t|--timeout <seconds>
                    When listening for a single event, time out after
                    waiting for an event for <seconds> seconds.
                    If <seconds> is 0, inotifywait will never time out.
    -e|--event <event1> [ -e|--event <event2> ... ]
        Listen for specific event(s).  If omitted, all events are 
        listened for.

Exit status:
    0  -  An event you asked to watch for was received.
    1  -  An event you did not ask to watch for was received
          (usually delete_self or unmount), or some error occurred.
    2  -  The --timeout option was given and no events occurred
          in the specified interval of time.

Events:
    access      file or directory contents were read
    modify      file or directory contents were written
    attrib      file or directory attributes changed
    close_write file or directory closed, after being opened in
                writeable mode
    close_nowrite   file or directory closed, after being opened in
                read-only mode
    close       file or directory closed, regardless of read/write mode
    open        file or directory opened
    moved_to    file or directory moved to watched directory
    moved_from  file or directory moved from watched directory
    move        file or directory moved to or from watched directory
    **create**      file or directory created within watched directory
    **delete**      file or directory deleted within watched directory
    delete_self file or directory was deleted
    unmount     file system containing file or directory unmounted

Here's a list to explain the meaning of each parameter in detail

inotifywait parameter Meaning explanation
-r --recursive Recursive Query Catalogue
-q --quiet Print very little information, only monitor event information
-m,--monitor Always maintain event monitoring status
--exclude When excluding files or directories, case-insensitive.
--timefmt Format of specified time output
--format Printing uses a specified output similar to a format string
-e,--event This parameter allows you to specify events to be monitored, as shown in the following list

- e:--event's event Meanings

Events Meaning
access Files or directories are read
modify File or directory contents are modified
attrib File or directory attributes changed
close Files or directories are closed, regardless of read/write mode
open Files or directories are opened
moved_to Files or directories are moved to another directory
move Files or directories are moved to another directory or from another directory to the current directory
create Files or directories are created in the current directory
delete Files or directories deleted
umount File system uninstalled

5.4 Manual Testing and Monitoring Events

Open two windows

5.4.1 Test create

Enter the following in the first window:
[root@backup ~]# ls /backup
[root@backup ~]# inotifywait -mrq --timefmt '%y %m %d %H %M' --format '%T %w%f' -e create /backup

In the second window: enter the following
[root@backup ~]# cd /backup
[root@backup backup]# touch chensiqi

Back to the first window, the following appears:
17 03 11 07 19 /backup/chensiqi

# Command description
 inotifywait: ionotify's command tool
 - M R q:-q only input short message-r, recursively monitor the entire directory including subdirectory-m for continuous monitoring
 - timefmt specifies the time format of the output 
Format: Specifies the format of the output information
 - e create: Establish the time type of monitoring and monitor the creation event.

5.4.2 Test delte

The first window enters the following information:
[root@backup ~]# inotifywait -mrq --timefmt '%y %m %d %H %M' --format '%T %w%f' -e delete /backup

The second window enters the following information:
[root@backup backup]# rm -rf chensiqi

At this point, the first window will display the following information:
17 03 11 07 29 /backup/chensiqi

# Instructions:
- e delete: Specifies the type of event to listen on. Listen for delete events

5.4.3 Test close_write

The first window enters the following information:
inotifywait -mrq --timefmt '%y %m %d %H %M' --format '%T %w%f' -e close_write /backup
 The second window enters the following information:
[root@backup backup]# touch close_write.log
[root@backup backup]# echo 111 >> close_write.log 
[root@backup backup]# rm -f close_write.log 
At this point, the first window will display the following information:
17 03 11 07 38 /backup/close_write.log
17 03 11 07 39 /backup/close_write.log

# Instructions:
- e close_write: Specifies the listening type. Close the listening file write mode.

5.4.4 Test move_to

The first window enters the following information:
[root@backup ~]# inotifywait -mrq --timefmt '%y %m %d %H %M' --format '%T %w%f' -e moved_to /backup  
The second window enters the following information:

At this point, the first window will display the following information:
[root@backup backup]# touch chensiqi
[root@backup backup]# mv chensiqi chen
[root@backup backup]# mkdir ddddd
[root@backup backup]# mv chen ddddd/

5.5 Writing inotify real-time monitoring script

#!/bin/bash

backup_Server=172.16.1.41


/usr/bin/inotifywait -mrq --format '%w%f' -e create,close_write,delete /data | while read line
do
    cd /data
    rsync -az ./ --delete rsync_backup@$backup_Server::nfsbackup --password-file=/etc/rsync.password
done

Tips:

  • The reason that the script above is inefficient is that whenever the directory changes, everything in my entire directory will be pushed once. Therefore, we can make the following changes to improve efficiency
#!/bin/bash

Path=/data
backup_Server=172.16.1.41


/usr/bin/inotifywait -mrq --format '%w%f' -e create,close_write,delete /data  | while read line  
do
    if [ -f $line ];then
        rsync -az $line --delete rsync_backup@$backup_Server::nfsbackup --password-file=/etc/rsync.password       
    else
        cd $Path &&\
        rsync -az ./ --delete rsync_backup@$backup_Server::nfsbackup --password-file=/etc/rsync.password
    fi

done

The script can be added to boot up:
echo "/bin/sh /server/scripts/inotify.sh &" >> /etc/rc.local
Tips:
One - Represents running this command from the background.

5.6 Key Parameter Adjustment

There are three files in the / proc/sys/fs/inotify directory, which limit the inotify mechanism.
max_user_watches: The number of files (single process) that can be monitored by setting the inotifywait or inotifywatch commands
max_user_instances: Sets the number of processes per user can run inotifywait or inotifywatch commands
max_queued_events: Sets the number of events that the inotify instance event queue can hold.

Actual adjustment:

[root@nfs01 data]# cat /proc/sys/fs/inotify/max_
max_queued_events   max_user_instances  max_user_watches
[root@nfs01 data]# cat /proc/sys/fs/inotify/max_user_watches 
8192
[root@nfs01 data]# echo "50000000" > /proc/sys/fs/inotify/max_user_watches
[root@nfs01 data]# cat /proc/sys/fs/inotify/max_user_watches 
50000000
[root@nfs01 data]# cat /proc/sys/fs/inotify/max_queued_events 
16384
[root@nfs01 data]# echo "326790" > /proc/sys/fs/inotify/max_queued_events
[root@nfs01 data]# cat /proc/sys/fs/inotify/max_queued_events 
326790
[root@nfs01 data]# sysctl -p

5.7 Rsync+inotify Real-time Data Synchronization Concurrent Simple Test

10K-100K

100 concurrencies per second

[root@nfs01 data]# paste inotify_100_server.log
inotify_100_backup_server.log > inotify_100.txt
[root@nfs01 data]# cat inotify_100.txt
23:05       34227   23:05   34227
23:05       34387   23:05   34387
23:05       35027   23:05   35027
23:05       35587   23:05   35587
23:05       36473   23:05   36473
23:05       36707   23:05   36707
23:05       37587   23:05   37587 
//The following is omitted ____________

Inotify real-time concurrency:

CONCLUSION: After testing, 200 files per second are concurrent and data synchronization is almost without delay (less than 1 second).

5.8 Advantages of inotify:

1) Monitor the changes of file system events and realize real-time data synchronization through synchronization tools.

5.9 Intify shortcomings

1) If concurrency is greater than 200 files (10-100k), synchronization will be delayed.
2) The scripts we wrote before are all pushed once at a time, but they are incremental. It can also synchronize only the changing files, without changing them.
3) When events are monitored, rsync synchronization is a single process, while sersync is a multi-process synchronization. With inotify-tools, why develop sersync?

5.10 serysync has many functions: (inotify+rsync command)

1) Support for configuration file management
2) Real daemon socket
3) Timely retransmit of failed files (Timing Task Function)
4) Third party HTTP interface (e.g. updating cdn cache)
5) Default multi-process rsync synchronization

5.11 Summary of Real-time Synchronization Scheme for High Concurrent Data:

1) inotify (sersync) + rsync, which is file level.
2) drbd file system level, file system level, block-based synchronization, drawbacks: standby node data is not available
3) Synchronization function of third-party software: mysql synchronization (master-slave replication), oracle, mongodb
4) The program is written in pairs and two servers are written directly.
5) Use product business logic to solve (read-write separation, backup can not read, reader)


Explain:
The images or attachments uploaded by users are separately stored on the main NFS server.
Users read data from two NFS backup servers.
NFS master and two NFS backups are synchronized in real time by inotify+rsync.

6) NFS Cluster (1,4,5 Scheme Integration) (Dual Write Master Storage, Standby Storage inotify (sersync) +rsync

Posted by Jackdaw on Thu, 18 Apr 2019 03:06:33 -0700