shell script -- regular expression

Keywords: vim network SELinux Mac

Basic regularity

^word     ##Search for the beginning of the ^ line in vi/vim starting with word
word$     ##Search for the end of the $line in vi/vim that ends in word
^$        ##Blank line
.         ##Represents and can only represent any one character
\         ##For example, \. It only represents the point itself, escape symbol, and allows characters with special identity meaning to take off vest and restore
\n        ##Newline character
\r        ##Matching carriage return
\w        ##Match any character and number
*         ##Repeat the preceding character 0 or more times
.*        ##Match all characters. Example: ^. * starts with any number of characters,. * $ends with any number of characters
[abc]     ##Match any character in the character set
[^abc]    ##Matches content that does not contain any characters after ^. The ^ in brackets is reversed
[1-9]     ##Represents any character within the range of matching brackets
a\{n,m\}  ##Repeat the previous repeated character n to m times. If egrep and sed-r are used, the slash can be removed
\{n,\}    ##Repeat at least n previous characters. If egrep and sed-r are used, the slash can be removed
\{n\}     ##Repeat the previous repeated character n times. If egrep and sed-r are used, the slash can be removed
\{,m}\    ##Repeat less than m times

Note: egrep, grep – E or sed – r filtering general special characters can not be escaped

2, Extended regularity (egrep or grep-e)

+To repeat the preceding character once or more
 Repeat the previous character 0 times or once
 |To find multiple matching strings by means of or
 () find the string in brackets


As one of the three most commonly u sed text processing tools in linux, it is necessary to master its usage.

Let's talk about the common format of grep command: grep [options] "mode" [file]

There are three grep families: grep, egrep, fgrep.

Common options:

-E: Turn on the regular expression for the extension.

-i: ignore case.

-v: In turn, only those that do not match will be printed, while those that match will not.

-n: Show line numbers

-w: The matched text can only be a word, not a part of the word. For example, there is a liker in the text, and what I am searching for is just like, so I can use the - W option to avoid matching liker

-c: Show how many rows have been matched instead of what has been matched. Note that if you use the - cv option at the same time, show how many rows have not been matched.

-o: Only the strings matched by the pattern are displayed.

– color: highlight the matched content in color.

-A n: display the line of the matched string and the next N lines, after

-B N: display the line of the matched string and its first n lines, before

-C n: display the line of the matched string and the n lines before and after it, context

cat /proc/meminfo |grep -e Mem -e Cache -e Swap     ##View system memory, cache, swap partition - e to match multiple expressions
grep -R -o -n -E  '[a-z0-9_]+\@[a-z0-9_]+\.[a-z]{2,4}' /etc/     ##Find the email addresses in all files in the / etc directory; - R recursion, - n represents the matching line number, - o only outputs the matching content, - E supports extended regular expressions,
grep -R -c 'HOSTNAME' /etc/ |grep -v "0$"     ##The number of times to find the file containing "HOSTNAME" in the / etc / directory, - c counts the number of matches, - v negates
grep -R -l 'HOSTNAME' /etc/           ##Find the filename containing 'HOSTNAME', - l show matching filenames, - l show mismatched filenames
dmesg | grep -n --color=auto 'eth'       ##Find the row of eth in kernel log, display color and line number
dmesg | grep -n -A3 -B2 --color=auto 'eth'     ##Use dmesg to list the core information, and then use grep to find out the row containing eth. The first two lines and the last three lines of the row where the keyword is located will be found out and displayed together
cat /etc/passwd |grep -c bash$         ##Count the number of users that can log in in the system
touch /tmp/{123,123123,456,1234567}.txt   ##Create a test file. The following three commands have the same effect. Match the file name 123, which can contain one or more
ls |grep -E '(123)+'
ls |grep '\(123\)\+'
ls |egrep '(123)+'
ps -ef |grep -c httpd             ##Count the number of httpd processes
grep -C 4 'games' --color /etc/passwd       ##Display 4 lines before and after "- C" matched by games
grep ^adm /etc/group             ##Viewing adm group information
ip a |grep -E '^[0-9]' |awk -F : '{print $2}'     ##Get network card name
ifconfig eth0 |grep -E -o 'inet addr:[^ ]*' |grep -o '[0-9.]*'   ##Intercept the ip address, [^] * indicates the non empty character as the end character, [0-9.] * indicates the combination of number and point
ip a |grep inet |grep eth0 |grep -o "inet[^/]*" |grep -o "[0-9.]*"  ##Intercept ip address
ifconfig eth0 |grep -i hwaddr |awk '{print $5}'   ##Intercept MAC address
ip a |grep -A 3 "eth0" |grep link/ether |grep -o "ether[^r]*" |grep -o -E "[0-9a-f:]+"|grep -E "[0-9a-f:]{2}$"      ##Intercept MAC address
grep "^m" oldboy.log       ##Filter output lines beginning with m
grep "m$" oldboy.log 
grep -vn "^$" oldboy.log       ##Filtered blank lines
grep -o "0*" oldboy.log 
grep -o "oldb.y" oldboy.log 
grep "\.$" oldboy.log       ##Lines ending with
grep "0\{3\}" oldboy.log       ##Repeat three times.


Syntax: sed [options] 'command' file(s)

-n suppress auto print pattern space, sed default output all, - n cancel default output
 -i edit file
 -r supports extended regular expressions

1. changes:

Syntax: sed '/Regular matching condition/s/old/new/g' file
sed 's/dhcp/static/g' /etc/sysconfig/network-scripts/ifcfg-eth1 ##Display only, do not modify
sed -i 's/dhcp/static/g' /etc/sysconfig/network-scripts/ifcfg-eth1 ##Modify only, not display
sed -i 's/dhcp/static/g' ip ##Replace all dhcp with static
sed -i '/^IP1/s/static/dhcp/g' ip ##Replace the line at the beginning of IP1
sed -i '2s/static/dhcp/g' ip ##Specify specific line number 2 line replacement
cat -n /etc/selinux/config ##View and display line numbers
sed -i '7s/disabled/enforcing/g' /etc/selinux/config ##Enable selinux

2. delete:

Syntax: sed '/Expression/d' file
vim ip ##Add blank line
sed '/^$/d' ip ##Delete blank lines and display on screen
sed -i '/IP1/d' ip ##Delete row with IP1
sed -i '/^IP2/d' ip ##Delete rows starting with IP2
sed -i '2d' ip ##Delete second line

3. increase:

Syntax: sed ' /Expression/a "Text to add"' file
sed 'a IP3=static' ip ##Add IP3=static after each line
sed '3a IP3=static' ip ##Only add IP3=static after line 3, and the display will not be modified
sed '3i IP3=static' ip ##Only add IP3=static before line 3, the display will not be modified
sed -i '3a IP3=static' ip ##Modify, do not display
sed -i '/^IP3/a "test add"' ip ##Add after line starting with IP3

4. check:

Syntax: sed -n '/Expression/p' file
sed -n '2p' /etc/hosts ##View second line
sed -n '/www/p' /var/named/chroot/var/named/ ##View resolution records containing www
sed -n '/.100$/p' /var/named/chroot/var/named/ ##View lines ending with. 100
sed -n '2~2p' ip ##From the second line, display every two lines
ifconfig eth0|sed -n '2p'|sed 's#.*dr:##g'|sed 's# Bc.*##g'    ##Note: when sed If the content processed by the command is multiline, the/As a separation of expressions, if sed The content processed by the command is single line content, which serves as the intercepting function#Number as separator; 
ifconfig eth0|sed -n '2p'|sed -r 's#(.*dr:)(.*)(Bc.*$)#\2#g'    ##-r supports extended regular, 2 escapes 2 and prints out the second range (. *) 
ifconfig eth0|sed -n '2p'|sed -r 's#.*dr:(.*) Bc.*$#\1#g' 
ifconfig eth0|sed -nr '2s#^.*dr:(.*) Bc.*$#\1#gp'
ifconfig eth0|sed -nr '1s#^.*dr (.*)#\1#gp'
ifconfig eth0|sed -n '1p'|sed -r 's#(^.*dr )(.*)#\2#g'
ifconfig eth0|sed -n '1p'|sed 's#^.*dr ##g'     
ifconfig eth0|sed -nr '1s#^.*t (.*) 00.*$#\1#gp'
stat /etc/hosts|sed -n '4p'               
Access: (0644/-rw-r--r--) Uid: (  0/  root)  Gid: (  0/  root)
stat /etc/hosts|sed -n '4p'|sed 's#^.*ss: (##g'|sed 's#/-.*$##g'
stat /etc/hosts|sed -n '4p'|sed -r 's#^.*s: \((.*)/-.*$#\1#g'
stat /etc/hosts|sed -nr '4s#^.*s: \((.*)/-.*$#\1#gp'
stat /etc/hosts|sed -nr '4s#(^.*s: \()(.*)(/-.*$)#\2#gp'


awk [options] 'mode {action}' file 1 file 2

Options: - F specifies the input separator, which can be a string or a regular expression
 Common actions: print, printf
chkconfig --list |grep 3:Enable |awk '{print $1}'
tail -1 /etc/passwd |awk -F ':' 'BEGIN{OFS="---"}{print $1,$6,$7}'  ##OFS specifies the output separator
ifconfig eth1 |awk -F '[ :]+' 'NR==2 {print $4}'
ifconfig eth1 |awk -F '[ :]+' 'NR==2 {print "eth1_ip="$4}' ##Display content can be added
awk 'BEGIN {print "line one \nline two\nline three"}'

Matching range: the specified matching range in the format of part1,part2

awk -F : '$3==3,$3==10{print $1,$3,$7}' /etc/passwd
awk -F : '$1=="root",$1=="adm"{print $1,$3,$7}' /etc/passwd
awk -F : '/^r/,/^a/{print $1,$3,$7}' /etc/passwd

Principle of awk block:
Regional composition:

BEGIN {action} ා񖓿񖓿ා񖓿operation before processing the first line of text
 {action} ා񖓿ා񖓿񖓿ා񖓿processing operation for each line of text
 END {action} {action after the last line of text is processed

Execution process:
The initialization operation in BEGIN {} block is performed first;
Then loop through the specified data file to read a data row (automatically update NF, NR, $0, $1 And execute 'pattern or condition' {action} ';
Finally, the subsequent processing operations in the END {} block are performed.

awk -F : 'BEGIN{printf "%-10s%-10s%-20s\n","UserName","ID","Shell"}{printf "%-10s%-10s%-20s\n",$1,$3,$7}' /etc/passwd ##Print header BEGIN {} before awk processing
ifconfig eth0 |awk -F':' 'NR==2{print $2,$4}'|awk 'BEGIN{OFS=" / "}{print "IP="$1,"MASK="$3}'

Variables for awk:

awk variables:
FS: column separator, default bit blank
 RS: line separator, default line break
 OFS: output column separator
 ORS: output line separator
 awk built in variables:
NR: number of rows in process
 FNR: number of lines in a single file
 NF: number of columns

Case study:

ifconfig eth1 |awk '{print NR}'
ifconfig eth1 |awk '{print NF}'

Custom variable case:

awk 'BEGIN{test="";print test}'
awk -v test="" 'BEGIN{print test}'

Use of printf:
Format: printf "format", list 1, list 2

a. Format must be specified to specify the output format of the following item (list)
b. The printf statement does not automatically print line breaks: \ n
 c.format is% plus one character, as follows:
%c: ASCII code of display character
 %d. % i: decimal integer
 %f: Show floating point numbers (decimal)
%s: Display string
 %u: Unsigned integer
 %%: display%

d. Modifier: N: display width, -: align left, +: display numeric symbol, such as% - C (align left)
Case study:

chkconfig --list |grep 3:Enable |awk '{printf "%-10s",$1}' ##Display on unified line
awk -F : '{printf "%-15s %-10d %-10s\n",$1,$3,$7}' /etc/passwd

awk operators:

Arithmetic operators: x^y, x/y, x+y, x-y, x%y
 Comparison operators: >=
Logical operators: & &, |!

Common pattern types of awk

regexp: awk -F: '/ ^ u/{print }' / etc/passwd

expression: if the value bit is not 0 or the bit is not empty, the condition is satisfied, such as $1 ~ /foo / or $1 = = "root"
Case study:

awk -F : '$3>=500{print $1,$3,$7}' /etc/passwd ##Print ordinary users
awk -F : '$3+1<=100&&$3+1>=10{print $1,$3,$7}' /etc/passwd ##Users with UID between 10-100
awk -F : '$2=="!!"{print $1,$2}' /etc/shadow ##Check for uninitialized users
passwd -d u01 
awk -F : '$2==""{print $1}' /etc/shadow ##Print users with blank password
awk -F : '$7~"bash$"{print $1,$3,$7}' /etc/passwd ##Match $7 for bash end line
awk -F : '$7!~"bash$"{print $1,$3,$7}' /etc/passwd


The above is the regular expression, grep, sed and awk of shell script introduced by Xiaobian to you. I hope it can help you. If you have any questions, please leave me a message, and Xiaobian will reply to you in time!

86 original articles published, 29 praised, 40000 visitors+
Private letter follow

Posted by Timewell on Sun, 08 Mar 2020 21:01:00 -0700