Grep command

1. Examples of Basic Regular Expressions - Finding Specific Characters

Here we will take the / etc/passwd file of all the local users as an example.

Demo1

[root@localhost ~]# Grep-n "root"/etc/passwd//-n denotes the display line number
1:root:x:0:0:root:/root:/bin/bash
10:operator:x:11:0:operator:/root:/sbin/nologin

2. Examples of Basic Regular Expressions: Finding Collection Characters

When there are duplicate characters, set matching can be done with "[]", only one character in "[]" is matched at a time.

Demo2

[root@localhost ~]# grep -n "[fn]tp" /etc/passwd
12:ftp:x:14:50:FTP User:/var/ftp:/sbin/nologin
27:ntp:x:38:38::/etc/ntp:/sbin/nologin

3. Examples of Basic Regular Expressions - Reverse Selection

Add "^" in parentheses in "[]" to indicate reverse selection. (Friends who have a certain foundation must know that "^[]" means to locate the header of a line, where the meaning of the position inside and outside of "^" will be completely different.)

Demo3

[root@localhost ~]# Grep-n "^[^ root]" / etc / passwd // / matches all options except those starting with root
2:bin:x:1:1:bin:/bin:/sbin/nologin
3:daemon:x:2:2:daemon:/sbin:/sbin/nologin
......
42:named:x:25:25:Named:/var/named:/sbin/nologin

4. Instances of Basic Regular Expressions - Escapes

A metacharacter in a regular expression, so here you need to use the escape character "" to convert a character of special significance into a common character.

Demo4

[root@localhost ~]# grep -n '\.$' test.txt 1:he was short and fat.
2:He was wearing a blue polo shirt with black pants. 3:The home of Football on BBC Sport online.
5:google is the best tools for search keyword.

5. Ex amp les of Basic Regular Expressions: Finding Any Character & Finding Duplicate Characters

The decimal point (.) in a regular expression is also a metacharacter, representing any character.

Demo5-1

[root@localhost ~]# Grep-n "r. t" / etc / passwd /(.) decimal point here represents any character
1:root:x:0:0:root:/root:/bin/bash
10:operator:x:11:0:operator:/root:/sbin/nologin
12:ftp:x:14:50:FTP User:/var/ftp:/sbin/nologin

In the above results, the "root" string "r. t" matches the rule. If you want to query oo, ooo, OOo and other information, you need to use the asterisk (*) metacharacter. However, it should be noted that "*" represents the repetition of zero or more previous single characters. "O*" means having zero (i.e., empty characters) or more characters than or equal to one "o"

Demo5-2

[root@localhost ~]# grep -n "oo*" /etc/passwd
1:root:x:0:0:root:/root:/bin/bash
2:bin:x:1:1:bin:/bin:/sbin/nologin
3:daemon:x:2:2:daemon:/sbin:/sbin/nologin
4:adm:x:3:4:adm:/var/adm:/sbin/nologin
5:lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin
......

6. Examples of Basic Regular Expressions: Finding Continuous Character Range

For example, to find three to five consecutive characters of o, you need to use the bounded character "{}" in the underlying regular expression. Because "{}" has a special meaning in Shell, when using "{}" character, we need to use the escape character "" to convert "{}" character into ordinary character.

Demo6

[root@localhost ~]# Grep-n "0{2,}" / etc/passwd// denotes a string containing more than 2 o in it
11:games:x:12:100:games:/usr/games:/sbin/nologin
41:zhy:x:1000:1000:zhy:/home/zhy:/bin/bash

Egrep command

In addition, the grep command only supports basic regular expressions, and if you use extended regular expressions, you need to use egrep or awk commands. The awk command is explained later. Here we use the egrep command directly. The use of the egrep command is basically similar to that of the grep command. (The egrep command that grep command can use can also be used)

Extended regular expression metacharacters	Effect
+	Function: Repeat one or more previous characters
？	Function: The first character of zero or one
|	Function: Use or find multiple characters
()	Function: Find the Group string
()+	Function: Identify multiple repetitive groups

Demo Demo Demo

[root@localhost ~]# Egrep-n "10+" / etc / passwd // Use "+" Extended Metacharacter
11:games:x:12:100:games:/usr/games:/sbin/nologin
31:qemu:x:107:107:qemu user:/:/sbin/nologin
41:zhy:x:1000:1000:zhy:/home/zhy:/bin/bash

[root@localhost ~]# Egrep-n "10?" / etc / passwd // / use "? Extended metacharacter
2:bin:x:1:1:bin:/bin:/sbin/nologin
9:mail:x:8:12:mail:/var/spool/mail:/sbin/nologin
10:operator:x:11:0:operator:/root:/sbin/nologin
11:games:x:12:100:games:/usr/games:/sbin/nologin

[root@localhost ~]# Egrep-n'root | zhy'/ etc/passwd // Use "|" Extended Metacharacter
1:root:x:0:0:root:/root:/bin/bash
10:operator:x:11:0:operator:/root:/sbin/nologin
41:zhy:x:1000:1000:zhy:/home/zhy:/bin/bash

[root@localhost ~]# Egrep-n'(f | n) tp'/ etc / passwd // using "()" extended metacharacter, which can be used with "|"
12:ftp:x:14:50:FTP User:/var/ftp:/sbin/nologin
27:ntp:x:38:38::/etc/ntp:/sbin/nologin

Sed command

sed is a good file processing tool. It is a pipeline command. It is mainly processed by behavior units. It can replace, delete, add and select data rows.

sed's workflow mainly includes three processes: reading, executing and displaying.

Click Read: sed reads a line of content from the input stream (file, pipeline, standard input) and stores it in a temporary slowdown

Stroke area (also known as pattern space).

Check execution: By default, all sed commands are executed sequentially in the schema space, unless the address of the line is specified, the SED command will execute sequentially on all lines.

Bah display: Send the modified content to the output stream. After sending the data again, the schema space will be emptied.

Before all the file contents are processed, the above process will be repeated until all the contents are processed.

Note: By default, all sed commands are in schema space and are not saved.

Sed command format

Common Options

Commonly used "operation" parameters

Examples of basic usage:

Output all, the effect is the same as the cat command

[root@localhost ~]# Sed-n'p'/etc/passwd//effect equivalent to cat command
root:x:0:0:root:/root:/bin/bash
bin:x:1:1:bin:/bin:/sbin/nologin
daemon:x:2:2:daemon:/sbin:/sbin/nologin
adm:x:3:4:adm:/var/adm:/sbin/nologin
......

Output a particular line, or a section of line

[root@localhost ~]# Sed-n'10p'/etc/passwd//Output Line 10
operator:x:11:0:operator:/root:/sbin/nologin

[root@localhost ~]# Sed-n'2,4p'/etc/passwd//output 2-4 lines
bin:x:1:1:bin:/bin:/sbin/nologin
daemon:x:2:2:daemon:/sbin:/sbin/nologin
adm:x:3:4:adm:/var/adm:/sbin/nologin

Output all odd rows

[root@localhost ~]# Sed-n'n; p'/etc/passwd//output odd rows, even behavior p;
bin:x:1:1:bin:/bin:/sbin/nologin
adm:x:3:4:adm:/var/adm:/sbin/nologin
sync:x:5:0:sync:/sbin:/bin/sync
halt:x:7:0:halt:/sbin:/sbin/halt
......

Similarly, besides the basic usage, the sed command can also be used in conjunction with regular expressions.

Output lines containing specific content (like grep commands, you can use ^, $to locate the beginning and end of a line)

[root@localhost ~]# sed -n '/root/p' /etc/passwd
root:x:0:0:root:/root:/bin/bash
operator:x:11:0:operator:/root:/sbin/nologin

Output lines containing specific words

[root@localhost ~]# Sed-n'/ root > p'/ etc/passwd / passwd \ represents word boundaries
root:x:0:0:root:/root:/bin/bash
operator:x:11:0:operator:/root:/sbin/nologin

Replace eligible text

sed 's/the/THE/' test.txt //Replace THE first of THE lines with THE

sed 's/l/L/3' test.txt //Replace the third L in each row with L

sed 's/the/THE/g' test.txt //Replace all THE thes in THE file with THE

sed 's/o//g' test.txt //Delete all o in the file (replace with empty string)

sed 's/^/#/' test.txt //Insert #: at the beginning of each line

sed '/the/s/^/#/' test.txt //Insert # at the beginning of each line containing the

sed 's/$/EOF/' test.txt //Insert the string EOF at the end of each line

sed '3,5s/the/THE/g' test.txt //Replace all THE thes in lines 3 to 5 with THE

sed '/the/s/o/O/g' test.txt //Replace o in all rows containing the with O

Migrate text

sed '/the/{H;d};$G' test.txt //Migrate the row containing the to the end of the file, {;} for multiple operations

sed '1,5{H;d};17G' test.txt //Transfer lines 1-5 to lines 17

sed '/the/w out.file' test.txt //Save the row containing the as a file out.file

sed '/the/r /etc/hostname' test.txt //After adding the contents of the file / etc/hostname to each line containing the

sed '3aNew' test.txt //Insert a new line after line 3, New

sed '/the/aNew' test.txt //Insert a New line after each line containing the

sed '3aNew1\nNew2' test.txt //Insert multi-line content after line 3, withn in the middle representing newline

AWK command

AWK is a programming language tool for text processing. AWK is similar to shell programming language in many ways, although it has its own grammar and is one of the most powerful data processing engines available in any environment.

Basic structure

Basic use examples

[root@localhost ~]# awk -F: '{print $1,$3}' /etc/passwd
//With ":" as the separator, filter out the contents of the first column and the third column, and output (by default, the separator of the field is a space or tab key).
root 0
bin 1
daemon 2
adm 3
......

Special built-in variables (can be directly introduced for use)

Output text by line

[root@localhost ~]# awk 'NR==2,NR==4{print}' /etc/passwd       
//Output the second to fourth lines of text content
bin:x:1:1:bin:/bin:/sbin/nologin
daemon:x:2:2:daemon:/sbin:/sbin/nologin
adm:x:3:4:adm:/var/adm:/sbin/nologin

[root@localhost ~]# awk -F ":" 'NR==2,NR==4{print $1,$3}' /etc/passwd
//Output the first and third columns of the second to fourth rows
bin 1
daemon 2
adm 3

[root@localhost ~]# awk '(NR==1)||(NR==4){print}' /etc/passwd
//Output the first and fourth lines
root:x:0:0:root:/root:/bin/bash
adm:x:3:4:adm:/var/adm:/sbin/nologin

Output odd and even lines (in awk, you can use the logical operator "&", "and", "and", "or", "or"! It can also perform simple mathematical operations, such as +, -,*, /,%, ^ for addition, subtraction, multiplication, division, redundancy and multiplication, respectively.

[root@localhost ~]# Awk'NR%2==1 {print}'/etc/passwd// Output odd rows
root:x:0:0:root:/root:/bin/bash
daemon:x:2:2:daemon:/sbin:/sbin/nologin
lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin
shutdown:x:6:0:shutdown:/sbin:/sbin/shutdown
......

[root@localhost ~]# Awk'NR%2=== 0 {print}'/etc/passwd//output even rows
bin:x:1:1:bin:/bin:/sbin/nologin
adm:x:3:4:adm:/var/adm:/sbin/nologin
sync:x:5:0:sync:/sbin:/bin/sync
halt:x:7:0:halt:/sbin:/sbin/halt
......

Output lines starting with root

[root@localhost ~]# awk '/^root/{print}' /etc/passwd
root:x:0:0:root:/root:/bin/bash

Statistics of rows ending in / bin/bash

[root@localhost ~]# awk 'BEGIN {x=0} ; /\/bin\/bash$/{x++};END {print x}' /etc/passwd
2

Number of text paragraphs separated by blank lines:

[root@localhost opt]# vim name.txt
zhangsan:lisi:wangwu
zhaoliu:liuliu
heiba:heihei
[root@localhost opt]# awk 'BEGIN{RS=":"};END{print NR}' /opt/name.txt
5
//Statistical rules: encounter key symbols, break lines

Call the w command and use it to count the number of online users:

[root@localhost opt]# awk 'BEGIN {while ("w" | getline) n++;{print n-2}}'
1