Shell script three swordsmen - Grep (advanced version of egrep), Sed, Awk commands

Keywords: Linux ftp shell Programming Google

Grep command

1. Examples of Basic Regular Expressions - Finding Specific Characters

Here we will take the / etc/passwd file of all the local users as an example.


[root@localhost ~]# Grep-n "root"/etc/passwd//-n denotes the display line number

2. Examples of Basic Regular Expressions: Finding Collection Characters

When there are duplicate characters, set matching can be done with "[]", only one character in "[]" is matched at a time.


[root@localhost ~]# grep -n "[fn]tp" /etc/passwd
12:ftp:x:14:50:FTP User:/var/ftp:/sbin/nologin

3. Examples of Basic Regular Expressions - Reverse Selection

Add "^" in parentheses in "[]" to indicate reverse selection. (Friends who have a certain foundation must know that "^[]" means to locate the header of a line, where the meaning of the position inside and outside of "^" will be completely different.)


[root@localhost ~]# Grep-n "^[^ root]" / etc / passwd // / matches all options except those starting with root

4. Instances of Basic Regular Expressions - Escapes

A metacharacter in a regular expression, so here you need to use the escape character "" to convert a character of special significance into a common character.


[root@localhost ~]# grep -n '\.$' test.txt 1:he was short and fat.
2:He was wearing a blue polo shirt with black pants. 3:The home of Football on BBC Sport online.
5:google is the best tools for search keyword.

5. Ex amp les of Basic Regular Expressions: Finding Any Character & Finding Duplicate Characters

The decimal point (.) in a regular expression is also a metacharacter, representing any character.


[root@localhost ~]# Grep-n "r. t" / etc / passwd /(.) decimal point here represents any character
12:ftp:x:14:50:FTP User:/var/ftp:/sbin/nologin

In the above results, the "root" string "r. t" matches the rule. If you want to query oo, ooo, OOo and other information, you need to use the asterisk (*) metacharacter. However, it should be noted that "*" represents the repetition of zero or more previous single characters. "O*" means having zero (i.e., empty characters) or more characters than or equal to one "o"


[root@localhost ~]# grep -n "oo*" /etc/passwd

6. Examples of Basic Regular Expressions: Finding Continuous Character Range

For example, to find three to five consecutive characters of o, you need to use the bounded character "{}" in the underlying regular expression. Because "{}" has a special meaning in Shell, when using "{}" character, we need to use the escape character "" to convert "{}" character into ordinary character.


[root@localhost ~]# Grep-n "0{2,}" / etc/passwd// denotes a string containing more than 2 o in it

Egrep command

In addition, the grep command only supports basic regular expressions, and if you use extended regular expressions, you need to use egrep or awk commands. The awk command is explained later. Here we use the egrep command directly. The use of the egrep command is basically similar to that of the grep command. (The egrep command that grep command can use can also be used)

Extended regular expression metacharacters Effect
+ Function: Repeat one or more previous characters
Function: The first character of zero or one
| Function: Use or find multiple characters
() Function: Find the Group string
()+ Function: Identify multiple repetitive groups

Demo Demo Demo

[root@localhost ~]# Egrep-n "10+" / etc / passwd // Use "+" Extended Metacharacter
31:qemu:x:107:107:qemu user:/:/sbin/nologin

[root@localhost ~]# Egrep-n "10?" / etc / passwd // / use "? Extended metacharacter

[root@localhost ~]# Egrep-n'root | zhy'/ etc/passwd // Use "|" Extended Metacharacter

[root@localhost ~]# Egrep-n'(f | n) tp'/ etc / passwd // using "()" extended metacharacter, which can be used with "|"
12:ftp:x:14:50:FTP User:/var/ftp:/sbin/nologin

Sed command

sed is a good file processing tool. It is a pipeline command. It is mainly processed by behavior units. It can replace, delete, add and select data rows.

sed's workflow mainly includes three processes: reading, executing and displaying.

Click Read: sed reads a line of content from the input stream (file, pipeline, standard input) and stores it in a temporary slowdown

Stroke area (also known as pattern space).

Check execution: By default, all sed commands are executed sequentially in the schema space, unless the address of the line is specified, the SED command will execute sequentially on all lines.

Bah display: Send the modified content to the output stream. After sending the data again, the schema space will be emptied.

Before all the file contents are processed, the above process will be repeated until all the contents are processed.

Note: By default, all sed commands are in schema space and are not saved.

Sed command format

sed [option]'operation'parameter

sed [option] - f scriptfile parameter // scriptfile represents script file

Common Options

- e: Represents processing input text files with specified commands or scripts.

- f: Represents that the input text file is processed with the specified script file.

- h: Display help.

- n: Represents only the results after processing.

- i: Edit text files directly.

Commonly used "operation" parameters

A: Increase by adding a specified line below the current line.

c: Replace, replacing the selected row with the specified content.

d: Delete, delete selected rows

i: Insert, insert a specified line above the selected line.

p: Print, which is usually used with the "-n" option

s: Replace, replace the specified character.

y: Character conversion.

Examples of basic usage:

Output all, the effect is the same as the cat command

[root@localhost ~]# Sed-n'p'/etc/passwd//effect equivalent to cat command

Output a particular line, or a section of line

[root@localhost ~]# Sed-n'10p'/etc/passwd//Output Line 10

[root@localhost ~]# Sed-n'2,4p'/etc/passwd//output 2-4 lines

Output all odd rows

[root@localhost ~]# Sed-n'n; p'/etc/passwd//output odd rows, even behavior p;

Similarly, besides the basic usage, the sed command can also be used in conjunction with regular expressions.

Output lines containing specific content (like grep commands, you can use ^, $to locate the beginning and end of a line)

[root@localhost ~]# sed -n '/root/p' /etc/passwd

Output lines containing specific words

[root@localhost ~]# Sed-n'/ root > p'/ etc/passwd / passwd \ represents word boundaries

Replace eligible text

sed 's/the/THE/' test.txt //Replace THE first of THE lines with THE

sed 's/l/L/3' test.txt //Replace the third L in each row with L

sed 's/the/THE/g' test.txt //Replace all THE thes in THE file with THE

sed 's/o//g' test.txt //Delete all o in the file (replace with empty string)

sed 's/^/#/' test.txt //Insert #: at the beginning of each line

sed '/the/s/^/#/' test.txt //Insert # at the beginning of each line containing the

sed 's/$/EOF/' test.txt //Insert the string EOF at the end of each line

sed '3,5s/the/THE/g' test.txt //Replace all THE thes in lines 3 to 5 with THE

sed '/the/s/o/O/g' test.txt //Replace o in all rows containing the with O

Migrate text

sed '/the/{H;d};$G' test.txt //Migrate the row containing the to the end of the file, {;} for multiple operations

sed '1,5{H;d};17G' test.txt //Transfer lines 1-5 to lines 17

sed '/the/w out.file' test.txt //Save the row containing the as a file out.file

sed '/the/r /etc/hostname' test.txt //After adding the contents of the file / etc/hostname to each line containing the

sed '3aNew' test.txt //Insert a new line after line 3, New

sed '/the/aNew' test.txt //Insert a New line after each line containing the

sed '3aNew1\nNew2' test.txt //Insert multi-line content after line 3, withn in the middle representing newline

AWK command

AWK is a programming language tool for text processing. AWK is similar to shell programming language in many ways, although it has its own grammar and is one of the most powerful data processing engines available in any environment.

Basic structure

awk [option]'mode or condition {edit command}'file 1 file 2 // filter and output matching content

The awk-f script file file 1 file 2// is called from the script and output

Basic use examples

[root@localhost ~]# awk -F: '{print $1,$3}' /etc/passwd
//With ":" as the separator, filter out the contents of the first column and the third column, and output (by default, the separator of the field is a space or tab key).
root 0
bin 1
daemon 2
adm 3

Special built-in variables (can be directly introduced for use)

FS: Specifies a field separator for each line of text, defaulting to a space or tab.

NF: Number of fields in rows currently processed.

NR: The line number (ordinal number) of the row being processed.

$0: The entire line of the row being processed.

$n: The nth field (column n) of the current processing row.

FILENAME: The name of the file being processed.

RS: Data records are separated by default to n, which means one record per action.

Output text by line

[root@localhost ~]# awk 'NR==2,NR==4{print}' /etc/passwd       
//Output the second to fourth lines of text content

[root@localhost ~]# awk -F ":" 'NR==2,NR==4{print $1,$3}' /etc/passwd
//Output the first and third columns of the second to fourth rows
bin 1
daemon 2
adm 3

[root@localhost ~]# awk '(NR==1)||(NR==4){print}' /etc/passwd
//Output the first and fourth lines

Output odd and even lines (in awk, you can use the logical operator "&", "and", "and", "or", "or"! It can also perform simple mathematical operations, such as +, -,*, /,%, ^ for addition, subtraction, multiplication, division, redundancy and multiplication, respectively.

[root@localhost ~]# Awk'NR%2==1 {print}'/etc/passwd// Output odd rows

[root@localhost ~]# Awk'NR%2=== 0 {print}'/etc/passwd//output even rows

Output lines starting with root

[root@localhost ~]# awk '/^root/{print}' /etc/passwd

Statistics of rows ending in / bin/bash

[root@localhost ~]# awk 'BEGIN {x=0} ; /\/bin\/bash$/{x++};END {print x}' /etc/passwd

Number of text paragraphs separated by blank lines:

[root@localhost opt]# vim name.txt
[root@localhost opt]# awk 'BEGIN{RS=":"};END{print NR}' /opt/name.txt
//Statistical rules: encounter key symbols, break lines

Call the w command and use it to count the number of online users:

[root@localhost opt]# awk 'BEGIN {while ("w" | getline) n++;{print n-2}}'

Posted by Buddski on Wed, 09 Oct 2019 04:30:29 -0700