awk tools

In Linux/UNIX system, awk is a powerful editing tool. It reads input text line by line, searches according to the specified matching mode, formats and outputs qualified content or filters it. It can realize quite complex text operation without interaction. It is widely used in Shell scripts to complete various automatic configuration tasks.

1. Common usage of awk

• The command format used by awk is as follows. Single quotation marks plus braces "{}" are used to set the processing actions for data. Awk can process the target file directly, or it can process the target file by reading the script "-f".

  awk option'mode or condition {editing instructions}'file 1 file 2 // filters and outputs the contents of file character conditions
   awk -f script file file 1 file 2 / / from the script to edit instructions, filter and output content

• The sed command is often used for processing an entire line, while awk prefers to divide a line into multiple "fields" and then process it, and by default the field separator is a space or tab key. The result of awk execution can be printed and displayed by print function. In the process of using awk command, we can use the logical operator'&','and','|'to indicate'or','non', and simple mathematical operations such as +, -,*, /,%, ^ to indicate addition, subtraction, multiplication, division, surplus and multiplication, respectively.

• In Linux system, / etc/passwd is a very typical formatted file. Each field is separated by ":" as a separator. Most of the log files in Linux system are formatted files. Extracting relevant information from these files is one of the daily work of operation and maintenance. If you need to find out the user name, user ID, group ID and other columns of / etc/passwd, execute the following awk command.

  [root@localhost ~]# awk -F ':' '{print $1,$3,$4}' /etc/passwd
  root 0 0
  bin 1 1
  daemon 2 2
  adm 3 4
  ...//Eliminate part of the content.
  //Awk reads information from input file or standard input, and like sed, information is read line by line. The difference is that awk treats a line in a text file as a record and a part (column) in a row as a field (field) in the record. To manipulate these different fields, awk borrows a location variable-like approach in the shell to represent the different fields in rows (records) in order of $1, $2, $3_. In addition, awk represents the entire line (record) with $0. Different fields are separated by specified characters. The default separator for awk is a space. Awk allows delimiters to be specified on the command line in the form of "-F delimiters".

• awk contains several special built-in variables (available directly) as follows

2. Examples of usage

1) Output text by line

[root@localhost opt]# Awk'{print}'bbb.txt// Output all content, equivalent to cat bbb.txt
this is
the wood
wood
wod
the wood
this is test

[root@localhost opt]# Awk'{print $0}'bbb.txt// Output all content, equivalent to cat bbb.txt
this is
the wood
wood
wod
the wood
this is tes

[root@localhost opt]# Awk'NR== 1, NR== 3 {print}'bbb.txt// Output Lines 1-3
this is
the wood
wood

[root@localhost opt]# Awk'(NR >= 1) & (NR<= 3) {print}'bbb.txt// / Output Line 1-3
this is
the wood
wood

[root@localhost opt]# Awk'NR== 1 | | NR== 3 {print}'bbb.txt// Output Line 1, Line 3
this is
wood

[root@localhost opt]# Awk'(NR%2)=== 1 {print}'bbb.txt// Outputs the contents of all odd rows
this is
wood
the wood

[root@localhost opt]# Awk'(NR%2)=== 0 {print}'bbb.txt// Outputs all even lines
the wood
wod
this is test

[root@localhost opt]# Awk'/^ the /{print}'bbb.txt // / Outputs all lines starting with the
the wood
the wood

[root@localhost opt]# Awk'/ wood $/{print}'bbb.txt // / Outputs all lines ending with wood
the wood
wood
the wood

[root@localhost opt]# awk 'BEGIN {x=0};/\/bin\/bash$/{x++};END {print x}' /etc/passwd
2                           //Statistically, the number of rows ending in / bin/bash is equivalent to grep - C "/ bin/bash $"/ etc / passwd

[root@localhost opt]# Awk'BEGIN {RS="}; END {print NR}'httpd.txt// / Statistics of text paragraphs separated by blank lines
38

2) Output text by field

[root@localhost opt]# vim bbb.txt//Edit text content
this is txt
the wood aaa
wood is bbb                      //Editor adds content
wod is ccc
the wood AAA
this is test
~                                                                            
~                                                                                                 
:wq                              //Save exit
[root@localhost opt]# Awk'{print $3}'bbb.txt// Output the third field in each line (separated by spaces or tabs)
txt
aaa
bbb
ccc
AAA
test

[root@localhost opt]# Awk'{print $1, $3}'bbb.txt// Outputs the first and third fields in each line
this txt
the aaa
wood bbb
wod ccc
the AAA
this test

[root@localhost opt]# Awk-F ":"'$2="*"{print}'/etc/shadow// Shadow record of user with output password of "*"
bin:*:17110:0:99999:7:::
daemon:*:17110:0:99999:7:::
adm:*:17110:0:99999:7:::
lp:*:17110:0:99999:7:::
sync:*:17110:0:99999:7:::
shutdown:*:17110:0:99999:7:::
halt:*:17110:0:99999:7:::
mail:*:17110:0:99999:7:::
operator:*:17110:0:99999:7:::
games:*:17110:0:99999:7:::
ftp:*:17110:0:99999:7:::
nobody:*:17110:0:99999:7:::

[root@localhost opt]# awk 'BEGIN{FS=":"};$2=="*";END{print}' /etc/shadow 
bin:*:17110:0:99999:7:::                        //shadow records of users whose password is "*"
daemon:*:17110:0:99999:7:::
adm:*:17110:0:99999:7:::
lp:*:17110:0:99999:7:::
sync:*:17110:0:99999:7:::
shutdown:*:17110:0:99999:7:::
halt:*:17110:0:99999:7:::
mail:*:17110:0:99999:7:::
operator:*:17110:0:99999:7:::
games:*:17110:0:99999:7:::
ftp:*:17110:0:99999:7:::
nobody:*:17110:0:99999:7:::
named:!!:18178::::::

[root@localhost opt]# awk -F ":" '$7~"/bash"{print $1}' /etc/passwd
root                               //The output is colon-separated and the first field of the row containing / bash in the seventh field 
sun

[root@localhost opt]# awk '($1~"nfs")&&(NF==8){print $1,$2}' /etc/services 
nfs 2049/tcp                          //The output contains eight fields and the first field contains the first and second fields of rows with nfs
nfs 2049/udp
nfs 2049/sctp
netconfsoaphttp 832/tcp
netconfsoaphttp 832/udp
netconfsoapbeep 833/tcp
netconfsoapbeep 833/udp

[root@localhost opt]# awk -F ":" '($7!="/bin/bash")&&($7!="/sbin/nologin"){print}' /etc/passwd
sync:x:5:0:sync:/sbin:/bin/sync         //Output field 7 is neither / bin/bash nor all rows of / sbin/nologin 
shutdown:x:6:0:shutdown:/sbin:/sbin/shutdown
halt:x:7:0:halt:/sbin:/sbin/halt

3) Call Shell commands through pipes and double quotation marks

[root@localhost opt]# awk -F : '/bash/{print | "wc -l"}' /etc/passwd
2                    //Calling the wc-l command counts the number of users using bash, which is equivalent to grep-c "bash$"/etc/passwd

[root@localhost opt]# awk 'BEGIN {while ("w" | getline) n++ ; {print n-2}}'
1     //Call the w command and use it to count the number of online users. Use the while loop to match the number of output lines of the w command. Gettline shows only the number of lines, n-2 subtracts the first two lines.

[root@localhost opt]# awk 'BEGIN { "hostname" | getline ; print $0}'  
localhost.localdomain           //Call hostname and output the current hostname