awk of shell three swordsmen
In Linux/UNIX system, awk is a powerful editing tool. It reads input text line by line, searches according to the specified matching mode, formats and outputs qualified content or filters it. It can realize quite complex text operation without interaction. It is widely used in Shell scripts to complete various automatic configuration tasks.
The command format used by awk is as follows, in which single quotation marks plus braces "{}" are used to set the processing actions for data. Awk can process the target file directly, or it can process the target file by reading the script "-f".
awk tool command format
awk option'mode or condition {edit instructions}'file 1 file 2// filter and output the contents of file character conditions
awk -f script file 1, file 2, / / / from the script, edit instructions, filter and output contents.
7Awk prefers to divide a row into multiple "fields" and then process it, and by default the field separator is a space or tab key. The result of awk execution can be printed and displayed by print function. In the process of using awk command, we can use the logical operator "&", "and", "or", "and", "or", "and"." It can also carry out simple mathematical operations, such as +, -,*, /,%, ^ for addition, subtraction, multiplication, division, redundancy and multiplication, respectively.
Find out the user name, user ID, group ID and other columns of / etc/passwd, and execute the following awk command.
[root@localhost ~]# awk -F : '{print $1,$3,$4}' /etc/passwd
root 0 0
bin 1 1
daemon 2 2
adm 3 4
lp 4 7
sync 5 0
shutdown 6 0
halt 7 0
mail 8 12
operator 11 0
games 12 100
ftp 14 50
nobody 99 99
systemd-network 192 192
dbus 81 81
polkitd 999 998
abrt 173 173
libstoragemgmt 998 996
rpc 32 32
colord 997 995
saslauth 996 76
rtkit 172 172
pulse 171 171
chrony 995 991
rpcuser 29 29
nfsnobody 65534 65534
ntp 38 38
tss 59 59
usbmuxd 113 113
geoclue 994 989
qemu 107 107
radvd 75 75
setroubleshoot 993 988
sssd 992 987
gdm 42 42
gnome-initial-setup 991 986
sshd 74 74
avahi 70 70
postfix 89 89
tcpdump 72 72
chen 1000 1000
Awk reads information from input file or standard input, and like sed, information is read line by line. The difference is that awk treats a line in a text file as a record and a part (column) in a row as a field (field) in the record. To manipulate these different fields, awk borrows a location variable-like approach in the shell to represent the different fields in rows (records) in order of $1, $2, $3_. In addition, awk represents the entire line (record) with $0. Different fields are separated by specified characters. The default separator for awk is a space. Awk allows delimiters to be specified on the command line in the form of "-F delimiters".
awk contains several special built-in variables (available directly) as follows:
(1) FS: Specifies a field separator for each line of text, defaulting to a space or tab.
(2) NF: Number of fields in rows currently processed.
(3) NR: The line number (ordinal number) of the row being processed.
(4) $0: The entire line of the row being processed.
(6) FILENAME: The name of the file being processed.
(7) RS9: Data records are segregated by default to \n, i.e. one record per action.
I. awk usage
(1) Output of all content, equivalent to cat chen.txt,
[root@localhost ~]# awk '{print $0}' chen.txt
[root@localhost ~]# The awk'{print}'chen. TXT // / command has the same effect as the one above.
#version=DEVEL
#System authorization information
aulth --enableshadow --passalgo=sha512
#Use CDROM installation media
cdlrom.
thethethe.(2) Output lines 1-3
[root@localhost ~]# Awk'NR== 1, NR== 3 {print}'chen.txt// This command has the same effect as the following command
[root@localhost ~]# awk '(NR>=1)&&(NR<=3){print}' chen.txt
#version=DEVEL
#System authorization information
aulth --enableshadow --passalgo=sha512(3) Output of all odd lines
[root@localhost ~]# awk '(NR%2)==1{print}' chen.txt
Use CDROM installation media
cdrom.
thethethe.
THE
THEASDHAS
Use graphical install.
graphical.
best
test
ASSDJFXYxyzC
AxyzxyzxyzC
keyboard --vckeymap=cn --xlayouts='cn'
System language(4) Output the contents of all even rows
[root@localhost ~]# awk '(NR%2)==0{print}' chen.txt
Use graphical install.
graphical.
best
test
ASSDJFXYxyzC
AxyzxyzxyzC
keyboard --vckeymap=cn --xlayouts='cn'
System language
lang zh_CN.UTF-8
Network information
network --bootproto=dhcp --device=ens33 --onboot=off --ipv6=auto --no-activate
network --hostname=localhost.localdomain
Root password
rootpw --iscrypted $6$L.egxzosoP/0k9Nj$wna7vPXZjeH0jFcNZUymYKF8ySXq5HxQuvxTFxIpEAAxuDj7MQJtXBds5E0LxAftI1H5JbJuYpN44d5n6t1AZ.
System services(5) Output the contents of all even rows
awk '/^The/{print}' chen.txt
THE
THEASDHAS(6) Output lines ending with limit.
[root@localhost ~]# awk '/limit.$/{print}' chen.txt
Use graphical install limit.(7) The number of rows ending in / bin/bash is equivalent to greo-c "/ bin/bash" chen.txt.
[root@localhost ~]# awk 'BEGIN {x=0} ; /\/bin\/bash$/{x++};END{print x}' /etc/passwd
(8) count the number of paragraphs separated by the
[root@localhost ~]# awk 'BEGIN {RS="the"};END{print NR}' chen.txt
(2) Text output by field
[root@localhost ~]# awk -F : '{print $3}' /etc/passwd
0
1
2
3
4
5
6
7
8
9
10
11(1) Output the first and third fields in each line
[root@localhost ~]# awk -F : '{print $1,$3}' /etc/passwd
root 0
bin 1
daemon 2
adm 3
lp 4
sync 5
shutdown 6
halt 7
mail 8
operator 11
games 12
ftp 14
nobody 99
systemd-network 192(2) shadow records of users whose password is "*"
[root@localhost ~]# awk -F : '$2== "*"{print}' /etc/shadow
bin:*:17110:0:99999:7:::
daemon:*:17110:0:99999:7:::
adm:*:17110:0:99999:7:::
lp:*:17110:0:99999:7:::
sync:*:17110:0:99999:7:::
shutdown:*:17110:0:99999:7:::
halt:*:17110:0:99999:7:::
mail:*:17110:0:99999:7:::
operator:*:17110:0:99999:7:::
games:*:17110:0:99999:7:::(3) The output is colon-separated and the first field of the row containing / bash in the seventh field
[root@localhost ~]# awk '($1~"nfs")&&(NF=7){print $1,$2}' /etc/passwd
nfsnobody:x:65534:65534:Anonymous NFS(4) Output field 7 is neither / bin/bash nor all rows of / sbin/nologin
[root@localhost ~]# awk -F : '($7!="/bin/bash")&&($7!="/sbin/nologin")' /etc/passwd sync:x:5:0:sync:/sbin:/bin/sync shutdown:x:6:0:shutdown:/sbin:/sbin/shutdown halt:x:7:0:halt:/sbin:/sbin/halt
(2) invoking shell commands with double quotation marks through pipes
Calling the wc-l command counts the number of users using bash, which is equivalent to grep-c "bash$"/etc/passwd
[root@localhost ~]# awk -F : '/bash$/{print | "wc -l"}' /etc/passwd
2(1) Call the w command and use it to count the number of online users
[root@localhost ~]# awk 'BEGIN {while ("w" | getline) n++ ;{print n-2}}'
2
(2) Call hostname and output the current hostname
[root@localhost ~]# awk 'BEGIN {"hostname" | getline ; print $0}'
localhost.localdomainIII. sort Tool
Sort is a tool for sorting the contents of files in units of behavior, and it can also be sorted according to different data types. For example, data and characters play different cards. The syntax of the sort command is "sort [option] parameter", and the commonly used options include the following.
(1) - f: ignore case;
(2) - b: Ignore the space in front of each line;
(3) - M: sort by month;
(4) - n: sort by number;
(5) - r: reverse ordering;
(6) - u: the same as uniq, which means that the same data only shows one row;
(7) - t: Specify the delimiter and use the [Tab] key by default.
(8) - O < output file >: transfer the sorted results to the specified file;
(9) - k: Specifies the sorting area.
1: Sort the accounts in the / etc/passwd file.
The sorting rules are alphabetical at the beginning, and small to large at the second letter if the beginning is the same.
[root@localhost ~]# sort /etc/passwd abrt:x:173:173::/etc/abrt:/sbin/nologin adm:x:3:4:adm:/var/adm:/sbin/nologin avahi:x:70:70:Avahi mDNS/DNS-SD Stack:/var/run/avahi-daemon:/sbin/nologin bin:x:1:1:bin:/bin:/sbin/nologin chen:x:1000:1000:chen:/home/chen:/bin/bash chrony:x:995:991::/var/lib/chrony:/sbin/nologin colord:x:997:995:User for colord:/var/lib/colord:/sbin/nologin daemon:x:2:2:daemon:/sbin:/sbin/nologin dbus:x:81:81:System message bus:/:/sbin/nologin ftp:x:14:50:FTP User:/var/ftp:/sbin/nologin games:x:12:100:games:/usr/games:/sbin/nologin
(1) reverse sort the third column in the / etc/passwd file
[root@localhost ~]# sort -t : -rk 3 /etc/passwd nobody:x:99:99:Nobody:/:/sbin/nologin polkitd:x:999:998:User for polkitd:/:/sbin/nologin libstoragemgmt:x:998:996:daemon account for libstoragemgmt:/var/run/lsm:/sbin/nologin colord:x:997:995:User for colord:/var/lib/colord:/sbin/nologin saslauth:x:996:76:Saslauthd user:/run/saslauthd:/sbin/nologin chrony:x:995:991::/var/lib/chrony:/sbin/nologin geoclue:x:994:989:User for geoclue:/var/lib/geoclue:/sbin/nologin setroubleshoot:x:993:988::/var/lib/setroubleshoot:/sbin/nologin sssd:x:992:987:User for sssd:/:/sbin/nologin gnome-initial-setup:x:991:986::/run/gnome-initial-setup/:/sbin/nologin postfix:x:89:89::/var/spool/postfix:/sbin/nologin dbus:x:81:81:System message bus:/:/sbin/nologin
(2) Sort the third column in the / etc/passwd file and save the output to the user.txt file
[root@localhost ~]# sort -t : -k 3 /etc/passwd -o chench.txt [root@localhost ~]# cat chench.txt root:x:0:0:root:/root:/bin/bash chen:x:1000:1000:chen:/home/chen:/bin/bash qemu:x:107:107:qemu user:/:/sbin/nologin operator:x:11:0:operator:/root:/sbin/nologin usbmuxd:x:113:113:usbmuxd user:/:/sbin/nologin bin:x:1:1:bin:/bin:/sbin/nologin games:x:12:100:games:/usr/games:/sbin/nologin ftp:x:14:50:FTP User:/var/ftp:/sbin/nologin pulse:x:171:171:PulseAudio System Daemon:/var/run/pulse:/sbi
(4) uniq tools
Uniq tools are commonly used in Linux systems in conjunction with sort commands to report or ignore duplicate lines in files. The specific command grammar format is: uniq [option] parameter. Common options include the following.
(1) - c: counting;
(2) - d: only duplicate rows are displayed;
(3) - u: Shows only rows that appear once;
1: Delete duplicate lines in a file.
[root@localhost ~]# cat a.txt centos7 6.2 centos7 6.2 centos7 6.2 centos7 6.2 centos7 6.2 centos7 6.6 centos7 6.2 centos7 6.2 centos7 6.3 centos7 6.5 linux 1 linux 2 linux 3 linux 4 linux 5 linux 6
[root@localhost ~]# uniq a.txt centos7 6.2 centos7 6.6 centos7 6.2 centos7 6.3 centos7 6.5 linux 1 linux 2 linux 3 linux 4 linux 5 linux 6
uniq tool deletes duplicates but only continuous duplicates
(2) Delete duplicate lines in a file and display the number of duplicates at the beginning of the line
[root@localhost ~]# uniq -c a.txt
1
5 centos7 6.2
1 centos7 6.6
2 centos7 6.2
1 centos7 6.3
1 centos7 6.5
1 linux 1
1 linux 2
1 linux 3
1 linux 4
1 linux 5
1 linux 6
1 (3) Find duplicate lines in the test file.
[root@localhost ~]# uniq -d a.txt centos7 6.2 centos7 6.2