1. Practical Small Commands
1.1,cat
The cat command is mainly used to view file contents, create files, merge files, append file contents and other functions.
Common Options
-n: Display line numbers when viewing text
-b: Display line numbers when viewing text, excluding blank lines
-E: Displays whether or not a line break occurs, ending with'$'to indicate a line break
-A: Shows if TAB has keys, and a'^I'indicates a TAB
-s: compress adjacent empty behavior one
(1)Test File
[root@kube-master ~]# cat F1
123
456
789
123123
(2)-n :Display line numbers when viewing text
[root@kube-master ~]# cat -n F1
1 123
2 456
3 789
4 123123
5
6
(3)-b :Display line numbers when viewing text, excluding blank lines
[root@kube-master ~]# cat -b F1
1 123
2 456
3 789
4 123123
(4)-E :Show whether to wrap, ending with"$"End indicates a line break
[root@kube-master ~]# cat -E F1
123$
456$
789 $
123123 $
$
$
(5) -A :display TAB Is there a key, one"^I"Represents a TAB
[root@kube-master ~]# cat -A F1
123$
456$
789 $
123123^I^I $
$
$
(6)-s :Compress adjacent empty behavior one
[root@kube-master ~]# cat -s F1
123
456
789
123123
(7)Generate Files
[root@kube-master ~]# cat > F2
test
ctrl + d Sign out
[root@kube-master ~]# cat F2
test
(9)Merge Files
[root@kube-master ~]# cat F1 F2 > F3
[root@kube-master ~]# cat F3
123
456
789
123123
test1.2,tac
View text upside down
(1)View text upside down [root@kube-master ~]# tac F1 123123 789 456 123
1.3,rev
Reverse display
(1)Reverse display content [root@kube-master ~]# echo "abcd" | rev dcba [root@kube-master ~]# rev < /etc/fstab # batsf/cte/ # 9102 24:54:90 13 naJ uhT no adnocana yb detaerC # # 'ksid/ved/' rednu deniatniam era ,ecnerefer yb ,smetsyselif elbisseccA # ofni erom rof )8(diklb ro/dna )8(tnuom ,)8(sfdnif ,)5(batsf segap nam eeS # # 0 0 stluafed sfx / toor-sotnec/reppam/ved/ 0 0 stluafed sfx toob/ 5fe0eaa3ec3f-6eb9-5234-0aee-21b8c81d=DIUU 0 0 stluafed paws paws paws-sotnec/reppam/ved/
1.4,head
Display the first few lines or bytes
Common Options
-#: # denotes a number and the first # rows displayed
-n #: # denotes a number, showing the first few # rows, similar to -# above
-c #: # denotes a number, indicating the first few bytes of data displayed
(1)-# :#Number, top of display#That's ok [root@kube-master ~]# head -5 /etc/passwd root:x:0:0:root:/root:/bin/bash bin:x:1:1:bin:/bin:/sbin/nologin daemon:x:2:2:daemon:/sbin:/sbin/nologin adm:x:3:4:adm:/var/adm:/sbin/nologin lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin (2)-n # :#Number, top of display#Row, follow above-#Similar [root@kube-master ~]# head -n 5 /etc/passwd root:x:0:0:root:/root:/bin/bash bin:x:1:1:bin:/bin:/sbin/nologin daemon:x:2:2:daemon:/sbin:/sbin/nologin adm:x:3:4:adm:/var/adm:/sbin/nologin lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin (3)-c # :#Represents a number, showing the first few bytes of data [root@kube-master ~]# head -c 5 /etc/passwd root:
1.5,tail
Show the last few lines or bytes, of course tail is richer to use
Common Options
-#: Show lines after last count
-n #: like -#, show the last #line
-c #: Display the last #bytes
-f: Display the last line in real time
-F: Display the last line in real time, and when the file does not exist, the file does not exist
(1)-# :Show the last#That's ok [root@kube-master ~]# tail -5 /etc/passwd gitlab-prometheus:x:993:990::/var/opt/gitlab/prometheus:/bin/sh jenkins:x:992:989:Jenkins Automation Server:/var/lib/jenkins:/bin/false ntp:x:38:38::/etc/ntp:/sbin/nologin nginx:x:991:988:Nginx web server:/var/lib/nginx:/sbin/nologin mysql:x:990:987::/home/mysql:/sbin/nologin (2)-n # :with-#Same, after showing the last#That's ok [root@kube-master ~]# tail -n 5 /etc/passwd gitlab-prometheus:x:993:990::/var/opt/gitlab/prometheus:/bin/sh jenkins:x:992:989:Jenkins Automation Server:/var/lib/jenkins:/bin/false ntp:x:38:38::/etc/ntp:/sbin/nologin nginx:x:991:988:Nginx web server:/var/lib/nginx:/sbin/nologin mysql:x:990:987::/home/mysql:/sbin/nologin (3)-c # :Show the last#byte [root@kube-master ~]# tail -c 10 /etc/passwd n/nologin (4) -f :Show last line in real time, The default prints the last 10 lines and monitors the last one, displaying the data in real time when one line is added [root@kube-master ~]# tail -f F1 123 456 789 123123 [root@kube-master ~]# echo "testline" >> F1 [root@kube-master ~]# tail -f F1 123 456 789 123123 testline //Real-time monitoring shows only the last line [root@squid ~]# tail -n 0 -f /etc/fstab tesst (5) -F :Show the last line in real time, show the file does not exist when it does not exist [root@kube-master ~]# tail -F F1 123 456 789 123123 testline [root@kube-master ~]# rm -f F1 [root@kube-master ~]# tail -F F1 123 456 789 123123 testline tail: 'F1' has become inaccessible: No such file or directory
1.6,tailf
Like tail-f, but with higher performance, output content only when the file changes, falsely monitor the disk, save power, and reduce disk read and write.
[root@kube-master ~]# tailf /var/log/nginx/access.log
1.7,tr
Delete, replace, etc. text for standard input
Common Options
-d: Delete specified content from standard input
-s: Reduces the number of consecutive repeated characters to a specified single character
(1)Test File
[root@kube-master ~]# cat > F1
123 34545
asdasdas
123 7878
ctrl + d Sign out
[root@kube-master ~]# cat F1
123 34545
asdasdas
123 7878
(2)Delete file contents, delete with"123"Content of the word
[root@kube-master ~]# tr -d "123" < F1
4545
asdasdas
7878
(3)Replace lowercase letters with uppercase letters, standard output generation F3 file
[root@kube-master ~]# tr "a-z" "A-Z" < F1 > F3
[root@kube-master ~]# cat F3
123 34545
ASDASDAS
123 7878
(4)Compress multiple blanks into one column
[root@kube-master ~]# df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mapper/centos-root 120527360 54045264 66482096 45% /
devtmpfs 1919508 0 1919508 0% /dev
tmpfs 1931784 0 1931784 0% /dev/shm
tmpfs 1931784 11968 1919816 1% /run
tmpfs 1931784 0 1931784 0% /sys/fs/cgroup
/dev/sda1 1038336 145300 893036 14% /boot
tmpfs 386360 0 386360 0% /run/user/0
[root@kube-master ~]# df -h | tr -s " "
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/centos-root 115G 52G 64G 45% /
devtmpfs 1.9G 0 1.9G 0% /dev
tmpfs 1.9G 0 1.9G 0% /dev/shm
tmpfs 1.9G 12M 1.9G 1% /run
tmpfs 1.9G 0 1.9G 0% /sys/fs/cgroup
/dev/sda1 1014M 142M 873M 14% /boot
tmpfs 378M 0 378M 0% /run/user/01.8,cut
Parallel Cutting
Common Options
-d: what is the separator
-c: Which column to remove
--output-delimeter="#": what is the delimiter used to output content
(1)with/etc/passwd For the top 5 behavior columns [root@kube-master ~]# cat /etc/passwd | head -5 root:x:0:0:root:/root:/bin/bash bin:x:1:1:bin:/bin:/sbin/nologin daemon:x:2:2:daemon:/sbin:/sbin/nologin adm:x:3:4:adm:/var/adm:/sbin/nologin lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin (2)Remove column 1, user name [root@kube-master ~]# cut -d : -f 1 /etc/passwd | head -5 root bin daemon adm lp (3)Remove columns 1 and 3 [root@kube-master ~]# cut -d : -f 1,3 /etc/passwd | head -5 root:0 bin:1 daemon:2 adm:3 lp:4 (4)Remove columns 1 to 3 [root@kube-master ~]# cut -d : -f 1-3 /etc/passwd | head -5 root:x:0 bin:x:1 daemon:x:2 adm:x:3 lp:x:4 (5)Remove columns 1 through 3 to|Number as divider output [root@kube-master ~]# cut --output-delimiter="|" -d : -f 1-3 /etc/passwd | head -5 root|x|0 bin|x|1 daemon|x|2 adm|x|3 lp|x|4
1.9,paste
Merge the two files horizontally, merge the first line of the first file and the first line of the second file by default, and so on
Common Options
-d: Separator for merging intermediate output, default to space
-s: Merge all lines of each file into one line, merge all contents of the first file into the first line, and merge all contents of the second file into the second line
(1) As you all know if you have used ansible, if you use ansible to send SSH public keys in bulk, you need to use / etc/ansible/hosts
Define the SSH user and password for each host. Generally, when managing a host, we record information about the password class in xls.
This is where it comes in handy (if there are 100 hosts with different passwords) to copy the IP column of the xls record to the F1 file, not excluding
windwos'^M'return, clear with dos2unix, following F2, F3 files are the same, of course ansible_ssh_user,
ansible_ssh_pss is not an xls record. You need to merge F1 first. I won't talk about F2, as you will see below.
File 1
[root@kube-master ~]# cat F1
[host]
10.1.1.1
10.1.1.2
10.1.1.3
10.1.1.4
File 2
[root@kube-master ~]# cat F2
ansible_ssh_user=root
ansible_ssh_user=root
ansible_ssh_user=root
ansible_ssh_user=root
File 3
[root@kube-master ~]# cat F3
ansible_ssh_pass=123
ansible_ssh_pass=456
ansible_ssh_pass=789
ansible_ssh_pass=910
Composite files, F1,F2,F3 in sequence
[root@kube-master ~]# paste F1 F2 F3
[host]
10.1.1.1 ansible_ssh_user=root ansible_ssh_pass=123
10.1.1.2 ansible_ssh_user=root ansible_ssh_pass=456
10.1.1.3 ansible_ssh_user=root ansible_ssh_pass=789
10.1.1.4 ansible_ssh_user=root ansible_ssh_pass=910
(2) Effect of adding -d option
[root@kube-master ~]# paste -d : F1 F2 F3
[host]::
10.1.1.1:ansible_ssh_user=root:ansible_ssh_pass=123
10.1.1.2:ansible_ssh_user=root:ansible_ssh_pass=456
10.1.1.3:ansible_ssh_user=root:ansible_ssh_pass=789
10.1.1.4:ansible_ssh_user=root:ansible_ssh_pass=910
(3)-s: Merge all lines of each file into one line, merge all contents of the first file into the first line, and merge all contents of the second file into the second line
[root@kube-master ~]# paste -s F1 F2 F3
[host] 10.1.1.1 10.1.1.2 10.1.1.3 10.1.1.4
ansible_ssh_user=root ansible_ssh_user=root ansible_ssh_user=root ansible_ssh_user=root
ansible_ssh_pass=123 ansible_ssh_pass=456 ansible_ssh_pass=789 ansible_ssh_pass=9102.0,wc
-l: how many rows are counted
-w: count how many words
-c: how many bytes are counted
-m: how many characters are counted
-L: The length of the longest line in the display file
(1)Take this as a column [root@kube-master ~]# cat F1 [host] 10.1.1.1 10.1.1.2 10.1.1.3 10.1.1.4 (2)Defaults to 5 lines, 5 words, 43 bytes without any options [root@kube-master ~]# wc F1 5 5 43 F1 (3)-l :Count how many rows [root@kube-master ~]# cat F1 | wc -l 5 (4)-w :Count how many words [root@kube-master ~]# cat F1 | wc -w 5 (5)-c :Count how many bytes [root@kube-master ~]# cat F1 | wc -c 43 (6)-m :Count how many characters [root@kube-master ~]# cat F1 | wc -m 43 -L :Show the length of the longest line in the file [root@kube-master ~]# cat F1 | wc -L 8
2.1,sort
Sort or de-repeat content, by default, by the first character in each line
Common Options
-t: Specify the separator
-k: Specifies which column to sort, usually in conjunction with-t
-n: sorted by number, sorted by ascending by default, from smallest to largest 1,2,3,4,5,6
-r: reverse order
-u: Sort removes duplicate rows, and adjacent and non-adjacent duplicate rows are also deleted
-f: Ignore case
(1)with/etc/passwd For the top 5 behavior columns [root@kube-master ~]# cat /etc/passwd | head -5 root:x:0:0:root:/root:/bin/bash bin:x:1:1:bin:/bin:/sbin/nologin daemon:x:2:2:daemon:/sbin:/sbin/nologin adm:x:3:4:adm:/var/adm:/sbin/nologin lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin (2)By default, no option is added to sort the first character on each line. adm，bin，daemon，lp，root [root@kube-master ~]# cat /etc/passwd | head -5 | sort adm:x:3:4:adm:/var/adm:/sbin/nologin bin:x:1:1:bin:/bin:/sbin/nologin daemon:x:2:2:daemon:/sbin:/sbin/nologin lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin root:x:0:0:root:/root:/bin/bash (3)Specify column 3 UID Sort, and sort by number, in ascending order [root@kube-master ~]# cat /etc/passwd | head -5 | sort -t : -k 3 -n root:x:0:0:root:/root:/bin/bash bin:x:1:1:bin:/bin:/sbin/nologin daemon:x:2:2:daemon:/sbin:/sbin/nologin adm:x:3:4:adm:/var/adm:/sbin/nologin lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin (4)-r inverted [root@kube-master ~]# cat /etc/passwd | head -5 | sort -t : -k 3 -n -r lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin adm:x:3:4:adm:/var/adm:/sbin/nologin daemon:x:2:2:daemon:/sbin:/sbin/nologin bin:x:1:1:bin:/bin:/sbin/nologin root:x:0:0:root:/root:/bin/bash (5)Remove duplicate lines [root@kube-master ~]# cat F1 11 11 2 3 4 5 22 22 [root@kube-master ~]# cat F1 | sort -u 11 2 22 3 4 5
2.2,uniq
Deduplicate rows or count duplicate rows
Common Options
-c: row statistics for continuous and discontinuous repetitions
-u: Show only discontinuous duplicate rows
-d: Show only adjacent and consecutive rows
(1)Take this as an example [root@kube-master ~]# cat F1 11 11 2 3 4 5 22 22 (2)-c :Row statistics for continuous and discontinuous repetitions [root@kube-master ~]# cat F1 | uniq -c 2 11 1 2 1 3 1 4 1 5 2 22 (3) -u :Show only discontinuous and repeated rows [root@kube-master ~]# cat F1 | uniq -u 2 3 4 5 (4)-d :Show only adjacent and consecutive rows [root@kube-master ~]# cat F1 | uniq -d 11 22
2.3, Other
(1) 30 bytes before generating random numbers base64 openssl rand -base64 30 | head -c 30 Hexadecimal openssl rand -hex 30 | head -c 30 Randomly generate 30 numbers tr -dc '[[:digit:]]' < /dev/urandom | head -c 30 Randomly generate 30 lower case letters tr -dc '[[:lower:]]' < /dev/urandom | head -c 30 Randomly generate 30 uppercase letters tr -dc '[[:upper:]]' < /dev/urandom | head -c 30
2. Simple practice
1.1,UV
UV is the amount of user access, and the general statistics are measured by the sum of individual IP s that do not repeat
(1)With this nginx Log file as an example [root@kube-master ~]# cat access.log | head -1 192.168.126.111 - - [20/Feb/2019:15:35:16 +0800] "GET / HTTP/1.1" 200 1482 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.109 Safari/537.36" (2)First take out the first example, then sort it, repeat it after sorting, and then count [root@kube-master ~]# cat access.log | cut -d " " -f 1 | sort -n | sort -u | wc -l 4 [root@kube-master ~]# cat access.log | cut -d " " -f 1 | sort -n | uniq | wc -l 4
1.2,PV
PV is the amount of page visits, how many visits per page
(1)With this nginx Log file as an example [root@kube-master ~]# cat access.log | head -1 192.168.126.111 - - [20/Feb/2019:15:35:16 +0800] "GET / HTTP/1.1" 200 1482 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.109 Safari/537.36" (2)Remove first URI Column 7, then sorted by character, and then uniq -c Statistics duplicate or non-duplicate items, then sort them in ascending order to find out which pages are most visited URI [root@kube-master ~]# cat access.log | cut -d " " -f 7 | sort | uniq -c | sort -r -n 60 /favicon.ico 3 /zabbix 3 /lnmp.gif 3 / 1 /zabbix/api_jsonrpc.php
Step by step!!!!!!!!!!!!!!!!!!!!!!
In fact, it is very simple!!!!!!!!!!!!!!!!!!!!!!!