sed and awk of shell

Keywords: shell


sed editor

sed common options

sed common operations:

Basic Usage

sed script format


Search for alternatives

Change configuration file  


   Basic usage

Get IP address  

Common built-in variables in awk

sed editor

sed is a flow editor that edits the data flow based on pre provided – group rules before the editor processes the data.
The sed editor can process the data in the data stream according to commands, which are either input from the command line or stored in a command text file.

sed workflow mainly includes three processes: reading, executing and displaying:
● read: sed reads a line from the input stream (file, pipeline, standard input) and stores it in the temporary buffer (also known as pattern space)
● execution: by default, all sed commands are executed sequentially in the mode space. Unless the address of the line is specified, the SED command will be executed successively on all lines.
● display: send the modified content to the output stream. After sending data, the mode space will be cleared. Before all file contents are processed, the above process will be repeated until all contents are processed.

Before all file contents are processed, the above process will be repeated until all contents are processed.
Note: by default, all sed commands are executed in the mode space, so the input file will not change unless the output is stored by redirection.

Command format:
sed -e 'operation' File 1 file 2...
sed -n -e 'operation' File 1 file 2 ...
sed -f Script file 1 file 2 ...
sed -i -e 'operation' File 1 file 2 ... 

sed -e '{
Operation 1
 Operation 2
}' File 1 file 2 ...

sed common options

-e or--expression=: Indicates that the input text file is processed with the specified command. Only one operation command can be omitted. One-When executing multiple operation commands
-f or--file=:Indicates that the input text file is processed with the specified script file.
-h or--help: Displays help.
-n,--quiet or silent: prohibit:sed Editor output, but can work with p command- -Use to complete the output.
-i: Modify the target text file directly.

sed common operations:

s:Replace, replace the specified character.
d:Delete, deletes the selected row.
a: Add, add one below the current line-Line specifies the content.
i: Insert, inserts a row of specified content above the selected row.
c:Replace to replace the selected row with the specified content.
y:For character conversion, the length of characters before and after conversion must be the same.
p:Print. If a row is specified at the same time, it means that the specified row is printed:If you do not specify a line, it means that all contents are printed:If there are non printed characters, use ASCII Code output. It is usually associated with“-n"Option one-Start using.
=: Print line numbers.
1(a lowercase letter L):Print text and non printable in the data stream ASCII character(Such as Terminator $,Tab\t)

Basic Usage

Print the input by default

view file contents

  Support redirection

Supports pipe characters

sed script format

'address + command' composition

1. No address: process the full text (such as line number)

2. Single address:
   #: Specified line, $: last line
   / Pattern /: each line that can be matched by the pattern here

3. Address range:
   #,#     # From # line to # line, 3,6 from line 3 to line 6
   #,+#   # From # line to + # line, 3, + 4 means from line 3 to line 7
   / pat1/,/pat2/     The line between the first regular expression and the second regular expression
   #,/ pat/   From # No. 1 behavior until pat is found  
   / pat/,#   Until the # pat number is found

4. Step:~
      1 ~ 2 odd rows
      2 ~ 2 even rows


With automatic printing function, p print again

-The n option turns off automatic printing

  Display line 3 directly

Display the second line directly

  regular expression           Automatic printing needs to be turned off, otherwise all will be printed

Print the line containing root   / Root (what needs to be matched) / P (print)   file name  

Same function as grep root /etc/passwd

  Display range   Line number              Display range

  Add 4 lines after 3

  You can match lines between two regular expressions

Displays the line between b and f

Start looking for the beginning of b and continue to find the beginning of f
Then find the beginning of b again. As soon as you find the beginning of f, it will be displayed without the beginning of f
Repeat cycle

Find logs between what time and what time  

Odd even representation  

Modify file content, often used to modify configuration files
-i   And  - i.bak
Modify file   Back up files before modifying them  

-a added  

Search for alternatives

s/pattern/string/Modifier find replace,Other separators are supported, which can be in other forms: s@@@,s###
Replace modifier:
g Intra row global replacement
p Displays the rows that have been replaced successfully
w   /PATH/FILE Save the successfully replaced line to a file
I,i   ignore case

Change configuration file  


In Linux/UNIX systems, awk is a powerful editing tool. It reads the input text line by line. By default, it is separated by a space or tab key, and executes editing commands according to patterns or conditions. Awk prefers to divide a line into multiple fields and then process them. The reading of awk information is also to find the matching pattern specified line by line to find the matching pattern that meets the conditions The content can be formatted and output or filtered, which can realize quite complex text operations without interaction. It is widely used in Shell scripts to complete various automatic configuration tasks.

Working principle: as mentioned earlier, the sed command is often used to process a whole line, while awk prefers to divide a line into multiple "fields" and then process them. By default, the field separator is a space or tab key. The awk execution results can be printed and displayed through the print function.


awk [option] 'Mode condition{operation}' File 1 file 2....
awk -f|-v Script file 1 file 2.....

Unspecified means empty

/1/2/Represents a regular expression

Relational expression

   Basic usage

Print it again



Get IP address  


[root@localhost ~]#wc -l /etc/passwd
45 /etc/passwd
[root@localhost ~]#awk -F: '{print $0}' /etc/passwd
#$0 represents all elements
[root@localhost ~]#awk -F: '{print $1}' /etc/passwd
#Represents the first column
[root@localhost ~]#awk -F: '{print $1,$3}' /etc/passwd
#Represents the first and third columns
[root@localhost ky15]#awk '/^root/{print}' passwd 
#A line beginning with root has been
[root@localhost ky15]#grep -c "/bin/bash$" passwd 
#Counts the rows that currently end in / bin/bash
#####  BEGIN {} mode means that the specified actions in BEGIN mode need to be executed before processing the specified text; awk then process the specified text, and then execute the specified actions in END mode. In END {} statements, statements such as print results are generally placed.

Common built-in variables in awk

FS: Specifies the field separator for each line of text. The default is space or tab stop. It works the same as "- F" - v "FS =:"

NF: the number of fields in the currently processed row

NR: line number (ordinal number) of the currently processed line

$0: the whole line content of the currently processed line

$n: the nth field (column n) of the current processing line

FILENAME: the name of the file being processed

Rs: line separator. When awk reads data from a file, it will cut the data into many records according to the definition of RS, while awk only reads one record at a time for processing. The default is \ n

[root@localhost ky15]#awk -v FS=: '{print $1FS$3}' /etc/passwd
#Here FS is equivalent to a variable
[root@localhost ky15]#awk -F: '{print $1":"$3}' /etc/passwd

shell Variables in
[root@localhost ky15]#fs=":";awk -v FS=$fs '{print $1FS$3}' /etc/passwd
#Pass definition variable to FS
[root@localhost ky15]#fs=":";awk -v FS=$fs -v OFS="+" '{print $1,$3}' /etc/passwd
#Output separator

[root@localhost ky15]#awk -F: '{print NF}' /etc/passwd

[root@localhost ky15]#awk -F: '{print $NF}' /etc/passwd
#Last field

[root@localhost ky15]#df|awk -F: '{print $(NF-1)}'
#Penultimate line
[root@localhost ky15]#df|awk -F "[ %]+" '{print $(NF-1)}'

[root@localhost ky15]#awk '{print $1,NR}' /etc/passwd
##Line number
[root@localhost ky15]#awk 'NR==2{print $1}' /etc/passwd
#Take only the first field of the second row
[root@localhost ky15]#awk 'NR==1,NR==3{print}' passwd 
#Print out 1 to 3 lines
[root@localhost ky15]#awk 'NR==1||NR==3{print}' passwd
#Print out 1 and 3 lines
[root@localhost ky15]#awk '(NR%2)==0{print NR}' passwd
#Print out lines with function remainder of 0
[root@localhost ky15]#awk '(NR%2)==1{print NR}' passwd
#Print out the line with function remainder of 1
[root@localhost ky15]#awk 'NR>=3 && NR<=6{print NR,$0}' /etc/passwd

[root@localhost ky15]#seq 10|awk 'NR>5 && NR<10'
#Take between lines
[root@localhost ky15]#awk '$3>1000{print}' /etc/passwd

Posted by Tentious on Thu, 28 Oct 2021 06:10:13 -0700