Catalog

1. basic operation
   1.1. File operation
   1.2. Text operation
   1.3. Directory operation
   1.4. SSH, System Information-Network Operation
   
2. Basic Shell Programming
   2.1. variable
   2.2. String substitution
   2.3. function
   2.4. condition
   2.5. loop
   
3. Skill
   
4. debugging

1. Basic Operations

a. export

Display all environment variables. If you want to get more information about a variable, use echo $VARIABLE_NAME.

export

Example:

$ export
SHELL=/bin/zsh
AWS_HOME=/Users/adnanadnan/.aws
LANG=en_US.UTF-8
LC_CTYPE=en_US.UTF-8
LESS=-R

$ echo $SHELL
/usr/bin/zsh

b. whereis

whereis uses the database built automatically by the system to search for executable files, source files and manual pages.

whereis name

Example:

$ whereis php
/usr/bin/php

c. which

It searches the directory specified by the environment variable PATH for executable files. This command prints the full path of the executable file.

which program_name 

Example:

$ which php
/c/xampp/php/php

d. clear

Clear the contents of the window.

1.1. File Operations

ls	touch	cat	more	head	tail	mv	cp	rm	diff
chmod	gzip	gunzip	gzcat	lpr	lpq	lprm

a. ls

List your files. ls has many options: - l lists the "long format" file, which contains the exact size of the file, the person who owns the file, the right to view the file, and when the last modification was made. - a Lists all files, including hidden files. For more information on this command, check here link.

ls option

Example:

$ ls -al
rwxr-xr-x   33 adnan  staff    1122 Mar 27 18:44 .
drwxrwxrwx  60 adnan  staff    2040 Mar 21 15:06 ..
-rw-r--r--@  1 adnan  staff   14340 Mar 23 15:05 .DS_Store
-rw-r--r--   1 adnan  staff     157 Mar 25 18:08 .bumpversion.cfg
-rw-r--r--   1 adnan  staff    6515 Mar 25 18:08 .config.ini
-rw-r--r--   1 adnan  staff    5805 Mar 27 18:44 .config.override.ini
drwxr-xr-x  17 adnan  staff     578 Mar 27 23:36 .git
-rwxr-xr-x   1 adnan  staff    2702 Mar 25 18:08 .gitignore

b. touch

Create or update your files.

touch filename

Example:

$ touch trick.md

c. cat

It can be used under UNIX or Linux for the following purposes.

• Display text files on the screen
• Copy text files
• Merge text files
• Create a new text file

cat filename
cat file1 file2 
cat file1 file2 > newcombinedfile

d. more

Display the first part of the file (move with spaces and type q to exit).

more filename

e. head

The first 10 lines of the output file.

head filename

f. tail

Output the last 10 lines of file. For - f to output additional data as the file grows.

tail filename

g. mv

Move files from one location to another.

mv filename1 filename2

filename1 file source path, filename2 is the target path.

h. cp

Copy files from one location to another.

cp filename1 filename2

filename1 file source path, filename2 is the target path.

i. rm

Delete files. Using this command in the directory will show you an error: rm: directory: is a directory. To delete a directory, you have to pass - rf to recursively delete everything in the directory.

rm filename

j. diff

Compare documents and list their differences.

diff filename1 filename2

k. chmod

Let you change the read, write and execute permissions of files.

chmod -options filename

l. gzip

Compressed files.

gzip filename

m. gunzip

Unzip gzip compressed files.

gunzip filename

n. gzcat

Let you view the gzip compressed file without gunzip.

gzcat filename

o. lpr

Print document.

lpr filename

p. lpq

View the printer queue.

lpq

Example:

$ lpq
Rank    Owner   Job     File(s)                         Total Size
active  adnanad 59      demo                            399360 bytes
1st     adnanad 60      (stdin)                         0 bytes

q. lprm

Remove something from the print queue.

lprm jobnumber

1.2. Text Operations

awk	grep	wc	sed	sort	uniq	cut	echo	fmt
tr	nl	egrep	fgrep

a. awk

Awk is the most useful command for handling text files. It runs on the entire file line by line. By default, it uses spaces to separate fields. The most common syntax for awk commands is

awk '/search_pattern/ { action_to_take_if_pattern_matches; }' file_to_parse

Let's take the following file / etc/passwd. The following is the sample data contained in this file:

root:x:0:0:root:/root:/usr/bin/zsh
daemon:x:1:1:daemon:/usr/sbin:/usr/sbin/nologin
bin:x:2:2:bin:/bin:/usr/sbin/nologin
sys:x:3:3:sys:/dev:/usr/sbin/nologin
sync:x:4:65534:sync:/bin:/bin/sync

So now let's just get the user name from this file. - F specifies which partition field we want to base on. In our example: {print $1} means that the first matching field is printed out.

awk -F':' '{ print $1 }' /etc/passwd

After running the above command, you will get the following output.

root
daemon
bin
sys
sync

For more details on how to use awk, see the following link.

b. grep

Find the text in the file. You can use grep to search for text lines that match one or more regular expressions and output only the matching rows.

grep pattern filename

Example:

$ grep admin /etc/passwd
_kadmin_admin:*:218:-2:Kerberos Admin Service:/var/empty:/usr/bin/false
_kadmin_changepw:*:219:-2:Kerberos Change Password Service:/var/empty:/usr/bin/false
_krb_kadmin:*:231:-2:Open Directory Kerberos Admin Service:/var/empty:/usr/bin/false

You can also force grep to ignore word case by using the - i option. - r can be used to search all files in a specified directory, for example:

$ grep -r admin /etc/

- w only searches for words. For grep details, see the following link.

c. wc

Tell you how many lines, words and characters there are in a file.

wc filename

Example:

$ wc demo.txt
7459   15915  398400 demo.txt

7459 is line number, 15915 is word number, 398400 is character number.

d. sed

A stream editor for filtering and transforming text.

example.txt

Hello This is a Test 1 2 3 4

Replace all spaces with hyphens

sed 's/ /-/g' example.txt

Hello-This-is-a-Test-1-2-3-4

Replace all numbers with "d"

sed 's/[0-9]/d/g' example.txt

Hello This is a Test d d d d

e. sort

Sort lines of text files

example.txt

f
b
c
g
a
e
d

sort example.txt

sort example.txt

a
b
c
d
e
f
g

Randomize an example.txt for sorting

sort example.txt | sort -R

b
f
a
c
d
g
e

f. uniq

Reporting or omitting duplicate lines

example.txt

a
a
b
a
b
c
d
c

Show only the unique row of example.txt (first you need to sort, otherwise you can't see overlap)

sort example.txt | uniq

a
b
c
d

Display the unique item in each line and tell me how many instances I found

sort example.txt | uniq -c

    3 a
    2 b
    2 c
    1 d

g. cut

Delete parts from each line of file.

example.txt

red riding hood went to the park to play

Display spaces in columns 2, 7 and 9 as separators

cut -d " " -f2,7,9 example.txt

riding park play

h. echo

Display a line of text

Display "Hello World"

echo Hello World

Hello World

Display "Hello World" with line breaks between letters

echo -ne "Hello\nWorld\n"

Hello
World

i. fmt

Simple Best Text Formatting Program

example: example.txt (1 line)

Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet.

Output the line of example.txt to the width of 20 characters

cat example.txt | fmt -w 20

Lorem ipsum
dolor sit amet,
consetetur
sadipscing elitr,
sed diam nonumy
eirmod tempor
invidunt ut labore
et dolore magna
aliquyam erat, sed
diam voluptua. At
vero eos et
accusam et justo
duo dolores et ea
rebum. Stet clita
kasd gubergren,
no sea takimata
sanctus est Lorem
ipsum dolor sit
amet.

j. tr

Translate or delete characters

example.txt

Hello World Foo Bar Baz!

Turn all lowercase letters into capitals

cat example.txt | tr 'a-z' 'A-Z' 

HELLO WORLD FOO BAR BAZ!

Change all spaces into newline characters

cat example.txt | tr ' ' '\n'

Hello
World
Foo
Bar
Baz!

k. nl

Number of rows displaying files

example.txt

Lorem ipsum
dolor sit amet,
consetetur
sadipscing elitr,
sed diam nonumy
eirmod tempor
invidunt ut labore
et dolore magna
aliquyam erat, sed
diam voluptua. At
vero eos et
accusam et justo
duo dolores et ea
rebum. Stet clita
kasd gubergren,
no sea takimata
sanctus est Lorem
ipsum dolor sit
amet.

Show example.txt with line number

nl -s". " example.txt 

     1. Lorem ipsum
     2. dolor sit amet,
     3. consetetur
     4. sadipscing elitr,
     5. sed diam nonumy
     6. eirmod tempor
     7. invidunt ut labore
     8. et dolore magna
     9. aliquyam erat, sed
    10. diam voluptua. At
    11. vero eos et
    12. accusam et justo
    13. duo dolores et ea
    14. rebum. Stet clita
    15. kasd gubergren,
    16. no sea takimata
    17. sanctus est Lorem
    18. ipsum dolor sit
    19. amet.

l. egrep

Print line-extension expressions for matching patterns (alias:'grep-E')

example.txt

Lorem ipsum
dolor sit amet, 
consetetur
sadipscing elitr,
sed diam nonumy
eirmod tempor
invidunt ut labore
et dolore magna
aliquyam erat, sed
diam voluptua. At
vero eos et
accusam et justo
duo dolores et ea
rebum. Stet clita
kasd gubergren,
no sea takimata
sanctus est Lorem
ipsum dolor sit
amet.

Lines in which "Lorem" or "dolor" are displayed

egrep '(Lorem|dolor)' example.txt
or
grep -E '(Lorem|dolor)' example.txt

Lorem ipsum
dolor sit amet,
et dolore magna
duo dolores et ea
sanctus est Lorem
ipsum dolor sit

m. fgrep

Print line-FIXED pattern matching to match pattern (alias:'grep-F')
example.txt

Lorem ipsum
dolor sit amet,
consetetur
sadipscing elitr,
sed diam nonumy
eirmod tempor
foo (Lorem|dolor) 
invidunt ut labore
et dolore magna
aliquyam erat, sed
diam voluptua. At
vero eos et
accusam et justo
duo dolores et ea
rebum. Stet clita
kasd gubergren,
no sea takimata
sanctus est Lorem
ipsum dolor sit
amet.

Find the specific string'(Lorem | doloar)'in example.txt

fgrep '(Lorem|dolor)' example.txt
or
grep -F '(Lorem|dolor)' example.txt

foo (Lorem|dolor) 

1.3. Directory Operations

mkdir

cd

pwd

a. mkdir

Generate a new directory.

mkdir dirname

b. cd

Execute this and move from one directory to another.

$ cd

Move you to the home directory. This command accepts the optional dirname and moves you to the directory.

cd dirname

c. pwd

Tell you your current catalogue.

pwd

1.4. SSH, System Info & Network Operations

ssh	whoami	passwd	quota	date	cal	uptime	w	finger	uname
man	df	du	last	ps	kill	killall	top	bg	fg
ping	whois	dig	wget	scp

a. ssh

ssh (SSH client) is a program used to log on to a remote machine and execute commands.

ssh user@host

This command also accepts the option that - p can be used to connect to a specific port.

ssh -p port user@host

b. whoami

Returns the current login username.

c. passwd

Allow current logged-in users to change their passwords.

d. quota

Display your disk quota.

quota -v

e. date

Displays the current date and time.

f. cal

Display the calendar of months.

g. uptime

Displays the current uptime.

h. w

Show who's Online

i. finger

Displays information about user.

finger username

j. uname

Display kernel information.

uname -a

k. man

Manual showing the specified command.

man command

l. df

Display disk usage.

m. du

Displays disk usage of files and directories in file names (du -s gives only one total).

du filename

n. last

List the designated users you logged in last.

last yourUsername

o. ps

List your processes.

ps -u yourusername

p. kill

Use the ID you provide to kill (end) the process.

kill PID

q. killall

Kill all processes by name.

killall processname

r. top

Displays the current active process.

s. bg

List Jobs that stop or work in the background; restore Jobs that stop in the background.

t. fg

Recent Front Office Job.
Brings the most recent job in the foreground.

u. ping

Pings host and output results.

ping host

v. whois

Get the domain's who information.

whois domain

w. dig

Get DNS information for the domain.

dig domain

x. wget

Download the file.

wget file

y. scp

Transfer files between a local host and a remote host or between two remote hosts.

Copy from local host to remote host

scp source_file user@host:directory/target_file

Copy from remote host to local host

scp user@host:directory/source_file target_file
scp -r user@host:directory/source_folder farget_folder

This command also accepts the - P option that can be used to connect to a specific port.

scp -P port user@host:directory/source_file target_file

2. Basic Shell Programming

In bash you will write the first line of script file, called shebang. This line in any script determines the execution capability of the script, such as a separate executable file, rather than typing sh, bash, python, php, etc. in advance in the terminal.

#!/bin/bash

2.1. Variables

Creating variables in bash is similar to other languages. There is no data type. Variables in bash can contain numbers, characters, strings, etc. You don't need to declare a variable, just assign a value to its reference to create it.

Example:

str="hello world"

The above line creates a variable str and assigns it "hello world". Retrieve the value of a variable by placing $at the beginning of the variable name.

Example:

echo $str   # hello world

Like other languages, bash has arrays. An array is a variable that contains multiple values. There is no maximum limit on the size of the array. The array in Bash is zero. The first element is indexed to element 0. There are several ways to create an array in bash. What are the following?

Examples:

array[0] = val
array[1] = val
array[2] = val
array=([2]=val [0]=val [1]=val)
array(val val val)

To display values at a specific index, use the following syntax:

${array[i]}     # where i is the index

If no index is provided, the array element 0 is assumed. To understand how many values are in an array, use the following syntax:

${#array[@]}

Bash also supports ternary conditions. Here are some examples.

${varname:-word}    # If varname exists and is not null, its value is returned; otherwise, word is returned.
${varname:=word}    # If varname exists and is not null, its value is returned; otherwise, it is set and its value is returned.
${varname:+word}    # If varname exists and is not null, return word; otherwise return null 
${varname:offset:length}    # Perform substring extensions. It returns a substring of $varname, starting with offset and up to length

2.2 String Substitution

Check some syntax about how to manipulate strings

${variable#pattern}         # if the pattern matches the beginning of the variable's value, delete the shortest part that matches and return the rest
${variable##pattern}        # if the pattern matches the beginning of the variable's value, delete the longest part that matches and return the rest
${variable%pattern}         # if the pattern matches the end of the variable's value, delete the shortest part that matches and return the rest
${variable%%pattern}        # if the pattern matches the end of the variable's value, delete the longest part that matches and return the rest
${variable/pattern/string}  # the longest match to pattern in variable is replaced by string. Only the first match is replaced
${variable//pattern/string} # the longest match to pattern in variable is replaced by string. All matches are replaced
${#varname}     # returns the length of the value of the variable as a character string

2.3. Functions

Like almost any programming language, you can use functions to group code segments in a more logical way, or to practice the sacred art of recursion. Declaring a function is just a matter of writing the function my_func {my_code}. Calling a function is like calling another program. You just need to write its name.

functname() {
    shell commands
}

Example:

#!/bin/bash
function hello {
   echo world!
}
hello

function say {
    echo $1
}
say "hello world!"

When you run the above example, the Hello function will output "world!". The two functions mentioned above are the same as hello and say. The main difference is the function say. This function prints the first parameter it receives. The parameters in the function are handled in the same way as those given to the script.

2.4. Conditionals

Conditional statements in bash are similar to other programming languages. Conditions come in many forms, such as if expression then statement, which is executed only when the expression is true.

if [expression]; then
    will execute only if expression is true
else
    will execute if expression is false
fi

Sometimes, if the condition becomes confusing, you can use the same condition case statements.

case expression in
    pattern1 )
        statements ;;
    pattern2 )
        statements ;;
    ...
esac

Expression Examples:

statement1 && statement2  # The conditions on both sides are true
statement1 || statement2  # One side is true

str1=str2       # str1 matches str2
str1!=str2      # str1 does not match str2
str1<str2       # Is str1 less than str2
str1>str2       # Is str1 larger than str2
-n str1         # str1 is not empty (length greater than 0)
-z str1         # str1 is empty (length is 0)

-a file         # Document Existence 
-d file         # The file exists as a directory 
-e file         # The file exists; the same - a 
-f file         # A file exists as a regular file (that is, not a directory or other special type of file) 
-r file         # You have permission to read 
-r file         # File exists, not empty 
-w file         # You have write permission 
-x file         # You have permission to execute documents

file1 -nt file2     # file1 is newer than file2
file1 -ot file2     # file1 is older than file2

-lt     # less than 
-le     # Less than or equal to 
-eq     # Be equal to
-ge     # Greater than or equal to 
-gt     # greater than
-ne     # Not equal to

2.5. Loops

There are three different types of cycles in bash. for, while and until.

for grammar:

for x := 1 to 10 do
begin
  statements
end

for name [in list]
do
  statements that can use $name
done

for (( initialisation ; ending condition ; update ))
do
  statements...
done

while grammar:

while condition; do
  statements
done

until grammar:

until condition; do
  statements
done

3. Tricks

Set an alias

Bash_profile can be opened by running the following command. nano ~/.bash_profile

alias dockerlogin='ssh www-data@adnan.local -p2222' # add your alias in .bash_profile

Quickly remove specific directories

nano ~/.bashrc

export hotellogs="/workspace/hotel-api/storage/logs"

source ~/.bashrc
cd hotellogs

4. Debugging

You can easily debug the bash script bash by passing different options. For example - n, commands are not run and only grammatical errors are checked. - The vecho command precedes running them. - The echo command processed by the x command line.

bash -n scriptname
bash -v scriptname
bash -x scriptname

License