Event response can be defined as the action process taken whenever a computer or network security event occurs. As an event responder, you should always know what should and should not appear in the system.
Troubleshooting ideas
(1) First, monitor the security of user accounts, such as new accounts and suspicious accounts, and focus on the accounts that can be logged in remotely and accounts with high permissions.
(2) Use the history command of linux to view the history command, and the uptime command to view how long and how many users have logged in.
(3) Check the abnormal port and process, netstat checks the abnormal port, ps checks the abnormal process, and you can view the process id occupied by resources to judge whether there are suspected mining Trojans.
(4) Check the linux startup item and the system's scheduled task crontab. crontab -l checks whether there are abnormal tasks written in.
(5) Check linux Log Information / var/log / some system log information, security log, etc.
(6) Automatic killing software, online killing tools and killing scripts.
linux emergency response can be carried out in four links:
- Identify the phenomenon - remove the virus - find out the bottom in a closed loop - system reinforcement. First, start from the abnormal phenomenon of the server, and identify the suspicious signs of the virus according to the CPU, memory occupation and network traffic of the server
- Then locate the specific virus process and virus file through process, port opening, historical command, inbound and outbound traffic, log audit, etc
- Because viruses are usually repeatedly infected through some self startup items and daemons, we should check the attacker's account, scheduled tasks, malicious self startup services, system file hijacking, daemons, etc. to prevent repeated infection
- After the virus items on the host are cleared, finally strengthen the system, analyze the log, and find the specific location of the vulnerability, which is not patched, not upgraded or weak password. Carry out targeted disposal of the vulnerability to prevent the virus from invading again from the web
Recognition phenomenon
Abnormal behaviors are found on the host through system operation status, security device warning and monitoring system: abnormal traffic, abnormal port, abnormal CPU / memory occupancy, etc
Evaluate possible problems for different phenomena:
- SSH brute force cracking: for the connection with unknown address on port 22, there are a large number of login failure records in / var/log/secure
- Short link: the monitoring device finds that the server constantly sends requests to an address, and finds short links when refreshing ports and processes for many times
- Mining virus: the server continuously sends a connection to the external ip and downloads the virus source; Irregular abnormal process and abnormal Download
- Gates virus: CPU resources are abnormal, abnormal processes occupy a lot of network bandwidth, abnormal IP connections, abnormal processes and abnormal startup items
- DDOS virus: abnormal server network resources and high bandwidth occupancy, affecting the use of network services
process
After connecting to the system server, start to investigate the phenomenon and check the CPU, memory, process, system mount, etc
#Use the top command to view cpu usage in descending order
top
A process with a CPU utilization of more than 70% and a suspicious name may be a virus
Enumerating processes
ps -aux
Viruses often carry suspicious command lines. When you find a url or strange characters in the command line, you should pay attention to whether it is a virus download program
Locate a process
lsof -p <PID>
View security gateway or monitoring system
View the process communicating with the target ip in the gateway or monitoring system, and list the process through the command line
##Enumerate the processes of network Socket connection while true;do netstat -antp | grep <ip>; done
Sometimes a domain name is detected, and its corresponding ip is constantly changing. Filtering ip with the above method is not applicable. It is suspicious to bind the domain name as a random ip in the hosts file and monitor it
In this way, the malicious process can be obtained by filtering with the above command
After obtaining the malicious process and its PID, use the following command to find the location where the malicious process is stored
ls -al /proc/<pid>/exe
port
Check whether there are any abnormal ports opened
netstat -nap //Query open port and process
##Query open ports, including UDP and TCP ##The parameter - t represents TCP and - u represents UDP netstat -nutpl
Historical command
##View the history instruction of root history
##View the historical instructions of each user cat /home/user/.bash_history
Malicious file lookup
It is best to find malicious file paths through malicious processes, ports, networks, etc., but sometimes we also need to carefully check whether there are residual viruses in other directories
First, check that the files / usr / bin / LS, / usr / bin / lsof, / usr/bin/stat have not been modified
stat /usr/bin/ls stat /usr/bin/lsof stat /usr/bin/stat
Check the read / write execution directory
ls -al /tmp/; ls -al /var/tmp; ls -al /dev/shm
Check the directory file under the $PATH environment variable
echo $PATH ls -al /usr/local/sbin ls -al /usr/local/bin ... wait
View all files recursively
ls -aR
Use stat to see when any files are changed, created, etc
View files and process calls through the lsof command
- lsof lists all process calls
- lsof abc.txt shows the process of opening the file abc.txt
- lsof -c abc displays the files that the abc process is now opening
- lsof -p 1234 lists the files opened by the process with process number 1234
- lsof -g gid displays the progress of the home gid
- lsof +d /usr/local / displays the files opened by the process in the directory
- lsof +D /usr/local / is the same as above, but it will search the directory under the directory for a long time
- lsof -d 4 shows processes using fd 4
- Lsof - I: Port checks which process uses this port
- lsof -i is used to display the qualified processes
Find the file size, newly added and modified files through find
Compare the intrusion environment with the pure environment through the diff command
First, copy the pure environment to PC-x
diff -r <dir1> <dir2>
Analyze malicious programs
##View the path corresponding to the malicious program ls -al /proc/<pid>/exe ##View the command name and parameters of the running process cat /proc/<pid>/cmdline
##View malicious file types file /home/light/abc.txt ##Check the string in ELF file strings /tmp/.elf
If the malicious program is deleted, the malicious program can be exported from memory by memory dump
Restore deleted files from memory copy cp /proc/[pid]/exe /tmp/malware.dump Export process memory cat /proc/[pid]/maps 7ff48bb5d000-7ff48bb5e000 gdb --pid [pid] dump memory /tmp/malware.dump 0x7ff48bb5d000 0x7ff48bb5e000
Automatic killing
chkrootkit
wget ftp://ftp.pangeia.com.br/pub/seg/pac/chkrootkit.tar.gz tar zxvf chkrootkit.tar.gz cd chkrootkit-0.53 make sense ./chkrootkit
rkhunter
wget https://nchc.dl.sourceforge.net/project/rkhunter/rkhunter/1.4.4/rkhunter-1.4.4.tar.gz Or this one below wget https://fossies.org/linux/privat/rkhunter-1.4.6.tar.gz tar -zxvf rkhunter-1.4.6.tar.gz cd rkhunter-1.4.6 ./installer.sh --install rkhunter -c
Clear virus
Through the above investigation, a malicious process was found
ls -al /proc/<pid>/exe
And locate the location where the malicious file exists, and you can upload the malicious file to the virus inspection website for inspection - virustotal.
End malicious process
##Malicious processes found ps -elf | grep <pid> ##-9 forced termination process kill -9 <pid>
Closed loop pocket
Attack maintenance
Check Linux account
Check the security of the system account to see if there are new suspicious, temporary and high authority accounts.
##View all accounts cat /etc/passwd ##View privileged users (uid 0) grep :0: /etc/passwd ##View account and password related information cat /etc/shadow ##View user login time uptime ##Query the utmp file and report each user currently logged in who ##Query the utmp file and display the processes of each user and its formation in the current system w ##List recent login reports for all users lastlog ##View remote SSH and telnet logins tail /var/log/auth.log tail /var/log/secure ##View sudo user list cat /etc/sudoers ##Multiple accounts can be disabled or deleted usermod -L user //Disable user account userdel user //Delete user account userdel -r user //Delete the user account and delete the user directory under the / home directory
Check for suspicious scheduled tasks
##List the current user cron service details crontab -l //The file is saved in / var/spool/cron/user ##Check for malicious scripts in the following directory cat /etc/crontab * /etc/crontab * /etc/cron.d/* * /etc/cron.daily/* * /etc/cron.hourly/* * /etc/cron.monthly/* * /etc/cron.weekly/ * /etc/anacrontab * /var/spool/cron/* * /var/spool/anacron/*
Check for suspicious services
View startup and self startup
Traverse directories and files beginning with init and rc in the / etc / directory
/etc/init.d
Query the service started since startup
service --status-all
Service self start modification
##The first modification method chkconfig [--level Run level] [Independent service name] [on|off] chkconfig –level 2345 httpd on Turn on self start chkconfig httpd on (default level Yes (2345)
##The second modification method modify/etc/re.d/rc.local file join /etc/init.d/httpd start
##The third modification method use ntsysv Command management self start
Check system log
Log storage location: / var/log/
View log configuration: more /etc/rsyslog.conf
Troubleshooting ssh
View / root /. ssh / known_ The ssh public key in the hosts file to see which part of the host is connected through ssh
cat /root/.ssh/known_hosts
Web service patch
Tomcat weak password attack
Weblogic WLS component vulnerability
Jboss deserialization vulnerability
structs2 series RCE vulnerabilities
thinkphp5.XRCE vulnerability
Redis unauthorized access vulnerability
ConfluenceRCE vulnerability (CVE_2019_3396)
DrupalRCE vulnerability (CVE-2018-7600)
ThinkPHPRCE vulnerability (CVE-2019-9082)
Invasion cause
Weak password / default password
Check the open services through netstat, and confirm whether the services (mysql, redis, zookeeper, tomcat, etc.) have configuration authentication and whether they use the default password or weak password
Check these service log information to see if there are intrusion records
view log
System log and application log
* /var/log/cron Logs related to system scheduled tasks are recorded * /var/log/cups Log of printing information * /var/log/dmesg It records the information of kernel self-test when the system is powered on * /var/log/mailog Record mail information * /var/log/message Log recording important information of the system * /var/log/btmp Log error logins. To use lastb Command view * /var/log/lastlog Log the last login time of all users in the system. To use lastlog Command view * /var/log/wtmp Permanently record the login and logout information of all users, and record the startup, restart and shutdown events of the system last Command view * /var/log/utmp Record the currently logged in user information. To use w,who,users Command view * /var/log/secure Record verification and authorization information, such as SSH Sign in, su Switch users, sudo to grant authorization other web Middleware logs, such as apache,mysql,ngnix
View ssh login records
cat /var/log/secure | grep 'Accepted'
Malicious process associated files
In most cases, the parent process of the malicious process is 1. In some cases, the parent process of the malicious process may not be 1. For example, the parent process is httpd. In this case, you can boldly guess that the attacker is exploiting the vulnerability of the parent process.
Use the command ps -ef to view the parent process pid of the process, that is, ppid
Check the user started by the malicious process through ps auxef. If it is found that the user started by mysql, for example, it can be inferred that the intrusion is through MySQL service.
Safety reinforcement
(1) Disable or delete useless accounts, check special accounts (accounts with remote login and high user permissions), prohibit remote login users from logging in if necessary, only log in locally, and set multiple login failure to lock the account.
(2) Check the permissions of important directories and files, chmod increase permissions, prevent tampering, etc.
(3) Shut down unnecessary services. Services unrelated to the enterprise can be shut down temporarily.
(4) Close unnecessary protocols, such as ftp, ssh, telnet, etc., which may have protocol vulnerabilities.
(5) Close unnecessary ports, and some ports may have port vulnerabilities. (6) check the security log from time to time and observe the log information.
(7) You can use the equipment of the security manufacturer to check in real time, or use the system anti-virus software to check and kill.
(8) Modify the weak password or the default password to strong password
(9) Timely upgrade components or middleware and add system patches