server compromised - finding the intruder

Today received a call from someone who told me his customer server has been compromised which setup by me since almost 1 year ago.

Ask my favor to do some checking.

according to him, the intruder left 3 files into one of the web directory.
2 is text files and another 1 is an application.
i had check these 2 text files which are harmless code and the application is eggdrop which is IRC Bot application.

since i know it is eggdrop, i see where the files has been installed.
[root@www ~]# find / -name egg*
/var/www/html/crew/libraries/x/conf/eggdrop.chan
/var/www/html/crew/libraries/x/doc/html/egg-core.html
/var/www/html/crew/libraries/x/doc/man1/eggdrop.1
/var/www/html/crew/libraries/x/doc/eggdrop.doc
/var/www/html/crew/libraries/x/eggdrop-1.6.6
/var/www/html/crew/libraries/x/logs/eggdrop.log.20090608
/var/www/html/crew/libraries/x/logs/eggdrop.log.20090606
/var/www/html/crew/libraries/x/logs/eggdrop.log.20090607
/var/www/html/crew/libraries/x/logs/eggdrop.log.20090609
/usr/share/pixmaps/gnobots2/eggs.png
/usr/lib/python2.4/site-packages/gtk-2.0/egg
from the result result i found out the application installed on /var/www/html/crew/libraries/


Now, see when the application install on the server
[root@www ~]# ls -la /var/www/html/crew/libraries/
-rwxr-xr-x 1 loon loon 16091 Nov 8 2008 unzip.lib.php
-rwxr-xr-x 1 loon loon 6512 Nov 8 2008 url_generating.lib.php
drwxr-xr-x 10 loon loon 4096 Jun 9 19:32 x
-rw-r--r-- 1 loon loon 1868180 Jun 6 05:48 x.tar.gz
-rwxr-xr-x 1 loon loon 6486 Nov 8 2008 zip.lib.php
----- --------------- more ----------------------------------------
see the line with red color, hints is there.


Now, go further... look at /var/log/html/crew/libraries/x
[root@www x]# ls -la /var/www/html/crew/libraries/x
total 2536
drwxr-xr-x 10 loon loon 4096 Jun 9 19:32 .
drwxr-xr-x 11 loon loon 4096 Jun 6 05:48 ..
-rw-r--r-- 1 loon loon 0 Jun 9 22:23 av.db
-rw-r--r-- 1 loon loon 71820 Jun 9 22:12 bs_data.eggdrop
-rw-r--r-- 1 loon loon 71031 Jun 9 22:12 bs_data.eggdrop.bak
drwxr-xr-x 2 loon loon 4096 Jun 6 05:54 conf
-rw-r--r-- 1 loon loon 0 Aug 18 2006 cs_spam_word.conf
drwxr-xr-x 5 loon loon 4096 Apr 5 2007 doc
-rw-r--r-- 1 loon loon 1163288 Sep 6 2001 eggdrop-1.6.6
drwxr-xr-x 3 loon loon 4096 Apr 5 2007 filesys
drwxr-xr-x 4 loon loon 4096 Apr 5 2007 help
-rw-r--r-- 1 loon loon 0 Aug 18 2006 klines
drwxr-xr-x 2 loon loon 4096 Apr 5 2007 language
drwxr-xr-x 2 loon loon 4096 Jun 9 00:01 logs
-rw-r--r-- 1 loon loon 5 Jun 6 05:57 pid.eggthrone
-rw-r--r-- 1 loon loon 465 Aug 18 2006 run
drwxr-xr-x 2 loon loon 4096 Jun 6 05:49 scripts
-rw-r--r-- 1 loon loon 14368 Feb 25 2004 shade
drwxr-xr-x 2 loon loon 4096 Apr 5 2007 text
-rwxrwxrwx 1 loon loon 1163288 Sep 6 2001 x
-rw-r--r-- 1 loon loon 7745 Jun 6 05:49 x3
-rwxr-xr-x 1 loon loon 21516 May 30 2007 xh
see, most of them are 6th Jun 2009, i believe the application runs on that day. Yet, i also believe it is installed on the same day.

The IRC Bot are runing by both same user and group, loon. Then, i talk to the user and he said he didnt share his account with anyone.

let's see anyone inside the server or not
[root@www x]# w
20:03:13 up 167 days, 43 min, 4 users, load average: 0.00, 0.02, 0.00
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT
alex pts/2 118.100.121.74 17:51 2:10m 0.04s 0.03s sshd: alex [priv]
loon pts/3 60.52.19.122 17:54 1:06m 0.02s 0.02s -bash
alex pts/5 118.100.125.164 19:24 35:32 0.03s 0.01s sshd: alex [priv]
alex pts/6 118.100.125.164 19:30 0.00s 0.08s 0.01s sshd: alex [priv]
shit, someone inside and that user said he didnt login to the server. i believe this is intruder.

i do whois and see where this IP address came from
[root@www x]# whois 60.52.19.122
[Querying whois.apnic.net]
[whois.apnic.net]
% [whois.apnic.net node-2]
% Whois data copyright terms http://www.apnic.net/db/dbcopyright.html

inetnum: 60.48.0.0 - 60.54.255.255
netname: XDSLSTREAMYX
descr: Telekom Malaysia Berhad
descr: Network Strategy
descr: Wisma Telekom
descr: Jalan Pantai Baru
descr: 50672 Kuala Lumpur
country: MY
from the return result , seems like the user from TMNET users.

do the netstat to see what connection available now
[root@www x]# netstat -an|more
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 127.0.0.1:2208 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:804 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:3306 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:6667 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:44526 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:10000 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:21 0.0.0.0:* LISTEN
tcp 0 0 127.0.0.1:631 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:25 0.0.0.0:* LISTEN
tcp 0 0 127.0.0.1:2207 0.0.0.0:* LISTEN
tcp 0 0 203.114.11.102:51646 194.14.236.50:6667 ESTABLISHED
tcp 0 0 203.114.11.102:55317 202.30.50.120:43 ESTABLISHED
tcp 0 0 :::8897 :::* LISTEN
--------------------------- more ---------------------------------------------------
see the line with red color, see the port 6667, which is IRC port.


now try whois that IP
[root@www x]# whois 194.14.236.50
[Querying whois.ripe.net]
[whois.ripe.net]
% This is the RIPE Whois query server #2.
% The objects are in RPSL format.
%
% The RIPE Database is subject to Terms and Conditions.
% See http://www.ripe.net/db/support/db-terms-conditions.pdf

% Note: This output has been filtered.
% To receive output for a database update, use the "-B" flag

% Information related to '194.14.236.0 - 194.14.236.255'

inetnum: 194.14.236.0 - 194.14.236.255
netname: DALNET-2
descr: DALnet unrouted servers
remarks: ################################################
remarks: If you get scanned by 194.14.236.50 you are NOT
---------------------------- more ------------------------------------------------
it is dalnet server.
lol, The intruder must be malaysian because malaysian very very like dalnet IRC.
i'm one of them in N year ago.

go further to see the port 6667.
[root@www alex]# lsof -i tcp:6667
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
x 2709 loon 6u IPv4 37479003 TCP 203.114.11.102:51646->pool.dal.net:ircd (ESTABLISHED)
lsof is coolz..
found out the PID is 2709 and user is loon.
that mean the x is executed by loon and PID is 2709.
application is x

now use ps command to see the x
[root@www ~]# ps auxww |grep x
root 409 0.0 0.0 0 0 ? S<>
mysql 1006 0.0 1.0 143508 21536 ? Sl Jan08 1:14 /usr/libexec/mysqld --basedir=/usr --datadir=/var/lib/mysql --user=mysql --pid-file=/var/run/mysqld/mysqld.pid --skip-external-locking --socket=/var/lib/mysql/mysql.sock
root 1565 0.0 0.0 6744 1772 ? Ss Jan08 0:14 /usr/libexec/postfix/master
postfix 1568 0.0 0.9 24980 19952 ? S Jan08 0:47 qmgr -l -t fifo -u
root 2308 0.0 0.0 1672 404 ? Ss 2008 0:00 klogd -x
loon 2709 0.0 0.2 8032 4152 ? S Jun06 0:01 -bash -m x3
root 7579 0.0 0.0 1900 376 ? Ss 2008 0:00 gpm -m /dev/input/mice -t exps2
root 7820 0.0 0.0 1752 448 ? S 2008 0:00 /var/ossec/bin/ossec-execd
------------------------------------------------ more ------------------------------------------------
the process command showed that loon runs X since 6th Jun 2009.


look at process 2709 by using netstat
[root@www ~]# netstat -anp |grep 2709
tcp 0 0 0.0.0.0:44526 0.0.0.0:* LISTEN 2709/-bash
tcp 0 0 203.114.11.102:51646 194.14.236.50:6667 ESTABLISHED 2709/-bash
udp 0 0 0.0.0.0:54629 0.0.0.0:* 2709/-bash
it is listening my tcp and udp port and established my tcp port with port number 51646


let see what loon doing now
[root@www alex]# lsof -u loon
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
x 2709 loon cwd DIR 9,3 4096 17892533 /var/www/html/crew/libraries/x
x 2709 loon rtd DIR 9,0 4096 2 /
x 2709 loon mem REG 9,0 11370676 /lib/libnss_files-2.5.so (path inode=11370543)
x 2709 loon 3u IPv4 37478934 TCP *:44526 (LISTEN)
sshd 24370 loon 3u IPv6 38236234 TCP 203.114.11.102:8897->52.60.in-addr.arpa.tm.net.my:newlixreg (ESTABLISHED)
bash 24371 loon cwd DIR 9,3 4096 17860330 /var/www/html/www.aviocall.com/upload
bash 24371 loon 255u CHR 136,3 5 /dev/pts/3
it is too long. i had cut down most of them, left the important one.
see the red line. see properly.
he using Ipv6 connected to my server ???????????????
and what he want to do over that upload directory?????

let's see upload directory
[root@www www.aviocall.com]# ls -la upload/
total 113672
drwxr-xr-x 6 loon loon 4096 Jun 9 23:01 .
drwxr-xr-x 11 loon loon 4096 Jun 9 19:10 ..
drwxr-xr-x 2 loon loon 4096 May 30 14:32 files
drwxr-xr-x 2 loon loon 4096 May 30 14:33 images
-rw-r--r-- 1 loon loon 111723227 Jun 9 12:56 Manual Patch 090609.exe
-rw-r--r-- 1 loon loon 4521984 Jun 9 23:03 Manual Patch 090609.exe.1
drwxr-xr-x 2 loon loon 4096 May 30 14:33 pdf
drwxr-xr-x 2 loon loon 4096 Jun 4 00:41 programs
[root@www www.aviocall.com]# date
Tue Jun 9 23:03:54 MYT 2009
look, the time of server and the date of file he modified.
he is doing something. well, i dont mind..
i want to monitor him now.

see, what he doing..
top - 23:05:48 up 167 days, 3:45, 4 users, load average: 0.01, 0.00, 0.00
Tasks: 164 total, 2 running, 162 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.1%us, 0.1%sy, 0.0%ni, 99.8%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 2066076k total, 1617796k used, 448280k free, 215712k buffers
Swap: 2096376k total, 120k used, 2096256k free, 1118104k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
2709 loon 15 0 8032 4188 1196 S 0.0 0.2 0:01.06 x
10053 loon 15 0 7268 1696 1376 S 0.0 0.1 0:00.09 wget
24370 loon 15 0 9892 1664 1112 S 0.0 0.1 0:00.56 sshd
24371 loon 15 0 4528 1416 1200 S 0.0 0.1 0:00.03 bash
using wget to get something from somewhere.

see what he get from wget
[root@www ~]# ps -afed|grep wget
loon 10053 24371 0 23:01 pts/3 00:00:00 wget http://bayou.asiasoft.net/CABAL/pds/Manual%20Patch%20090609.exe
root 10371 10342 0 23:13 pts/8 00:00:00 grep wget
download CABAL online game's thingy.