simple command for troubleshoot the network applications

most of time i do troubleshooting on the network application, these are few command i will use it with log as well.

1st, netstat
2nd, lsof

From server itself, you can use the below command
1st, netstat
[root@www errs]# netstat -an|grep 3306
tcp 0 0 0.0.0.0:3306 0.0.0.0:* LISTEN

note: see the port is opening :P


2nd, lsof
[root@www errs]# lsof -i tcp:3306
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
mysqld 7472 mysql 10u IPv4 12294 TCP *:mysql (LISTEN)

see, you can see those port 3306 is used by service, mysqld
and its PID is 7472
then is USER is mysql and etc.


especially lsof, it will tell you which application are bonded with the particular port.
explore lsof more :P you will get more with this command



from client you can try by localhost see whether the firewall is block it or not :P
C:\Documents and Settings\a13x>telnet www.aviocall.com 3306
Connecting To www.aviocall.com...Could not open connection to the host, on port
3306: Connect failed

see, it is failed to connect. i believe is my firewall block it.

Let see the log form server.
Dec 8 17:56:11 www kernel: Shorewall:net2all:DROP:IN=eth0 OUT= MAC=00:1e:68:9a:f6:19:00:13:7f:c4:47:1a:08:00 SRC=118.100.42.9 DST=203.114.11.102 LEN=48 TOS=0x00 PREC=0x00 TTL=113 ID=40540 DF PROTO=TCP SPT=50960 DPT=3306 WINDOW=65535 RES=0x00 SYN URGP=0
Dec 8 17:56:15 www psad: scan detected: 118.100.42.9 -> 203.114.11.102 tcp: [3306] flags: SYN tcp pkts: 1 DL: 1
see the log, the destination port is 3306 :P
that mean my firewall block it :P

so, the conclusion is mysql port 3306 only accessible by localhost or certain IP addresses :P


that's all for this part. Once you know what is the root cause, then only you can perform the troubleshooting.


another way , use sniffer like wireshark or tcpdump to troubleshoot it :P
you can get more :P but i dont plan to teach as im not really good in wireshark and this is dangerous tools for people who have evil mindset :P

fwknopd - Firewall Knock Operator aka SPA, Single Packet Authentication

fwknop is a tool used with the iptable to provide the single packet authentication.

I know the fwknop since i brought the bok Linux Firewalls by Michael Rash. He is the guy who created the fwknop, PSAD, fwsnort and etc security tools.

fwknop is a tool use to add the SSH service with specific IP into the iptables' rule and it is just available for 30 seconds to connect to server. Once time exceed, it will remove the rule from iptables, so that no one can connect to server.

For more information , please read the book Linux Firewalls and official website from Michael Rash http://www.cipherdyne.org/fwknop/

Now I have 2 boxes,

Node1 = 192.168.145.41 ( also known as server )

Node2 = 192.168.145.42 ( also known as client )

Node1 has been setup with firewall block the port 22

At, node2, now we do ssh to node1

1st, i scan the node1

[root@node2 ~]# nmap -P0 192.168.145.41 Starting Nmap 4.11 ( http://www.insecure.org/nmap/ ) at 2008-11-23 11:56 MYT

Interesting ports on node1 (192.168.145.41): Not shown: 1677 filtered ports

PORT STATE SERVICE

80/tcp open http

113/tcp closed auth

8080/tcp closed http-proxy

MAC Address: 00:0C:29:5E:8B:28 (VMware) Nmap finished: 1 IP address (1 host up) scanned in 22.151 seconds

2nd, i do ssh to node1

[root@node2 ~]# ssh 192.168.145.41

It has no return result.

At node1, let see the log

Nov 23 09:04:18 node1 kernel: Shorewall:net2all:DROP:IN=eth0 OUT= MAC=00:0c:29:5e:8b:28:00:0c:29:23:90:aa:08:00 SRC=192.168.145.42 DST=192.168.145.41 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=32176 DF PROTO=TCP SPT=44194 DPT=22 WINDOW=5840 RES=0x00 SYN URGP=0

See, my firewall has blocked out the 192.168.145.42.

Node1 :

[root@node1 fwknop]# vi /etc/fwknop/fwknop.conf

EMAIL_ADDRESSES chenghui81@gmail.com;

[root@node1 fwknop]# vi /etc/fwknop/access.conf

SOURCE: ANY;

OPEN_PORTS: tcp/22; ### for ssh (change for access to other services)

KEY: a13xch0ng;

FW_ACCESS_TIMEOUT: 30;

start the fwknop service

[root@node1 fwknop]# /etc/init.d/fwknop start

Starting the fwknop daemons: [ OK ]

tail the log

[root@node1 fwknop]# tail -f /var/log/messages

Nov 23 10:03:29 node1 init: Trying to re-exec init

Nov 23 11:33:28 node1 fwknopd: starting fwknopd v1.9.9 (file revision: 1353)

Nov 23 11:33:28 node1 fwknopd: flushing existing iptables fwknop chains

Nov 23 11:33:29 node1 fwknopd: imported access directives (1 SOURCE definitions).

Nov 23 11:33:29 node1 fwknopd: imported previous tracking digests from disk cache: /var/log/fwknop/digest.cache

Nov 23 11:33:29 node1 kernel: device eth0 entered promiscuous mode

(see the service started with the promiscuous mode, that mean it is sniffing the packet thru interface eth0.

you will see such message when you running sniffer )

Node2, now we do the Single Packet Authentication by using fwknop client

[root@node2 ~]# fwknop -A tcp/22 -a 192.168.145.42 -k 192.168.145.41

( note:: tcp/22 is refer to the port you wanted to connect, -a is refer to your source IP, -k is refer to your destination IP )

[+] Starting fwknop client (SPA mode)...

[+] Enter an encryption key. This key must match a key in the file

/etc/fwknop/access.conf on the remote system.

Encryption Key:

(it is asking the encryption key, the key is referring to the key you setup in the server side )

[+] Building encrypted Single Packet Authorization (SPA) message...

[+] Packet fields:

Random data: 2421962114322037

Username: root

Timestamp: 1227411567

Version: 1.9.9

Type: 1 (access mode)

Access: 192.168.145.42,tcp/22

SHA256 digest: uBZSsbkeoAH/pA44IHxLGvHt3rC8F513ry4XCVKimkU

[+] Sending 182 byte message to 192.168.145.41 over udp/62201...

Once you see such message that mean you are successfully initiated the connection.

Node1: from the /var/log/messages

Nov 23 11:39:45 node1 kernel: Shorewall:net2all:DROP:IN=eth0 OUT= MAC=00:0c:29:5e:8b:28:00:0c:29:23:90:aa:08:00 SRC=192.168.145.42 DST=192.168.145.41 LEN=210 TOS=0x00 PREC=0x00 TTL=64 ID=43837 DF PROTO=UDP SPT=64210 DPT=62201 LEN=190

Nov 23 11:39:45 node1 fwknopd: received valid Rijndael encrypted packet from: 192.168.145.42, remote user: root, client version: 1.9.9 (SOURCE line num: 26)

Nov 23 11:39:45 node1 fwknopd: add FWKNOP_INPUT 192.168.145.42 -> 0.0.0.0/0(tcp/22) ACCEPT rule 30 sec

(note : see the last line that has added our IP to the iptables in order to let us connect the SSH in 30 seconds)

Node2 : I login the node1 asap

[root@node2 ~]# ssh 192.168.145.41 -l root

The authenticity of host '192.168.145.41 (192.168.145.41)' can't be established.

RSA key fingerprint is be:73:8b:27:48:96:ce:f5:16:7c:c5:0d:81:99:c7:3d.

Are you sure you want to continue connecting (yes/no)? yes

Warning: Permanently added '192.168.145.41' (RSA) to the list of known hosts.

Address 192.168.145.41 maps to node1, but this does not map back to the address - POSSIBLE BREAK-IN ATTEMPT!

root@192.168.145.41's password:

Last login: Sun Nov 23 07:04:39 2008 from 192.168.145.1

( note: yes, we connected to the server thru ssh )

See the cool things from node1 , /var/log/messages

Nov 23 11:40:56 node1 fwknopd: add FWKNOP_INPUT 192.168.145.42 -> 0.0.0.0/0(tcp/22) ACCEPT rule 30 sec

Nov 23 11:41:28 node1 fwknop(knoptm): removed iptables FWKNOP_INPUT ACCEPT rule for 192.168.145.42 -> 0.0.0.0/0(tcp/22), 30 sec timeout exceeded

See the last 2nd line that told you the iptables has added the new rules to allow IP 192.168.145.42 (aka node2) to login thru port 22 within 30 sec

See the last line that told you the iptables has removed the rules from the iptables which is ip192.168.145.32 port 22 after 30 seconds

Yet, my email, please take a look : -

see that 2 unread emails, one is telling you the IP has been added into the iptables and another is telling you the time is exceed and the rule has been removed from iptables

fwknop is designed used to prevent the 0 day exploit..

cause once they scan your server, there is no ssh port opening :P

so, no way they can access into your server thru ssh.

cool is it??

quickly implement into your server to make another security layer for ssh.

PSAD - how it works ???

Here is the /var/log/messages

Nov 22 23:22:36 www kernel: Shorewall:net2all:DROP:IN=eth0 OUT= MAC=00:1e:68:9a:f6:19:00:13:7f:c4:47:1a:08:00 SRC=123.12.89.26 DST=203.114.11.102 LEN=40 TOS=0x00 PREC=0x00 TTL=106 ID=55810 PROTO=TCP SPT=6000 DPT=3389 WINDOW=16384 RES=0x00 SYN URGP=0
Nov 22 23:22:41 www psad: src: 123.12.89.26 signature match: "MISC MS Terminal Server communication attempt" (sid: 100077) tcp port: 3389
Nov 22 23:22:41 www psad: scan detected: 123.12.89.26 -> 203.114.11.102 tcp: [3389] flags: SYN tcp pkts: 1 DL: 2

Nov 22 23:28:03 www kernel: Shorewall:net2all:DROP:IN=eth0 OUT= MAC=00:1e:68:9a:f6:19:00:13:7f:c4:47:1a:08:00 SRC=221.10.240.100 DST=203.114.11.102 LEN=48 TOS=0x00 PREC=0x00 TTL=114 ID=48158 DF PROTO=TCP SPT=4121 DPT=4899 WINDOW=16384 RES=0x00 SYN URGP=0
Nov 22 23:28:07 www psad: src: 221.10.240.100 signature match: "MISC Radmin Default install options attempt" (sid: 100204) tcp port: 4899
Nov 22 23:28:07 www psad: scan detected: 221.10.240.100 -> 203.114.11.102 tcp: [4899] flags: SYN tcp pkts: 1 DL: 2

Nov 22 23:36:02 www kernel: Shorewall:net2all:DROP:IN=eth0 OUT= MAC=00:1e:68:9a:f6:19:00:13:7f:c4:47:1a:08:00 SRC=77.194.196.82 DST=203.114.11.102 LEN=48 TOS=0x00 PREC=0x00 TTL=110 ID=22232 DF PROTO=TCP SPT=4390 DPT=4899 WINDOW=65535 RES=0x00 SYN URGP=0
Nov 22 23:36:07 www psad: src: 77.194.196.82 signature match: "MISC Radmin Default install options attempt" (sid: 100204) tcp port: 4899
Nov 22 23:36:07 www psad: scan detected: 77.194.196.82 -> 203.114.11.102 tcp: [4899] flags: SYN tcp pkts: 1 DL: 2

see this 3 part, once the packet drop by my shorewall ( iptables ), PSAD then perform the job.
Then, It will read the signature if match it will block the IP for certain period

coool, is it?????

PSAD - update signature

[root@www ~]# psad --sig-update
[+] Archiving original /etc/psad/signatures -> signatures.old1
[+] Downloading latest signatures from:
http://www.cipherdyne.org/psad/signatures
--02:04:17-- http://www.cipherdyne.org/psad/signatures
Resolving www.cipherdyne.org... 204.174.223.204
Connecting to www.cipherdyne.org|204.174.223.204|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 45096 (44K) [text/plain]
Saving to: `signatures'

100%[=====================================================================================================================>] 45,096 36.9K/s in 1.2s

02:04:20 (36.9 KB/s) - `signatures' saved [45096/45096]

[+] New signature file /etc/psad/signatures has been put in
place. You can restart psad (or use 'psad -H') to import the
new sigs.

note:: see, the signature is updated.


Now do restart PSAD with the below command :-
[root@www ~]# psad -H
[+] HUP signal sent to psadwatchd (pid: 32406)
[+] HUP signal sent to psad (pid: 32404)

PSAD - how to

still in the redhat/centos :P

what is PSAD ??
im lazy to explain and please go to read it..
http://www.cipherdyne.org/psad/docs/

Here we go, go to download the PSAD package.
http://www.cipherdyne.org/psad/download/

ok, download the package by wget :P
[alex@www ~]$ wget http://www.cipherdyne.org/psad/download/psad-2.1.4-1.i386.rpm
--19:21:37-- http://www.cipherdyne.org/psad/download/psad-2.1.4-1.i386.rpm
Resolving www.cipherdyne.org... 204.174.223.204
Connecting to www.cipherdyne.org|204.174.223.204|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 748833 (731K) [audio/x-pn-realaudio-plugin]
Saving to: `psad-2.1.4-1.i386.rpm'
100%[============================================================>] 748,833 82.4K/s in 9.0s
19:21:47 (81.1 KB/s) - `psad-2.1.4-1.i386.rpm' saved [748833/748833]

note:: done


Install it , go go go :-
[alex@www ~]$ sudo rpm -ivh psad-2.1.4-1.i386.rpm
Preparing... ########################################### [100%]
1:psad ########################################### [100%]
Can't open /etc/psad/psadwatchd.conf: No such file or directory.
[+] You can edit the EMAIL_ADDRESSES variable in /etc/psad/psad.conf
/etc/psad/psadwatchd.conf to have email alerts sent to an address
other than root\@localhost

note:: see , it is so obvious that asking you to modify the psad.conf and psadwatchd.conf enable to send you the email alert :P

before modify must always backup the original config file, use this as a practice
[alex@www psad]$ sudo cp psad.conf psad.conf.old

Modify it la la la la
[alex@www psad]$ sudo vi psad.conf
Change this line to ur email address
EMAIL_ADDRESSES alex@gmail.com;


Make it auto startup after system reboot
[alex@www psad]$ sudo /sbin/chkconfig psad on
[alex@www psad]$ sudo /sbin/chkconfig --level 35 psad on


[alex@www psad]$ sudo /etc/init.d/psad start
Starting psad: [ OK ]

Check you /var/log/messages : -
Nov 15 19:32:58 www psad: imported valid icmp types and codes Nov 15 19:32:58 www psad: imported p0f-based passive OS fingerprinting signatures Nov 15 19:32:58 www psad: imported TOS-based passive OS fingerprinting signatures Nov 15 19:32:58 www psad: imported Snort classification.config Nov 15 19:32:58 www psad: imported original Snort rules in /etc/psad/snort_rules/ for reference info Nov 15 19:32:58 www psad: imported 205 psad Snort signatures from /etc/psad/signatures
if you saw above message, that mean your system has been added another security layer :P


Then, i do scanning by my freeBSD ( so sorry this part i will exclude it.. it is dangerous to let u guy learn it :P hehe kidding lah, i just being lazy to write it out )


now, see i had received an email from my system that report have somebody scan my server :P
Sorry for i couldnt reveal the information that belong to my customer :-


































continue the picture

securing your /tmp directory - how to

[alex@www tmp]$ touch alex.sh
[alex@www tmp]$ pwd
/tmp
[alex@www tmp]$ ls
alex.sh gconfd-root lost+found mapping-root
[alex@www tmp]$


Write a sample and stupid simple script to do testing:-
[alex@www tmp]vi /tmp/alex.sh
echo a;

note: just a line echo a will do, then save and quit.


Then, do chmod executable for the alex.sh to make it able to execute
[alex@www tmp]$ chmod a+x alex.sh

Run the script and see: -
[alex@www tmp]$ ./alex.sh
a
Note: SHIT, it is able to execute in the /tmp .. It is dangerous and not allow for the production server.

Switch user to root
[alex@www tmp]$ su -l root
Password:

Then, modify the /etc/fstab
[root@www ~]#vi /etc/fstab
/dev/md0 / ext3 defaults 1 1
/dev/md3 /var ext3 defaults 1 2
/dev/md2 /tmp ext3 loop,nosuid,noexec,rw 1 2
tmpfs /dev/shm tmpfs defaults 0 0
devpts /dev/pts devpts gid=5,mode=620 0 0
sysfs /sys sysfs defaults 0 0
proc /proc proc defaults 0 0
/dev/md1 swap swap defaults 0 0

note : just add on the bold part in the /tmp line :P will do

After that, do remount to make it available : -
[root@www ~]# mount -o remount /tmp

Re-run the script and see :-
[alex@www tmp]$ ls -la
total 72
drwxrwxrwt 7 root root 4096 Nov 15 18:50 .
drwxr-xr-x 24 root root 4096 Nov 8 17:49 ..
-rwxrwxr-x 1 alex alex 9 Nov 15 18:49 alex.sh
drwxrwxrwt 2 root root 4096 Nov 8 17:49 .font-unix
drwx------ 2 root root 4096 Nov 8 17:32 gconfd-root
srw-rw-rw- 1 root root 0 Nov 8 17:49 .gdm_socket
drwxrwxrwt 2 root root 4096 Nov 8 17:49 .ICE-unix
drwx------ 2 root root 16384 Nov 8 15:09 lost+found
srwxr-xr-x 1 root root 0 Nov 8 15:25 mapping-root
-rw------- 1 root root 1024 Nov 8 15:13 .rnd
-r--r--r-- 1 root root 11 Nov 8 17:49 .X0-lock
drwxrwxrwt 2 root root 4096 Nov 15 18:18 .X11-unix
-r--r--r-- 1 alex alex 11 Nov 15 18:18 .X2-lock

[alex@www tmp]$ ./alex.sh
-bash: ./alex.sh: Permission denied

see, it is permission denied :P wakakak
not able to execute anymore :P

kill user in the linux system - how to

below is the guide for centos5.1
i think this is a native command of Linux , should be applicable to any disto of Linux

check the user in the system with w or who: -
[root@www ~]# w
18:41:09 up 7 days, 52 min, 3 users, load average: 0.08, 0.09, 0.02
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT
root pts/1 118.100.15.195 16:44 1:49m 0.01s 0.01s -bash
root pts/2 118.100.15.195 18:03 20:14 0.05s 0.05s -bash
alex pts/3 118.100.15.195 18:40 0.00s 0.02s 0.01s sshd: alex [pri

see, total is 3 users, as myself is alex, so i would like to kill user root for both session pts1 and pts2

So, this is the killing command, to kill the pts1 just do as the command like below: -
[root@www ~]# skill -KILL -v pts/1
pts/1 root 25170 bash

Now, i kill the pts2 sesssion: -
[root@www ~]# skill -KILL -v pts/2
pts/2 root 26041 bash

Check and ensure whether the user has been killed or not:-
[root@www ~]# w
18:41:31 up 7 days, 52 min, 1 user, load average: 0.06, 0.08, 0.02
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT
alex pts/3 118.100.15.195 18:40 0.00s 0.02s 0.01s sshd: alex [pri

yes, the user no more exists in the system :P

setting up VNC in the centos

note ::
all the command start with either : -
# = super user
$ = normal user

stop the vnc server
#/etc/init.d/vncserver stop

modify the /etc/sysconfig/vncservers
#vi /etc/sysconfig/vncservers
add below 2 lines: -

VNCSERVERS="3:hackintosh"
VNCSERVERSARGS[3]="-geometry 1024x800"

note: hackintosh must an user in the system

login as hackintosh and then issue this command:
$vncpasswd
then, change your vnc password

Then, try the below command and see the information is correct or not :-
$vncserver -httpd

if you see the below message that mean the hackintosh user are allow to use the VNC
xauth: creating new authority file /home/hackintosh/.Xauthority
New 'hackintosh.com:3 (hackintosh)' desktop is hackintosh.com:3
Creating default startup script /home/hackintosh/.vnc/xstartup
Starting applications specified in /home/hackintosh/.vnc/xstartup
Log file is /home/hackintosh/.vnc/hackintosh.com:3.log

Then, restart vnc server。
#/etc/init.d/vncserver restart

Then, you will see the messages some sort like below : -
Starting VNC server: 3:hackintosh
New 'hackintosh.com:3 (hackintosh)' desktop is hackintosh:3

Starting applications specified in /home/hackintosh/.vnc/xstartup
Log file is /home/hackintosh/.vnc/hackintosh.com:3.log

[ OK ]


you see the above message the hackintosh.com:3 that mean you use the port 5903 to connect to server :P

simple enough :P