Detect Memory

I have a private vps server works under linux ( centos ), sometimes am getting msg from csf/firewall subject:

lfd on website.com: Suspicious process running under user user account

when i check my cpanel/whm vps ( service status ) its shows that the memory limit 80% - 85% , It's had a good forum works with vb, but am wonder how to check my vps memory, i mean how to detect if there any script, or malware, or anything takes the vps memory out...

Is there any way to check,know what works under my vps, so it's take my memory limit 85%?

i check the tmp folder,

root@www [/home]# cd /tmp
root@www [/tmp]# ls -la
total 364
drwxrwxrwt 6 root root 4096 Jul 25 02:14 ./
drwxr-xr-x 21 root root 4096 Jul 18 02:21 ../
drwxrwxrwt 2 root root 4096 Jun 30 05:50 .ICE-unix/
drwxrwxrwx 18 root root 4096 Jul 2 17:33 eaccelerator/
lrwxrwxrwx 1 root root 27 Jul 18 02:13 mysql.sock -> ../var/lib/mysql/mysql.sock=
drwxr-xr-x 3 root root 4096 Jun 30 05:29 pear/
drwx------ 3 root root 4096 Jul 5 18:31 spamd-23647-init/
-rw------- 1 root root 343335 Jul 19 02:50 whatis.bk6140
root@www [/tmp]# cd /home
and the df space

root@www [~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/simfs 80G 4.1G 76G 6% /
root@www [~]#
and the services running is

USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.0 0.0 2060 156 ? Ss Jun30 1:23 init [3]
root 7465 0.0 0.0 2444 156 ? S Jul03 0:00 /bin/sh /usr/bin/mysqld_safe --datadir=/var/lib/mysql --pid-file=/var/lib/mysql/www.website.com.pid
mysql 7491 0.0 2.5 33452 10440 ? Sl Jul03 11:33 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --pid-file=/var/lib/mysql/www.website.com.pid --skip-external-locking
root 10236 0.0 6.0 27396 24764 ? Ss Jul24 0:07 /usr/bin/spamd -d --allowed-ips=127.0.0.1 --pidfile=/var/run/spamd.pid --max-children=3 --max-spare=1
root 11447 0.0 1.9 18364 8020 ? S Jul24 0:00 cpsrvd - waiting for connections
root 11865 0.0 0.7 13672 3260 ? S Jul06 0:00 /usr/local/apache/bin/httpd -k start -DSSL
root 13537 0.0 3.1 15092 13064 ? Ss 00:00 0:02 lfd - sleeping
root 13703 0.0 0.3 3808 1284 ? SN Jul06 0:01 cpanellogd - sleeping for logs
root 13739 0.0 0.6 5856 2844 ? Ss Jul06 0:00 cPhulkd - processor
root 13795 0.0 1.5 14760 6304 ? S Jul06 0:00 cpdavd - accepting connections on 2077 and 2078
root 18161 0.0 0.0 1716 380 ? Ss Jun30 0:24 syslogd -m 0
root 18164 0.0 0.0 1668 72 ? Ss Jun30 0:00 klogd -x
dbus 18193 0.0 0.0 2736 212 ? Ss Jun30 0:00 dbus-daemon --system
root 18213 0.0 0.0 2716 172 ? Ss Jun30 0:00 xinetd -stayalive -pidfile /var/run/xinetd.pid
root 18399 0.0 5.9 27604 24404 ? S Jul24 0:06 spamd child
root 19461 0.0 0.1 3228 684 ? Ss Jun30 0:08 crond
root 19616 0.0 0.0 1820 124 ? Ss Jun30 0:00 /usr/sbin/portsentry -