What is the most effective, surest way to trace server load (centos / cpanel, shared hosting environment)? I'm aware of top and top -c but I'm looking at it and nothing shows high usage, yet whm server status shows about 12-20% cpu (dual xeon box, 4gb memory). Disk usage is 32% (10k scsi, 300GB) I've attached 4 samples of top -c
I have a cpanel, centos server that had to be rebooted due to the high load spike. I was unable to ssh in nor enter whm prior to the reboot so my only option was of course to reboot the server.
Now, I'm trying to trace the cause of the spike. RTG graphs show no indication of a ddos attack.
Inside whm, CPU/Memory/MySQL Usage shows no reds nor yellow warnings either except red for gzip (backup process) at 19% CPU. I doubt this is the cause.
This is from /var/log/messages of the logs slightly before and after the reboot.
Code: Apr 9 13:31:49 server pure-ftpd: (?@70.250.201.162) [INFO] Logout. Apr 9 13:32:05 server pure-ftpd: (?@xxx.xxx.85.3) [INFO] New connection from xxx.xxx.85.3 Apr 9 13:32:28 server pure-ftpd: (?@xxx.xxx.85.3) [INFO] New connection from xxx.xxx.85.3 .......
I have a server that has server load showing at 25-40 (once it was even 53!), running like that for hours. The server has 4 cpus - and yet the sites on the server seem to run fine when I check them. What I'm wondering is, what exactly is load in this context; and how can load run so high like that without the server crashing?
According to top, the load is caused by httpd processes running under user 'nobody', that often take up double digit CPU percentage.
Does Apache always run under 'nobody'?
Is there any way to trace an httpd processes - which account it's for, or which physical script or URL is calling it?
And for top itself, the TIME field on one server of mine is in the format xx:xx (e.g. 3:25), on another it's TIME+ and in the format xx:xx.xx (e.g. 30:02.77). What exactly does this mean? I would asume it's minutes:seconds and minutes:seconds:hundredths, but while watching top it doesn't seem to correlate with that.
I see that there are some connections from my server to some remote mySQL server, and I am curios to know which script is running them. (192.168.30.98:40493 207.45.xxx.xx:3306 5339/httpd)
I try through lsof but it is not that it points directly to the website running this connection.
I have a customer that uses mailman to send out newsletters. For some reason he has a few users on that list at AOL that feel they need to report them as spam instead of simply unsubscribing or deleting. I'm on the AOL feedback loop, and receive these complaints each time he sends.
AOL removes (redacts) all traces that might make it possible to simply remove the complainer, instead making me have to go in and trace the mail. Ok, privacy and all that. Fine. Unfortunately, since he has a few hundred AOL addresses on that list, it's nearly impossible (AFAIK) to figure out who it is so I can remove that user (or three).
I've searched through the mail logs using the message ID and timestamp, as well as grepping for AOL in the results. This has narrowed it down to about 120 addresses. From here I'm stuck. Can anyone recommend something else to try in order to narrow it down further? What am I missing?
Here are the headers that came with the complaint, edited for privacy.
Quote:
Return-Path: <redacted-bounces@customerdomain.com> Received: from rly-dd06.mx.aol.com (rly-dd06.mail.aol.com [172.19.141.153]) by air-dd03.mail.aol.com (v120.9) with ESMTP id MAILINDD034-b804748865c294; Sat, 24 Nov 2007 15:15:39 -0400 Received: from neo.myserver.com (ns3.myserver.com [my.server.ip.addy]) by rly-dd06.mx.aol.com (v120.9) with ESMTP id MAILRELAYINDD062-b804748865c294; Sat, 24 Nov 2007 15:15:29 -0400 Received: from localhost ([127.0.0.1] helo=neo.myserver.com) by neo.myserver.com with esmtp (Exim 4.68) (envelope-from <redacted-bounces@customerdomain.com>) id 1Iw1P8-0006og-3w; Sat, 24 Nov 2007 15:15:06 -0500 Received: from adsl-225-31-27.mia.bellsouth.net ([customer's.ip] helo=Home) by neo.myserver.com with esmtpa (Exim 4.68) (envelope-from <customer_email@customerdomain.com>) id 1Iw1P1-0006l0-Tc for redacted@customerdomain.com; Sat, 24 Nov 2007 15:15:00 -0500 Message-ID: <002d01c82ed6$b4485f20$210110ac@Home> From: <customer_email@customerdomain.com> To: redacted@customerdomain.com Date: Sat, 24 Nov 2007 15:15:05 -0500p
Anyone know of some good server load testers ( commercial )?
Im not looking for application based load testing, I need real web server load testing... need to see how much traffic this one site can take before it cries.
I'm having the oddest issue. For some reason, some of the websites on my server load fine, and some take a really long time to load (2 minutes).
Now, the server load is fine, and the size of the sites aren't the issue either. I've restarted Apache and a couple more services, and still the same sites seem to load very slow.
What could be causing this since it's only effecting certain websites?
I've been having trouble with my VPS for a while now. In the QoS alerts page in Virtuozzo it seems to be a problem with numtcpsock and tcprcvbuf, mainly numtcpsock.
Copy these into the browser: i18.photobucket.com/albums/b106/gnatfish/qosnumtcpsock2.jpg
Now-a-days server is having too much load due to http and in access logs we see following message : ======================================== 127.0.0.1 - - [11/Oct/2008:01:40:02 -0700] "OPTIONS * HTTP/1.0" 200 - 127.0.0.1 - - [11/Oct/2008:01:40:03 -0700] "OPTIONS * HTTP/1.0" 200 - 127.0.0.1 - - [11/Oct/2008:01:40:02 -0700] "OPTIONS * HTTP/1.0" 200 -=============================================
And due to this there is load on server. We are not able to understand why this is happening and how to stop this. So please suggest with some solution.
I am facing a strange issue with two RPG games sites that i am hosting on a 2GB ram server with softlayer, each of these sites was running on a 512MB VPS and they were doing fine but i decided to move them to better server so things run smoother when each has 50+ members online.
So the issue is when i have around 40 online members on each site, they start coughing up mysql errors, sometimes the sites just hang and stop loading, server load goes up to 3 or 4 but memory usage remains around 40%, I also notice that mysql uses the most of the CPU usage, around 75%
What i probably need is for someone to identify the cause of this, can it be an attack of some sort? or is it probably some code issues in the sites? do you recommend anyone that can have a look at this?
I built an online application using PHP / MySQL and it's pretty optimized (it's a very simple app). I've inadvertently picked up a huge client that could represent a few 1,000 "posts" per minute.
The process goes like this:
Consumer posts mobile data -> third party receives data -> third party sends data to my server -> my php script throws the data into a database.
I've got about a month to prepare things for these nightly "posts" before it's a real-life affair.
My questions are:
1) How can I "stress" or "load" test my machine and script to see what I'm up against.
2) What is my cheapest option for "redundancy" (Would something like MediaTemple's MySQL Grid products be a fix?)
When I say a few thousand posts per minute - each one is around 350 bytes.
I'm sure you may be wondering if I've bitten off more than I can chew - yea kind of but I made it perfectly clear to the client about my uncertainty and they are willing to give me benefit-of-a-doubt since I'm a very hard worker, very easy to deal with and loyal - not to mention my product is unlike any other that we have seen.
I have some serious server load issues on one of my web servers...
Hardware: Intel Pentium D 3.40GHz 1024 MB DDR RAM
Software: RH Linux with 2.6.9-42.0.3.ELsmp kernel cPanel with most recent Apache and PHP 5.2.x
Process list (top output, on a regular moment):
[url]
When everything is running normal (read: server load below 2) there's between 100 and 300 MB of free memory. Though sometimes (this happens about 2 times per day, at random times of the day) the server load dramatically increases.
I wrote a script (the chkApache.sh process in the top list) that constantly checks the server load and if it raises about 4, it will check if the sum of all httpd processes are consuming too much CPU or Memory. In this case, it will force a httpd restart... I need this to prevent Apache from crashing my server almost daily. If I let Apache do its thing without this script, it will happen that tons of httpd processes (50 and up) take in all the RAM and server load increases to 100 or more and the eventually I need to reboot the server using the SoftLayer control panel.
That chkApache script I wrote also sends me a detailed report of the state of the server when it needs to restart Apache, here's a report of an event that occurred today (includes server load info, memory info, top list, httpd processes info, netstat, etc):
[url]
In this case you would think that someone is attacking the youthforums.co.uk domain but I doubt that's the case... It doesn't always happen with that site, in fact I can't seem to find a pattern in the Apache status page so I don't think a single account is causing this...
Today I also used the Apache JMeter to "stress test" my server. I was shocked when I saw how easy it is to use that tool in order to make my server crash... I used 10 threads loading one PHP page (that makes some MySQL queries) and made a loop that kept requesting that information, with 10 connections at a time... The server load rapidly increased to 30 and above... I think it's unacceptable that something like this can happen so easilly...
I tried several things with my Apache configuration settings... Here's what I have at the moment:
LoadModule rewrite_module libexec/mod_rewrite.so LoadModule expires_module libexec/mod_expires.so LoadModule bwlimited_module libexec/mod_bwlimited.so LoadModule bytes_log_module libexec/mod_log_bytes.so LoadModule auth_passthrough_module libexec/mod_auth_passthrough.so LoadModule php5_module libexec/libphp5.so LoadModule security_module libexec/mod_security.so LoadModule evasive_module libexec/mod_evasive.so LoadModule limitipconn_module libexec/mod_limitipconn.so Does anyone have an idea what's wrong here? Anything I can do to get more detailed information on what's causing this? I've been on this problem for weeks now but I can't seem to find any proper solution.
In case it matters I'm also running: SIM, RPM, BFD, APF and SPRI
how the load on your web hosting machines looks like. I've seen companies that keep the load at values of 10-15 units and the server is still responsive. Though there might be different reasons for the high load and it's a sure indication for an upcoming problem it doesn't seem normal to have this high load on a hosting server. (well actually it would if the machine was running on 16 cpus but it's a normal dual xeon woodcrest for example).
I've very strange problem, server load reache to 290
I couldn't detrmine process casue it httpd, MySQL or exim, i typed small script to calculate server status every 5 minutes it's shoot of result of script
Code:
server time now is 20-01-07-11:40:00 AM -------------------------------- 11:41:01 up 2 days, 23:30, 0 users, load average: 18.69, 9.40, 5.21 ///////////////////////////// Uptime: 109830 Threads: 154 Questions: 7360473 Slow queries: 4330 Opens: 183336 Flush tables: 1 Open tables: 512 Queries per second avg: 67.017 ///////////////////////////// Current HTTP request: 106 *********************************** *********************************** server time now is 20-01-07-11:50:00 AM 11:53:16 up 2 days, 23:43, 0 users, load average: 205.83, 144.73, 74.31 ///////////////////////////// Uptime: 110565 Threads: 501 Questions: 7390472 Slow queries: 5156 Opens: 184687 Flush tables: 1 Open tables: 512 Queries per second avg: 66.843 ///////////////////////////// Current HTTP request: 260 *********************************** ***********************************
problem solved after restart httpd, MySQL and exim, but i need final solution. it's server specs:- CPU: Quad Xeon 3.0 GHz RAM: 2 GB O.S: CentOS 4.4
Also I installed apf firewall, but can not install ant ddos module on Apache because clients from arab Gulf area connected to internet using same IP.
I dont know much about what could be cuasing this so i come to you for advice, i am currently at wired tree on their VPS384 package with 348MB of ram and my site is really slow to load [url]sometimes it is so embaressing to show people i just dont bother, i am sure this is deferring visitors from my content how can i speed this up?
I can add more ram but it will push the price and if the price goes too high i might aswell move to a Hybrid with wiredtree.
Spent a lot of time on this and out of ideas so would appreciate any help. There is one site (database driven forum) with about 300 online at the peak. At peak the load spikes at 6-7 and CPU and RAM usage are very low...