after installed php 5.2.10
nobody /usr/local/apache/bin/httpd -k start -DSSL
Code:
5896 nobody 0
81.5
1.3 /usr/local/apache/bin/httpd -k start -DSSL
mysql (5.0.81-community)
os : centos 5.3
Has been very fast server. Guide is accurate and if you leave.
When I look at my Current CPU Usage / process manager, I see this:
Pid |Owner | Priority | Cpu % |Mem % | Command
13754 | nobody | 0 | 2.2 | 4.0 |/usr/local/apache/bin/httpd -k start -DSSL
I've got about 9 more of these, and it keeps eating up the memory. What is it and what can I do? Is it just apache running?
I have a cPanel server with 4GB RAM
My server hangs time to time, once or twice a day. This is the last status when it happens.
Server Load 75.90 (2 cpus)
Memory Used 92 %
Swap Used 65.78 %
When I was still inside SSH when it happens, the processes I can see inside "top" are a lot of httpd processes.
So I "killall -9 httpd", I had to do it 30~40 times
#killall -9 httpd
#killall -9 httpd
#killall -9 httpd
x 30~40 times until no pid process found & the server load is back to normal.
Before that, I check httpd fullstatus, they look very normal, same goes to what I see inside WHM Apache status & cpu usage status.
At first I thought it was a memory problem after consulting with a server admin, so I replaced all 4GB new ram stick (such a waste)
I have a e-card site. The e-mail and mysql are on other server, so this server only run apache.
load average: 25.52, 23.53, 26.48
Tasks: 125 total, 22 running, 97 sleeping, 0 stopped, 6 zombie
Cpu(s): 90.1% us, 9.3% sy, 0.0% ni, 0.0% id, 0.0% wa, 0.6% hi, 0.0% si
Mem: 2059596k total, 1443196k used, 616400k free, 210660k buffers
Swap: 2048276k total, 148420k used, 1899856k free, 451924k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
28864 nobody 15 0 10684 8120 2272 S 5.0 0.4 0:00.63 /usr/local/apache/bin/httpd -DSSL
28837 nobody 15 0 44348 40m 2272 S 4.0 2.0 0:02.13 /usr/local/apache/bin/httpd -DSSL
28872 nobody 15 0 7728 5220 2276 S 4.0 0.3 0:01.31 /usr/local/apache/bin/httpd -DSSL
28846 nobody 16 0 10708 8132 2288 S 3.4 0.4 0:00.94 /usr/local/apache/bin/httpd -DSSL
28855 nobody 15 0 7604 5104 2268 S 3.4 0.2 0:01.11 /usr/local/apache/bin/httpd -DSSL
28830 nobody 15 0 7604 5108 2272 S 3.1 0.2 0:01.50 /usr/local/apache/bin/httpd -DSSL
28874 nobody 15 0 7696 5176 2268 S 3.1 0.3 0:00.96 /usr/local/apache/bin/httpd -DSSL
28876 nobody 15 0 7716 5212 2272 S 3.1 0.3 0:01.25 /usr/local/apache/bin/httpd -DSSL
28968 nobody 15 0 7664 5212 2272 S 3.1 0.3 0:01.46 /usr/local/apache/bin/httpd -DSSL
28838 nobody 16 0 7744 5220 2276 S 2.8 0.3 0:00.96 /usr/local/apache/bin/httpd -DSSL
28844 nobody 16 0 7656 5132 2276 R 2.8 0.2 0:01.51 /usr/local/apache/bin/httpd -DSSL
28845 nobody 15 0 10580 8124 2272 S 2.8 0.4 0:00.51 /usr/local/apache/bin/httpd -DSSL
28869 nobody 16 0 7668 5144 2276 S 2.8 0.2 0:01.29 /usr/local/apache/bin/httpd -DSSL
28873 nobody 16 0 7704 5212 2272 S 2.8 0.3 0:01.13 /usr/local/apache/bin/httpd -DSSL
28960 nobody 16 0 7608 5112 2276 S 2.8 0.2 0:00.72 /usr/local/apache/bin/httpd -DSSL
29023 nobody 15 0 7604 5108 2272 S 2.8 0.2 0:00.59 /usr/local/apache/bin/httpd -DSSL
28858 nobody 15 0 7696 5128 2272 S 2.5 0.2 0:01.24 /usr/local/apache/bin/httpd -DSSL
Current Time: Thursday, 01-Mar-2007 14:41:43 EET
Restart Time: Thursday, 01-Mar-2007 14:37:06 EET
Parent Server Generation: 0
Server uptime: 4 minutes 37 seconds
Total accesses: 5468 - Total Traffic: 15.1 MB
CPU Usage: u34.89 s1.41 cu6.72 cs3.37 - 16.7% CPU load
19.7 requests/sec - 56.0 kB/second - 2904 B/request
38 requests currently being processed, 0 idle servers
WWWWWWWWWW.WWWWRWWWWWWWW.WRWWWWWWWWWWWWW........................
................................................................
................................................................
Timeout 30
MaxKeepAliveRequests 100
KeepAlive Off
KeepAliveTimeout 15
MinSpareServers 10
MaxSpareServers 18
StartServers 10
MaxClients 50
MaxRequestsPerChild 50
apache_1.3.37
how to optimize apache?
a topic long time ago that my server load is frequently high.
I'm talking about something like this
Server Load 158.86
Memory Used 28.2 %
Swap Used 99.57 %
[url]
The only way to solve this problem is to identify the load earlier and kill all httpd process. What I did was
#killall -9 httpd
#killall -9 httpd
#killall -9 httpd
x 30~40 times until no pid process found & the server load is back to normal.
On previous thread, I tried to update mysql & php and it works,
Right now again I am experiencing high server load again...
I'm very sure it's caused by httpd but I am still unable to find out the real cause of the problem and which account user is the culprit for causing this high load.
Can someone assist me by telling me where/how to begin with?
i have 10 web sites running on my server, but they are really small talking about traffic, according to statistics, my "star web site" have 300 visits per day
I also have a few shoutcast services running on my server, but i think ther are fine
According to cpanel, yesterday I had this cpu usage (only showing the top process)
User Domain %CPU %MEM MySQL Processes
nobody 79.10 25.07 0.0
and here you go a copy op "top" command in shell
Cpu(s): 39.5%us, 57.8%sy, 0.0%ni, 20.6%id, 0.0%wa, 0.0%hi, 1.2%si, 0.0%st
Mem: 2065856k total, 1464804k used, 601052k free, 71616k buffers
Swap: 2048276k total, 0k used, 2048276k free, 955020k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
10345 nobody 20 0 29484 16m 3360 R 21 0.8 0:41.87 httpd
10874 nobody 20 0 20416 9792 2852 R 20 0.5 0:10.19 httpd
10340 nobody 20 0 25748 12m 2928 R 14 0.6 0:28.20 httpd
4309 sc_dieg2 20 0 44288 4116 1788 S 1 0.2 0:18.59 sc_serv
4176 sc_maxi 20 0 37004 2816 1064 S 1 0.1 0:14.76 sc_serv
4244 sc_salme 20 0 37004 2688 1080 S 1 0.1 0:12.73 sc_serv
4316 sc_dieg3 20 0 44288 4392 1752 S 1 0.2 0:17.48 sc_serv
4352 sc_sergi 20 0 39684 3264 1320 S 1 0.2 0:19.24 sc_serv
4413 sc_eddyo 20 0 39684 2892 1368 S 1 0.1 0:16.68 sc_serv
4420 mexicanr 20 0 38524 1916 588 S 1 0.1 0:11.28 sc_serv
4534 sc_entor 20 0 38680 2720 1264 S 1 0.1 0:13.05 sc_serv
9547 root 20 0 2388 1076 796 R 1 0.1 0:09.37 top
4116 sc_jocuh 20 0 36848 1988 576 S 0 0.1 0:11.25 sc_serv
4553 sc_aleja 20 0 89004 8332 6864 S 0 0.4 0:11.94 sc_serv
6829 120 20 0 5988 2156 1664 S 0 0.1 0:01.64 glftpd
11166 root 20 0 7044 2332 1868 S 0 0.1 0:00.01 sshd
1 root 20 0 2136 644 556 S 0 0.0 0:00.82 init
2 root 15 -5 0 0 0 S 0 0.0 0:00.00 kthreadd
3 root RT -5 0 0 0 S 0 0.0 0:00.00 migration/0
4 root 15 -5 0 0 0 S 0 0.0 0:00.18 ksoftirqd/0
5 root RT -5 0 0 0 S 0 0.0 0:00.00 watchdog/0
6 root RT -5 0 0 0 S 0 0.0 0:00.00 migration/1
7 root 15 -5 0 0 0 S 0 0.0 0:00.13 ksoftirqd/1
8 root RT -5 0 0 0 S 0 0.0 0:00.00 watchdog/1
9 root 15 -5 0 0 0 S 0 0.0 0:00.02 events/0
10 root 15 -5 0 0 0 S 0 0.0 0:00.02 events/1
11 root 15 -5 0 0 0 S 0 0.0 0:00.00 khelper
62 root 15 -5 0 0 0 S 0 0.0 0:00.08 kblockd/0
63 root 15 -5 0 0 0 S 0 0.0 0:00.17 kblockd/1
66 root 15 -5 0 0 0 S 0 0.0 0:00.00 kacpid
67 root 15 -5 0 0 0 S 0 0.0 0:00.00 kacpi_notify
196 root 15 -5 0 0 0 S 0 0.0 0:00.00 cqueue/0
197 root 15 -5 0 0 0 S 0 0.0 0:00.00 cqueue/1
199 root 15 -5 0 0 0 S 0 0.0 0:00.00 ksuspend_usbd
205 root 15 -5 0 0 0 S 0 0.0 0:00.00 khubd
I tried to kill all "nobody" process, and then is everythin OK but after 5 minutes is happening again, the worst thing is that every minute this cpu usage is increasing untillall my web sites crashes (just website , shoutcast service works fine)
the process os "nobody" is: /usr/local/apache/bin/httpd -k start -DSSL
Code:
[root@serwer /]# httpd restart
/usr/sbin/httpd restart: httpd not running, trying to start
/usr/sbin/httpd restart: httpd could not be started
Code:
[root@serwer /]# httpd status
Looking up localhost
Making HTTP connection to localhost
Alert!: Unable to connect to remote host.
lynx: Can't access startfile [url]
New to Apache Web Server. Built the self signed certs, and tried to bring up server after configuring the httpd.conf file.I also included the section of the error log file that is pertinent.httpd.conf removed, use past bin.
View 1 Replies View Related[root@server1 ~]# service httpd start
no listening sockets available, shutting down
Unable to open logs
CENTOS Enterprise 4.6 i686 on standard - WHM X v3.1.0
apache 2.2
httpd.conf file was empty... I'm recompiling right now.
httpd server configuring.I share my files only from local disks but some of my resources are located on truecrypt secure volume.When I want to start my server now I have to mount my all drives first to have all path active. Otherwise httpd doesn't start with 'invalid path' error.My question is, how to force httpd starting without path or drive exists checking?
Platform Windows7 x64
Apache version 2.4.3
Package Xampp
My server is very slow, High CPU, High Apache load....
I got a new server and i had setup a script on it.
No server tweaks has been done.
I run a filehosting site so it gets very slow when traffic is high.
Here is some info:
Code:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
4135 filedip 19 0 12812 8892 2212 R 18.9 0.4 0:00.74 index.cgi
3140 apache 15 0 22624 9604 2768 S 17.3 0.5 1:54.45 httpd
3556 apache 15 0 22792 9592 2768 S 17.3 0.5 0:43.53 httpd
3739 apache 15 0 22672 9556 2744 S 17.3 0.5 1:09.67 httpd
3884 apache 15 0 22664 9560 2744 S 17.3 0.5 0:23.98 httpd
3474 apache 15 0 22672 9572 2756 R 16.1 0.5 1:34.09 httpd
3548 apache 15 0 22792 9560 2740 S 16.1 0.5 1:33.67 httpd
3991 apache 15 0 22640 9588 2756 S 15.1 0.5 0:22.44 httpd
3475 apache 15 0 22672 9580 2756 S 13.2 0.5 1:31.42 httpd
3493 apache 15 0 22664 9592 2768 S 12.9 0.5 1:15.52 httpd
3769 apache 15 0 22664 9580 2756 S 12.9 0.5 0:45.19 httpd
3638 apache 15 0 22664 9576 2756 S 12.3 0.5 0:32.82 httpd
3724 apache 16 0 22664 9580 2756 R 12.3 0.5 0:34.25 httpd
3626 apache 15 0 22672 9580 2756 S 11.7 0.5 0:52.54 httpd
4023 apache 16 0 22792 9560 2740 R 11.3 0.5 0:13.16 httpd
3882 apache 15 0 22664 9576 2756 S 10.1 0.5 0:38.56 httpd
3005 apache 15 0 22672 9588 2768 R 9.1 0.5 2:06.61 httpd
I dont know much about server managment...
how to fix this so server doesnt go slow?
I've search around quite a bit, but I can't seem to find a clear answer.
I've turned on mod_status, and on the status page, I'm worried Apache is being overworked.
Code:
CPU Usage: u334.883 s94.5703 cu177.422 cs0 - 43.3% CPU load
54.1 requests/sec - 181.5 kB/second - 3435 B/request
178 requests currently being processed, 52 idle workers
__RKKK__K___KW_CKCKC_KC_KCK_KKCCK_K_KKWCKCCKCWKKKCRCKKKK._K_C__K
KK_K__K_K_KK_C_KKKKK_KKW.KRCKWKKW_CCKWKRK..___KKCKKKKKKKR_KKKW.W
_KK_____RKRC.KKCKCCRK__CK._KWCK_KKRCCK.C__CR.CWCCKCRKK.RR.K_CR_K
C_KW._._K.K..KKK_KRCCKWKK__..R.CRWCKK.KCK.K.KK_K....._...KK.....
......KC.CK...C...CK......W........C............................
................................................................
................................................................
................................................................
At the time I took this particular snapshot, it's not near its peak... it's not uncommon for the CPU load approach 60%. Reading around, it seems the CPU load should normally be under 1% (such as 0.0139% or what-not). Is this true?
The weird thing though... I have no idea where that number is coming from, because according to "top", the CPU is actually 90% idle.
Here is my Apache MPM settings:
Code:
ServerLimit 512
<IfModule mpm_prefork_module>
StartServers 10
MinSpareServers 15
MaxSpareServers 50
MaxClients 1024
MaxRequestsPerChild 0
</IfModule>
I actually just raised the MaxClients from 512 to 1024 because I was hitting a constant cap of 40 requests/sec... and I was worried it was going to bottleneck. When I raised that value, the max requests per second now seem to be freed up.
If the actual CPU of the server is 90% idle... am I okay? Anyone know where Apache's getting the CPU Load info from?
if anyone has had problems with high cpu load after upgrading APACHE to 2.2.8. We were running 1.3.5 with comfortable CPU loads of 2-10 on dual Xeon 2.8's. Now the loads are 20-70 with most of the CPU being used by many many httpd processes. I've heard that APACHE v2 does consume more resources, but am wondering if there was a problem with the build, or is it that much more demanding.
OTHER INFO:
WHM:11.15
CPANEL: 11.18
OS: CENTOS 4.6
KERNAL: 2.6.20.4-ts.grh.mh.i386
We've a quadcore Xeon and start to have issues with its very high cpu load (which is made by apache & mysqld,).
So here is ps aux:
Code:
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.0 0.0 2040 620 ? Ss Jul13 0:13 init [3]
root 2 0.0 0.0 0 0 ? S Jul13 0:00 [migration/0]
root 3 0.0 0.0 0 0 ? SN Jul13 0:00 [ksoftirqd/0]
root 4 0.0 0.0 0 0 ? S Jul13 0:00 [watchdog/0]
root 5 0.0 0.0 0 0 ? S Jul13 0:00 [migration/1]
root 6 0.0 0.0 0 0 ? SN Jul13 0:00 [ksoftirqd/1]
root 7 0.0 0.0 0 0 ? S Jul13 0:00 [watchdog/1]
root 8 0.0 0.0 0 0 ? S Jul13 0:00 [migration/2]
root 9 0.0 0.0 0 0 ? SN Jul13 0:00 [ksoftirqd/2]
root 10 0.0 0.0 0 0 ? S Jul13 0:00 [watchdog/2]
root 11 0.0 0.0 0 0 ? S Jul13 0:00 [migration/3]
root 12 0.0 0.0 0 0 ? SN Jul13 0:00 [ksoftirqd/3]
root 13 0.0 0.0 0 0 ? S Jul13 0:00 [watchdog/3]
root 14 0.0 0.0 0 0 ? S< Jul13 0:00 [events/0]
root 15 0.0 0.0 0 0 ? S< Jul13 0:00 [events/1]
root 16 0.0 0.0 0 0 ? S< Jul13 0:00 [events/2]
root 17 0.0 0.0 0 0 ? S< Jul13 0:00 [events/3]
root 18 0.0 0.0 0 0 ? S< Jul13 0:00 [khelper]
root 19 0.0 0.0 0 0 ? S< Jul13 0:00 [kthread]
root 25 0.0 0.0 0 0 ? S< Jul13 0:01 [kblockd/0]
root 26 0.0 0.0 0 0 ? S< Jul13 0:01 [kblockd/1]
root 27 0.0 0.0 0 0 ? S< Jul13 0:01 [kblockd/2]
root 28 0.0 0.0 0 0 ? S< Jul13 0:03 [kblockd/3]
root 29 0.0 0.0 0 0 ? S< Jul13 0:00 [kacpid]
root 122 0.0 0.0 0 0 ? S< Jul13 0:00 [cqueue/0]
root 123 0.0 0.0 0 0 ? S< Jul13 0:00 [cqueue/1]
root 124 0.0 0.0 0 0 ? S< Jul13 0:00 [cqueue/2]
root 125 0.0 0.0 0 0 ? S< Jul13 0:00 [cqueue/3]
root 128 0.0 0.0 0 0 ? S< Jul13 0:00 [khubd]
root 130 0.0 0.0 0 0 ? S< Jul13 0:00 [kseriod]
root 205 0.0 0.0 0 0 ? S Jul13 0:46 [pdflush]
root 207 0.0 0.0 0 0 ? S< Jul13 0:43 [kswapd0]
root 208 0.0 0.0 0 0 ? S< Jul13 0:00 [aio/0]
root 209 0.0 0.0 0 0 ? S< Jul13 0:00 [aio/1]
root 210 0.0 0.0 0 0 ? S< Jul13 0:00 [aio/2]
root 211 0.0 0.0 0 0 ? S< Jul13 0:00 [aio/3]
root 383 0.0 0.0 0 0 ? S< Jul13 0:00 [kpsmoused]
root 433 0.0 0.0 0 0 ? S< Jul13 0:00 [ata/0]
root 434 0.0 0.0 0 0 ? S< Jul13 0:00 [ata/1]
root 435 0.0 0.0 0 0 ? S< Jul13 0:00 [ata/2]
root 436 0.0 0.0 0 0 ? S< Jul13 0:00 [ata/3]
root 437 0.0 0.0 0 0 ? S< Jul13 0:00 [ata_aux]
root 443 0.0 0.0 0 0 ? S< Jul13 0:00 [scsi_eh_0]
root 444 0.0 0.0 0 0 ? S< Jul13 0:00 [scsi_eh_1]...
They don't appear for some time if I kill them (a day or more). But it repeats again and again. One day there were 8 similar processes in total in max which used all 4 cores at 100% (and even ssh console was extremely slow to do something there).
I think that somebody is trying to make a small attack of some sort but I need to check it first. I tried to look at apache logs but there were too many posting requests from different IPs and no dublicates for little period of time so I had no success.
Anyway, that script worked for us for 4 years already and we didn't have any problems with it even on our old single core P4 2.8 ghz.
way to make sure is this an attack of some sort or just this script doesn't work correctly on our new machine?
Are there any ways to get IPs of visitors who are running posting.php with CPU overloading?
been checking out this site for a while and finally decided to register because I have a problem. Also hope this is the correct forum for this topic, sorry if it isn't.
So I have a problem with Apache. One of the sites that I run/host has a moderately large vBulletin board, and Apache just seems to eat up the CPU. Load averages have shot up between 20-30 and I've seen it as high as 80. Apache and MySQL are optimized already, I'm using suPHP for security because there are other sites on this box.
The funny thing about this is that it only started happening about a week ago. After checking for rootkits and all that garbage, I reinstalled the OS just to be on the safe side. Everything comes back clean still. I also got fed up and hired Platinum Server Management for a month, to see if they could find a solution (and I've been interested in reselling their services, but that's not relevant). So far the only thing they can come up with is disable suPHP, which isn't an option. I do realize that suPHP is ~20-25 times slower than mod_php, but what totally baffles me is that it worked beforehand and started going all crazy like this. I did try running the site using an dso configuration, the load did drop, but nothing to be proud of.
This site, and the server overall hasn't had any increase in load, I've held off putting new accounts on it until I get this fixed.
In the meantime, I have said forums running on lighttpd, which lowered the load. (Also writing a tutorial on having lighty work with cPanel)
write a cronjob so that apache will restart when the server reaches a load of 20.
View 13 Replies View RelatedI deal with a server that gets positivey slammed once a week for a few months per year. I'd tell you how many hits we got tonight, but I'm still waiting for AWSTATS to chew through the 2gb access_log file...
Tonight, I made some changes that SEEM to work, but I"mn not sure what the long-term effects could be. If we have any apache experts on the forums, I'd really like to bend your ear for a few to see what you know.
Obviously, with PHP, we're limited to prefork MPM.
First of all, I dropped Timeout from 300 to 120. That should be MORE than enough time to know that we've timed out. Then I dropped KeepAliveTimeout to 5 from 15.
Here's the radical one. Watching the process list and the load, it seemed that load spiked when the processes hit their end of useful life and respawned. Duh. This was happening every four seconds at the load we were under. MaxRequestsPerChild was set to 10,000. I upped this to 80,000 over a period of hours that we were under the load. I didn't see any significant memory leakage, but it's the change I'm worried about the most. I've seen Apache do some bad things when people allow this to go unlimited, and had always used the relatively low default as a guide.
Besides not loading a bunch of dynamic modules (also done, I usually do this so I'm not worried about it), what else can I do tuning-wise to keep load down? Please note that caching and load-balancing aren't acceptable solutions; I have one server to work with (for now) and the boss says no to caching because of how frequently our information updates. We also have extensive .htaccess files, so there's no LHTTPD in my future.
I cannot start httpd, it say this error:
Quote:
root 24012 1.0 1.3 23640 21996 ? S 09:32 0:00 /usr/local/cpanel/whostmgr/bin/whostmgr ./reshttpd
httpd has failed, please contact the sysadmin.
What can be?
When i go to my website or any page under the domain it's just white. Blank nothing, nada.
"Failed to receive status information from Apache. Unable to connect to local httpd server."
This error message appears when i click on "apache status".
Is this because of the network issue? Or is it because of something I did myself?
I did a google search and came up on some solution, I did them and nothing.
I have a site that is eating up my server resources and need to know what the best solution for this is. I'm thinking of getting another server just for mysql but do not know what specs the server should be to handle the current traffic/database load and have the site run smoothly without slowing down to a snail's pace.
An alternative is to get another server just for the videos being served and leave the database and html on the current server. This is where I'm stuck and don't know what route to take with this.
My current server stats:
Dual Xeon 5130
4GB RAM
250GB
50 Mbit/sec
CentOS 3.9
Website traffic stats:
15,000 visitors/day
150,000 pageviews/day
Serving videos
I've attached screenshots of top and bandwidth usage per day. Hopefully with this information you could tell me if I need another server or if there are any things I can do to the current server to help things move faster.
the error log shows nothing out of the ordinary.. but its taking about 2-3 minutes to start.. which before it would take about 10 seconds.
load is fine on the machine, free mem is fine also.
I trying:
Code:
[root@serwer /]# httpd restart
/usr/sbin/httpd restart: httpd not running, trying to start
/usr/sbin/httpd restart: httpd could not be started
Code:
[root@serwer /]# /etc/rc.d/init.d/httpd configtest
Syntax OK
Help
I noticed high cpu usage on my server since a couple of days ago. I neither updated software nor changed the configurations on the server. The server load was about 0.5 ~ 1.0 but now the average CPU load does not go below 5.0
when I stop apache (1.3) the load goes down and as soon as I start it up the load start increasing.
I checked for DoS but it seems nothing exists of this type.
I tried to follow the access_log file but from the first view nothing is strange there.
I don't have any control panel on the server. just Apache, FTP and SSH installed.
I am running in a High load problem lately. I have one of those cheap 1and1 servers which was running fine until 2 weeks ago. Once I rebooted accidentaly, it did not come back with some unrepairable kernel errors and I had to re-image it.
I chose to reimage the server with CentOS 5, for better support. The new re-image worked fine for some days, at least so I thought and now I am having high loads. The server crashes if not monitored every moment as the load is unpredictable.
Just a restart of the Apache will bring the server back to normality, but I am not sure if it is apache or some other script to be blamed.
I have beeing monitoring through apache server-status, but I cannot organize something unusual in the high load moments.
At this moment for example:
top - 09:00:36 up 15:43, 3 users, load average: 3.57, 4.60, 5.32
Tasks: 144 total, 2 running, 142 sleeping, 0 stopped, 0 zombie
Cpu(s): 10.6%us, 6.5%sy, 0.0%ni, 18.7%id, 63.5%wa, 0.2%hi, 0.5%si, 0.0%st
12:00:29 AM CPU %user %nice %system %iowait %steal %idle
12:10:01 AM all 9.14 0.00 5.52 44.66 0.00 40.68
12:20:14 AM all 6.83 0.00 3.98 27.88 0.00 61.32
12:30:10 AM all 6.44 0.00 4.20 81.25 0.00 8.11
12:40:09 AM all 5.25 0.00 4.09 81.93 0.00 8.73
12:50:15 AM all 5.11 0.00 3.79 90.74 0.00 0.36
01:00:07 AM all 7.22 0.00 4.52 57.11 0.00 31.15
01:10:13 AM all 6.89 0.00 4.01 55.38 0.00 33.71
01:20:14 AM all 4.37 0.00 3.27 41.88 0.00 50.48
01:30:25 AM all 4.26 0.00 3.29 63.42 0.00 29.03
01:40:06 AM all 27.18 0.00 4.75 58.27 0.00 9.80
01:50:03 AM all 29.64 0.00 6.61 51.50 0.00 12.25
02:00:07 AM all 27.00 0.00 8.48 55.49 0.00 9.03
02:10:10 AM all 19.29 0.00 4.97 73.80 0.00 1.94
02:20:04 AM all 37.85 0.00 6.78 40.70 0.00 14.67
02:30:05 AM all 15.65 0.00 4.80 68.47 0.00 11.08
02:40:08 AM all 9.06 0.00 5.60 37.49 0.00 47.86
02:50:07 AM all 5.36 0.00 3.62 42.29 0.00 48.73
03:00:02 AM all 6.05 0.00 4.08 47.27 0.00 42.60
03:10:02 AM all 4.22 0.00 3.68 38.17 0.00 53.93
03:20:02 AM all 4.06 0.00 3.75 41.37 0.00 50.82
03:30:22 AM all 4.42 0.00 3.93 45.25 0.00 46.41
03:40:11 AM all 4.34 0.00 3.95 39.58 0.00 52.13
03:50:02 AM all 4.67 0.00 4.01 32.53 0.00 58.80
04:00:08 AM all 3.72 0.00 3.87 28.40 0.00 64.02
04:10:02 AM all 13.49 0.00 6.58 20.82 0.00 59.10
04:20:01 AM all 6.70 0.00 4.63 6.06 0.00 82.61
04:30:02 AM all 1.44 0.00 1.21 4.75 0.00 92.59
04:40:01 AM all 12.42 0.00 8.12 7.65 0.00 71.81
04:50:02 AM all 1.43 0.00 1.07 4.02 0.00 93.47
05:00:02 AM all 1.60 0.00 1.40 8.62 0.00 88.38
05:10:10 AM all 3.80 0.00 3.02 17.86 0.00 75.32
05:20:06 AM all 5.10 0.00 4.22 23.34 0.00 67.34
05:30:02 AM all 1.54 0.00 1.40 11.22 0.00 85.85
05:40:05 AM all 1.75 0.00 1.89 13.12 0.00 83.23
05:50:12 AM all 2.15 0.00 2.22 18.92 0.00 76.72
06:00:02 AM all 1.92 0.00 2.01 12.87 0.00 83.20
06:10:02 AM all 2.27 0.00 2.16 11.53 0.00 84.04
06:20:03 AM all 3.56 0.00 3.02 25.26 0.00 68.16
06:30:10 AM all 2.66 0.00 2.05 18.13 0.00 77.16
06:40:02 AM all 2.58 0.00 2.25 22.87 0.00 72.30
06:50:02 AM all 2.68 0.00 1.92 15.77 0.00 79.63
07:00:03 AM all 3.06 0.00 2.48 26.01 0.00 68.46
07:10:03 AM all 3.65 0.00 3.20 36.54 0.00 56.61
07:10:03 AM CPU %user %nice %system %iowait %steal %idle
07:20:03 AM all 4.40 0.00 3.28 43.86 0.00 48.46
07:30:02 AM all 4.10 0.00 3.17 31.30 0.00 61.43
07:40:06 AM all 7.67 0.00 3.95 50.79 0.00 37.59
07:50:02 AM all 4.72 0.00 3.11 44.30 0.00 47.86
08:00:03 AM all 5.57 0.00 3.72 47.15 0.00 43.56
08:10:07 AM all 10.66 0.00 3.59 71.62 0.00 14.13
08:20:17 AM all 5.67 0.00 3.42 58.81 0.00 32.10
08:30:10 AM all 11.12 0.00 3.49 76.71 0.00 8.67
08:40:03 AM all 7.00 0.00 3.36 47.94 0.00 41.71
Average: all 7.53 0.00 3.76 38.90 0.00 49.81
Some configurations:
The reimage partittioning looks like this:
/dev/sda1 950M 138M 765M 16% /
/dev/sda5 4.7G 1.9G 2.9G 40% /usr
/dev/sda7 63G 58G 5.0G 93% /var
/dev/sda6 4.7G 23M 4.7G 1% /home
none 1.0G 500K 1.0G 1% /tmp
KeepAlive On
MaxKeepAliveRequests 300
KeepAliveTimeout 15
<IfModule prefork.c>
StartServers 10
MinSpareServers 20
MaxSpareServers 50
ServerLimit 256
MaxClients 180
MaxRequestsPerChild 2000
</IfModule>
<IfModule worker.c>
StartServers 2
MaxClients 150
MinSpareThreads 25
MaxSpareThreads 75
ThreadsPerChild 25
MaxRequestsPerChild 0
</IfModule>
[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
skip-locking
query_cache_limit=1M
query_cache_size=32M
query_cache_type=1
max_connections=250
interactive_timeout=100
wait_timeout=20
connect_timeout=10
thread_cache_size=80
key_buffer=64M
join_buffer=1M
max_allowed_packet=16M
table_cache=512
record_buffer=1M
sort_buffer_size=1M
read_buffer_size=1M
max_connect_errors=10
thread_concurrency=2
read_rnd_buffer_size=1M
myisam_sort_buffer_size=64M
server-id=1
processor : 1
vendor_id : GenuineIntel
cpu family : 15
model : 3
model name : Intel(R) Pentium(R) 4 CPU 2.80GHz
stepping : 4
cpu MHz : 2793.324
cache size : 1024 KB
I have extremely high CPU load average, but when I run TOP command there are not any process with high CPU
CPU load averages:
211.65 (1 min) 227.29 (5 mins) 247.89 (15 mins)
Tasks: 364 total, 2 running, 360 sleeping, 2 stopped, 0 zombie
Cpu(s): 0.0%us, 0.3%sy, 0.0%ni, 0.0%id, 99.7%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 2065460k total, 2012372k used, 53088k free, 60964k buffers
Swap: 2040212k total, 3280k used, 2036932k free, 619900k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
19871 root 15 0 4944 2488 1656 R 1 0.1 0:02.73 top
1 root 15 0 4208 1320 1136 S 0 0.1 0:01.55 init
2 root RT 0 0 0 0 S 0 0.0 0:00.00 migration/0
3 root 34 19 0 0 0 S 0 0.0 0:01.77 ksoftirqd/0
4 root RT 0 0 0 0 S 0 0.0 0:00.01 migration/1
5 root 34 19 0 0 0 S 0 0.0 0:01.16 ksoftirqd/1
6 root 10 -5 0 0 0 S 0 0.0 0:01.04 events/0
7 root 10 -5 0 0 0 S 0 0.0 0:00.65 events/1
8 root 10 -5 0 0 0 S 0 0.0 0:00.00 khelper
9 root 11 -5 0 0 0 S 0 0.0 0:00.00 kthread
35 root 10 -5 0 0 0 S 0 0.0 0:00.15 kblockd/0
36 root 10 -5 0 0 0 S 0 0.0 0:00.00 kblockd/1
37 root 15 -5 0 0 0 S 0 0.0 0:00.00 kacpid
127 root 10 -5 0 0 0 S 0 0.0 0:13.16 ata/0
128 root 16 -5 0 0 0 S 0 0.0 0:00.00 ata/1
129 root 10 -5 0 0 0 D 0 0.0 0:01.15 ata_aux
130 root 10 -5 0 0 0 S 0 0.0 0:00.00 kseriod
158 root 16 0 0 0 0 S 0 0.0 0:00.00 pdflush
159 root 15 0 0 0 0 D 0 0.0 0:01.00 pdflush
160 root 10 -5 0 0 0 S 0 0.0 0:10.68 kswapd0
161 root 11 -5 0 0 0 S 0 0.0 0:00.00 aio/0
So my server is "unresponsive" for abour 18 hours, burst net didnt answer my tickets and I dont know what to do. Ive been with this setup for almost 5 months with no problems, No changes have been made to hardware or software.
Im using Centos 5.0
I have 3.0 pentium with 2Gb RAM... It's the first time I use plesk and this is strange...
top - 23:20:00 up 2:42, 1 user, load average: 9.87, 13.60, 10.29
Tasks: 149 total, 29 running, 120 sleeping, 0 stopped, 0 zombie
Cpu(s): 93.9% us, 6.1% sy, 0.0% ni, 0.0% id, 0.0% wa, 0.0% hi, 0.0% si
Mem: 2064724k total, 1208656k used, 856068k free, 46252k buffers
Swap: 522104k total, 0k used, 522104k free, 354784k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
2354 mysql 20 0 290m 90m 4500 S 12.6 4.5 21:08.10 mysqld
29098 apache 20 0 44140 23m 4352 R 8.0 1.2 0:02.20 httpd
28621 apache 20 0 41560 23m 4360 R 7.3 1.1 0:06.38 httpd
28855 apache 20 0 43992 23m 4356 R 7.3 1.2 0:06.28 httpd
29252 apache 20 0 34672 15m 3344 R 7.3 0.8 0:00.22 httpd
27117 apache 20 0 44632 24m 4692 S 6.6 1.2 0:19.58 httpd
28517 apache 20 0 44196 23m 4500 R 6.6 1.2 0:10.76 httpd
28930 apache 20 0 44272 23m 4504 R 6.6 1.2 0:06.16 httpd
28994 apache 20 0 42756 22m 4396 R 6.6 1.1 0:03.12 httpd
29026 apache 20 0 43584 23m 4700 R 6.6 1.1 0:04.04 httpd
29173 apache 20 0 41212 22m 3708 R 6.6 1.1 0:01.86 httpd
29194 apache 20 0 41056 21m 3756 R 6.6 1.1 0:01.08 httpd
29229 apache 20 0 38132 18m 3668 R 6.6 0.9 0:01.04 httpd
29232 apache 20 0 42876 22m 4344 R 6.6 1.1 0:01.08 httpd
29236 apache 20 0 36708 17m 3728 R 6.6 0.9 0:00.74 httpd
28564 apache 20 0 44300 23m 4712 R 6.0 1.2 0:08.96 httpd
28926 apache 20 0 44228 23m 4408 R 6.0 1.2 0:04.44 httpd
29063 apache 20 0 43588 22m 4336 R 6.0 1.1 0:03.40 httpd
29073 apache 20 0 43944 23m 4388 R 6.0 1.2 0:03.94 httpd
29172 apache 20 0 41056 21m 3752 R 6.0 1.1 0:01.50 httpd
29192 apache 20 0 44704 24m 4408 R 6.0 1.2 0:03.20 httpd
29237 apache 20 0 37100 18m 3660 R 6.0 0.9 0:00.64 httpd
29254 apache 20 0 33432 14m 3340 R 6.0 0.7 0:00.18 httpd
29228 apache 20 0 35056 16m 3656 S 5.3 0.8 0:00.36 httpd
29105 apache 20 0 44744 24m 4452 R 4.7 1.2 0:04.74 httpd
29106 apache 20 0 43908 23m 4548 R 4.7 1.2 0:05.36 httpd
29193 apache 20 0 35124 16m 3652 R 4.7 0.8 0:00.68 httpd
29195 apache 20 0 43324 22m 4332 S 4.7 1.1 0:01.36 httpd
28551 apache 20 0 44240 23m 4596 R 4.0 1.2 0:10.64 httpd
28727 apache 20 0 44012 23m 4492 S 4.0 1.2 0:08.70 httpd
29223 apache 20 0 44048 23m 4428 S 3.3 1.2 0:01.26 httpd
29224 apache 20 0 43236 22m 4328 S 2.0 1.1 0:01.06 httpd
29219 apache 20 0 35108 16m 3648 S 1.3 0.8 0:00.70 httpd
29259 apache 20 0 28860 9816 3296 R 1.3 0.5 0:00.04 httpd
29260 apache 20 0 28464 9476 3284 R 1.3 0.5 0:00.04 httpd
I have some webs with 30K/40K visits each day (it's not too much)... A smaller CPU with 1Gb RAM with cPanel and Apache 1.3 worked perfectly, and now with a bigger computer and plesk I see this...
More info on server-status
Server Version: Apache/2.0.54 (Fedora)
Server Built: Jul 26 2006 11:00:19
Current Time: Monday, 28-Apr-2008 23:40:53 CEST
Restart Time: Monday, 28-Apr-2008 23:25:38 CEST
Parent Server Generation: 0
Server uptime: 15 minutes 14 seconds
Total accesses: 13025 - Total Traffic: 44.9 MB
CPU Usage: u1183.04 s50.79 cu.01 cs0 - 135% CPU load
14.3 requests/sec - 50.3 kB/second - 3616 B/request
34 requests currently being processed, 67 idle workers
C_C_C_____W__C_WC__WCW_W_R______C____W_W_W._W.___W W____WC.CC___W
___W__WC_W__W_________W__..WW_W.___W____._.__..... ..............
.................................................. ..............
.................................................. ........
My server is a dual cpu quad core box. So I see 8 processors in WHM status page. Load is between 1-2 most of the time, sometimes a little bit above 2.
I've never used a dual cpu quad core machine before. When should I start worrying about the server load? When load is above 8 or what?
I have a cheap managed server and support is of little help BUT they will do what I tell them, where should I tell them to start to fix high mysql load?
The server currently has a large IPB forum, a large 4images gallery, and a couple popular Wordpress blogs, and this combo keeps taking mysql down.
What can I do server side to help this? And what shouldn't I do?
I have a server that has server load showing at 25-40 (once it was even 53!), running like that for hours. The server has 4 cpus - and yet the sites on the server seem to run fine when I check them. What I'm wondering is, what exactly is load in this context; and how can load run so high like that without the server crashing?
According to top, the load is caused by httpd processes running under user 'nobody', that often take up double digit CPU percentage.
Does Apache always run under 'nobody'?
Is there any way to trace an httpd processes - which account it's for, or which physical script or URL is calling it?
And for top itself, the TIME field on one server of mine is in the format xx:xx (e.g. 3:25), on another it's TIME+ and in the format xx:xx.xx (e.g. 30:02.77). What exactly does this mean? I would asume it's minutes:seconds and minutes:seconds:hundredths, but while watching top it doesn't seem to correlate with that.
The last 2-3 days my websites on my vps all started acting funny. Randomly throughout the day they will all just stop loading. If you go to one of my sites they will just load and load and load and eventually time out after 5 minutes.
To fix this problem, all I have to do is reset Apache in my WHM. Everything will be fine for awhile until it decides to stop loading again.
I've been talking to tech support for 2 days. They're out of options. They tried banning some IPs as they thought I might be getting hacked. They tried moving me to a different host machine. They've checked all my logs... there are no errors or problems with the vps. The server load is always very low. Memory is always very low.
I run about 15-20 websites on this VPS. Two of them are vbulletin (v. 3.6.4 and v3.6.8) and all the rest are wordpress).
My VPS is hosted with ServInt.
Server Version: Apache/1.3.39 (Unix) PHP/5.2.4 mod_auth_passthrough/1.8 mod_log_bytes/1.2 mod_bwlimited/1.4
mysql (5.0.45-community-log)
I get these emails all day:
httpd failed @ Wed Jan 9 06:10:42 2008. A restart was attempted automagically.
I am completely out of ideas and options. Can anyone think of something? Think outside the box... make random suggestions. I don't care, I need to try something.
I most recently added 3 or so brand new websites to my vps about a week ago. However, they are just running wordpress so I can't imagine they're the problem.