Plesk 11.x / Linux :: How To Stop High Number Of Httpd Processes Crashing Server
May 17, 2014
Our server is running; Plesk 11.0.9 and CentOS 5.7 it has a Q8200 CPU @ 2.33GHz and 2GB of RAM. Now there are just two websites on the server plus a couple of redirects/forwarding domains, although lots of domains are still on the server but turned off in Plesk. Both websites are OSCommerce sites and I just need to keep these sites going until the end of the year when we will switch to our new Joomla based website.
We have seen an increasing number of server crashes and after various checks of the logs, fitting a new BIOS battery, check of the hardware by EasySpace who host the server, installation of ClamAV, LMD and RKHunter (which did find some Trojans and Suspect software), I have traced it down to some external Http activity that is taking all of my CPU time and RAM. Here is a screen capture of the Htop listing and when I killed these processes the CPU and RAM went back to normal. The problem is that I usually have to restart the HTTPD service and sometimes things get so bad that the server crashes and I have to request a power cycle.
a topic long time ago that my server load is frequently high.
I'm talking about something like this Server Load 158.86 Memory Used 28.2 % Swap Used 99.57 %
[url]
The only way to solve this problem is to identify the load earlier and kill all httpd process. What I did was
#killall -9 httpd #killall -9 httpd #killall -9 httpd x 30~40 times until no pid process found & the server load is back to normal.
On previous thread, I tried to update mysql & php and it works,
Right now again I am experiencing high server load again...
I'm very sure it's caused by httpd but I am still unable to find out the real cause of the problem and which account user is the culprit for causing this high load.
Can someone assist me by telling me where/how to begin with?
as many of you know, FreeBSD is a stable system... I have many other FreeBSD servers (with the same kernel as this one) that doesn't have problems but this server keeps rebooting once or twice a day (EVERY DAY)
it's just a reboot... something very very similar to someone pushing the reset button
1) messages, security, auth or dmesg has no entries just before the reset, so the kernel is not getting aware the server is rebooting
2) the server comes back after around 10 minutes (reboot time + fsck)
this is happening for long time, so I compiled a new kernel... and the problem didn't stop
I request the datacenter techs to replace hardwares and they told me everything was replaced: motherboard, CPU, memories... and yesterday also the power suply so I have no other idea on what to do
in fact I have one... setting a nobreak in this server power suply for 2 or 3 days to see if the problem stops, but the datacenter didn't like this idea
I've recently had problems where customers will upload PHP scripts that seem to use alot of CPU. I've got PRM installed but when a PHP script uses a lot of CPU, it doesn't seem to kill the processes or do anything to stop it crashing the server. I've checked the logs of PRM and it does kill some processes that use a lot of CPU/RAM though...
The ideal solution would be for PRM or something else to stop people being able to access the script causing excessive CPU/RAM usage. Even suspending for the reason of using excessive CPU/RAM would be sufficient.
For those interested the OS is CentOS 5.3 with cPanel 11, Apache and the latest PHP 5. Average load is always between 0.50 - 1.90.
My server had been crashing for while with Blue Screen of Death (BSOD) and bug check error code as Stop 0x00000050 PAGE_FAULT_IN_NONPAGED_AREA.It would literally stop by business till I reboot it again.So I tried pull up all information I could get and fix this.
Here is what I found-Possible causes:A faulty driver recently installed Faulty RAM Antivirus Corrupted NTFS file system
I checked the system logs and found errors related to NTFS. Well, my disk needed a chkdsk /r /f to fix this.Ran it at the command prompt and since it required a reboot to fix on the system drive (C, had to reboot. Came back successfully.It has been 14 days and it has not recurred.
I have a server that has server load showing at 25-40 (once it was even 53!), running like that for hours. The server has 4 cpus - and yet the sites on the server seem to run fine when I check them. What I'm wondering is, what exactly is load in this context; and how can load run so high like that without the server crashing?
According to top, the load is caused by httpd processes running under user 'nobody', that often take up double digit CPU percentage.
Does Apache always run under 'nobody'?
Is there any way to trace an httpd processes - which account it's for, or which physical script or URL is calling it?
And for top itself, the TIME field on one server of mine is in the format xx:xx (e.g. 3:25), on another it's TIME+ and in the format xx:xx.xx (e.g. 30:02.77). What exactly does this mean? I would asume it's minutes:seconds and minutes:seconds:hundredths, but while watching top it doesn't seem to correlate with that.
Apache keeps increasing processes and when it reaches 256 value, it crashes. These are the values from our conf file. This doenst happen everyday, it occurs once or twice a week and at random timing.
Timeout 120 KeepAlive Off MaxKeepAliveRequests 100 KeepAliveTimeout 15
I enabled server-status and found request for index.html, is holding the processes. Only thing i see in access logs is ""OPTIONS * HTTP/1.0" 200 - "-" "Apache (internal dummy connection)""
I have a brand new A8i server from 1&1 that has crashed twice today and could only be rebooted through the 1&1 control panel (not Plesk). Out of the blue the server is not accessible from the Plesk CP and is not responding to pings. I have looked through the logs and do not see anything that stands out (I am not very Linux savvy). Is there somewhere specific that I can look that would tell me why it is locking up/crashing? I have only had the server about 2 weeks and am only running email on it (no websites besides webmail and Plesk CP). It is possible that there is a hardware issue, but I cannot have 1&1 check it until after hours tonight.
Server is an 8 core Intel Atom 2.6GHz with 8GB RAM.
My Customers and I have Problems connecting to IMAP-Server. By moving through IMAP Folders I get the Massage "Unable to connect to your IMAP server. You may have exceeded the maximum number of connections to this server"
I know this Article: [URL] ... and all the other related to this issue.
I have small issue with load speed of pages when number of apache running processes goes over +-310, during peak traffic of day. IE: They load really slow.
Here's pic of what I mean:
=> [url]
Only thing that fixes it temporarily (at least until peak traffic ends) is to stop either httpd or mysql for several seconds, as everything cools down, then start it back up. Stop/starting mysql usually has longer terms temporary fix.
Anyway, would you have any ideas on what to configure or change within the system in order to keep the processes under 300?
The load is normal. Even at load 40, pages are blistering fast, as long as processes are under 300. So it's got nothing to do with load.
I recently modified my loadavg script to store in a database the output of a top command if there's ever server loads of over 1. Overnight I've had 12 such times logged to a database.
Upon inspecting things (I was expected there is a recurring problem), the top command reveals that there are always three queries running together which take over 30 seconds each, and take up ~9% of memory each:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 26119 nobody 25 0 73492 38m 4792 S 0 8.5 0:30.00 httpd 7313 nobody 25 0 76716 42m 4992 S 0 9.5 0:29.99 httpd 14212 nobody 19 0 70688 39m 4844 S 0 8.8 0:30.03 httpd
Is there a command that will tell me exactly what these processes are? Like in WHM's "CPU/Memory/MySQL Usage" whereby it says what account these httpd processes are coming from, and the actual page they are coming from as well?
If I could log these details (i.e. account and page these are coming from) along with the output of the top command, I can hopefully troubleshoot where this problem is coming from.
so here is a simple question that i just can't seem to figure out.. when i run the command top or ps -auxw.. they show the httpd processes as the command httpd or /usr/sbin/httpd, but how do i know what file that is? is there anyway to find out what file that is actually getting executed or served?
It's been happening on both my old and new server, was really hoping this would be fixed.
Basically, the httpd/apache service crashes or goes down making domain names and sites unable to load; although cPanel/WHM still accessible. This would go on for about 5 minutes OR until I restart httpd in root.
sites simply do not load or just timeout after a period of time.
This has become extremely annoying for both me and my clients because it makes it seem like we have downtime.
I got 5 wordpress and 5 statics website on this server and 100 visitors by 24H00 each day.
Question 1 : Why the memory is so low and the swap so high ? Question 2 : Why i don't find high usage process in top command ? Question 3 How can i resolve this problem ?
2014-06-24T10:01:15+11:00 INFO (6): Server health parameter "Services > Panel CPU usage" changed its status from "green" to "red". 2014-06-24T10:21:16+11:00 INFO (6): Server health parameter "Services > Apache CPU usage" changed its status from "red" to "green".
I have a Centos 6 server with Heart Internet, who yesterday had an issue with the issue and once back online have been struggling til now getting the mysql service working correctly again and have now refused to provide any further support and redirected me here.
I was following this article here: [URL] ....
I've gotten to point 4. "~# mysqldump -uadmin -p`cat /etc/psa/.psa.shadow` -A | sed '1iSET FOREIGN_KEY_CHECKS = 0;' > /root/dumpall.sql" however it's saying "got error: 145: table './avenue141_dabs/av_options' is marked as crashed and should be repaired when using LOCK TABLES and any attempts made to get into the DB to repair is giving me "error: 1045: Access denied for user 'root'@'localhost' (using password: NO)
This issue is urgent as we have a few websites on this server which use mysql databases for their website.
