I do a ps aux, and get a bunch of:
/usr/sbin/httpd
Each one takes up like 4% of the available ram - and when the ram is gone, the server dies (it doesn't have a swap file - half the time you can't even log in to it), and you have to reboot Apache.
I thought of limiting maxchilds, but would that break something else?
Should I just make a swap file? Will that defeat the point of creating child processes?
I am also seeing similar issue "AH02001: Connection closed to child with standard shutdown "
Following are the version information
Server: Apache/2.4.6 (Unix) OpenSSL/1.0.2a
I have some problems with apache.
As you can see below, long-live httpd processes use a lot of CPU / Ram.
Usually this processes caused high LA - 8-12.
Code:
%CPU CPU NI S TIME COMMAND
0.2 - 0 S 00:00:27 nginx: worker process
0.3 - 0 S 00:00:29 nginx: worker process
0.3 - 0 S 00:00:30 nginx: worker process
3.3 - 0 S 00:04:09 /usr/local/apache/bin/httpd -k start -DSSL
3.4 - 0 S 00:04:16 /usr/local/apache/bin/httpd -k start -DSSL
3.5 - 0 S 00:05:50 /usr/local/apache/bin/httpd -k start -DSSL
3.6 - 0 S 00:05:40 /usr/local/apache/bin/httpd -k start -DSSL
3.6 - 0 S 00:05:16 /usr/local/apache/bin/httpd -k start -DSSL
3.6 - 0 S 00:06:04 /usr/local/apache/bin/httpd -k start -DSSL
3.7 - 0 S 00:06:06 /usr/local/apache/bin/httpd -k start -DSSL
3.7 - 0 S 00:00:55 /usr/local/apache/bin/httpd -k start -DSSL
3.7 - 0 S 00:06:10 /usr/local/apache/bin/httpd -k start -DSSL
3.7 - 0 S 00:06:10 /usr/local/apache/bin/httpd -k start -DSSL
3.7 - 0 S 00:06:10 /usr/local/apache/bin/httpd -k start -DSSL
3.8 - 0 S 00:06:12 /usr/local/apache/bin/httpd -k start -DSSL
3.8 - 0 S 00:05:40 /usr/local/apache/bin/httpd -k start -DSSL
3.9 - 0 S 00:06:28 /usr/local/apache/bin/httpd -k start -DSSL
I have small issue with load speed of pages when number of apache running processes goes over +-310, during peak traffic of day. IE: They load really slow.
Here's pic of what I mean:
=> [url]
Only thing that fixes it temporarily (at least until peak traffic ends) is to stop either httpd or mysql for several seconds, as everything cools down, then start it back up. Stop/starting mysql usually has longer terms temporary fix.
Anyway, would you have any ideas on what to configure or change within the system in order to keep the processes under 300?
The load is normal. Even at load 40, pages are blistering fast, as long as processes are under 300. So it's got nothing to do with load.
Apache keeps increasing processes and when it reaches 256 value, it crashes. These are the values from our conf file. This doenst happen everyday, it occurs once or twice a week and at random timing.
Timeout 120
KeepAlive Off
MaxKeepAliveRequests 100
KeepAliveTimeout 15
<IfModule prefork.c>
StartServers 8
MinSpareServers 5
MaxSpareServers 20
ServerLimit 256
MaxClients 256
MaxRequestsPerChild 4000
</IfModule>
I enabled server-status and found request for index.html, is holding the processes. Only thing i see in access logs is ""OPTIONS * HTTP/1.0" 200 - "-" "Apache (internal dummy connection)""
I have 2 almost similar webservers:
debian 3.2, P4 3GHZ, 1 Gb ram, kernel 2.6, apache2, php4
Initial apache2.conf:
Code:
Timeout 4
KeepAlive Off
KeepAliveTimeout 2
ServerLimit 1500
<IfModule prefork.c>
StartServers 10
MinSpareServers 10
MaxSpareServers 15
MaxClients 1500
MaxRequestsPerChild 3000
</IfModule>
I update the sources.list on server 1 to mirrors of the new debian 4 etc . I run apt-get update and apt-get dist-upgrade . A whole bunch of things get updated (it was long time ago that I did this anyway). After some troubles with /boot/grub/menu.lst the server boots ok, and everything is well. This server used to have loads of 15-25 at peak times, but after the update its running very smooth with loads of 2-3 at the same peak times. I dont know why exactly, as I noticed updates in OS , kernel version (from 2.6.8x to 2.6.18) , apache2 , php (4.4.4-8+etch1) , and I also needed to update eaccelerator from 0.94 to 0.95 .
A few days later I update server 2. Everything seems to go the same, although the kernel version stays at 2.6.8-3-686. I dont think kernel version at start was exactly same at server 1. But the new php version is the same as server 1, and everything else looks the same too.
But when peak times are coming up, this server starts to have troubles. It is quickly rising to total of 200-300 processes , while server 1 always stays stable at 60-70.
Server 2 also reacts slow if I click somewhere on the site. It takes 5-10 seconds to show a new page. However the load stays pretty low at 1-2 . I see no big cpu usage and also no big memory usage. I have the impression that this server 2 is somehow wasting a lot of apache processes and is making things hard for itself without a real reason.
When I check the seperate mysql database server, I also notice a lot of processes.
Around 200-250 whereas it used to be 40-60. Sometimes this adds up so hard, that all webservers are blocked because mysql has too much processes. When I check the mysql connections, I see a few dozen things like 'unauthen ip:port Connect login' just hanging. All of them have the ip of webserver 2. Those extra apache processes are somehow hanging on to the mysql server without really doing something.
I dont know what is happening, but this server is underperforming very badly now. I managed to limit the problem by drastically lowering ServerLimit and MaxClients on webserver 2 , but this is no real solution. The server is still slow, at least now its not bringing down the others.
My question : what should I check for now ? I noticed a different structure in the conf files in debian etch, maybe something new has a bad influence on my old conf files? Is there something wrong with the combination of kernel+php version? I have no idea, please point me in the right direction so I can learn from this.
I'm not sure I understand the server-status page enough to know if this is a problem or not, but I have several processes that seem to run forever, or until I restart Apache. e.g.
Code:
13-1 21045 0/697/4264 W 59.45 19641 0 0.0 43.28 274.97 66.249.66.133 www.example.com GET /wp/2005/01/ HTTP/1.1
19-1 408 1/834/1831 C 83.52 32463 0 14.8 149.66 263.48 66.249.66.133 www.example.com GET /wp/ HTTP/1.1
30-1 14416 0/430/431 W 35.19 13347 0 0.0 37.42 37.44 66.249.66.133 www.example.com GET /wp/category/issues/ HTTP/1.1
They are almost always on a single domain (there's about 100 on the server) that's a Wordpress site. These processes are also almost always a search engine.
On the rare case I see them running on other domains on the same server they're always on Wordpress sites.
The longer the processes run, the more processor/memory they use, the more they slow the server down.
It seems to have just started in the past few weeks, I've had the site there for a couple years.
I'm new to fcgid on Windows. my current set up is as follows;
Windows Server 2008
Apache 2.4.4
PHP 5.3.27 (fcgid)
MySQL 5.6.14
1. Looking at windows task manager, each of my processes have 4 threads each. What does this mean?
2. Does the number of FcgidMaxProcesses x memory_limit = Total amount of memory required?
3. Number of Threadsperchild = number of FcgidMaxProcesses ?
I just want to know how these two work together.
I have just moved my sites from shared host to a vps. After several initial problems (cpanel issue, config not set up correctly, memory spikes and sites down every morning due to backup and stats) I thought everything was going to be ok. hmm.
My server load starts off fine first thing (less than 1) then creeps up nearly 1 per hour, ie been 3.5 hours now and it is at 2.57. Sites are ok at the moment but yesterday load got up to nearly 7 and sites were extremely slow.
CPU usage is HIGH for one site and goes up very quickly throughout the day. Yesterday it reached well over 90%. First thing today it was already amber and showing 14%. It is now 70.54% and shows below it :
Top Process %CPU 80.2 spamd child
Top Process %CPU 79.8 spamd child
Top Process %CPU 79.4 spamd child
I have a ticket open and they (Liquid Web) they are not sure what the deal is, but are apparently monitoring it to see if they can isolate the cause of the problem. But that was a couple of days ago and now the ticket is due for closure.
I telephoned them (expensive as I'm in the UK!) and raised my concerns, but was just told that everything looked ok on the vps, cpu usage was in fact not high and to ignore the warnings.
So, I am posting here to see if anybody can help me get to the root of this.
I understand that spamd child is to do with email / spam assassin?
My problem is that this is the first vps I have had and don't have a clue now where to go or what to do.
Anybody understand about spamd child that can explain it to a poor dumb blonde and how to fix it?
Recently the Apache recompailed with eAccelerator after that the below error log has created and also the Apache got crash when reach high traffic.
[notice] child pid 13013 exit signal Segmentation fault (11)
[notice] child pid 13054 exit signal Segmentation fault (11)
Due to this problem I ran the /scripts/upcp --force and again recompailed the Apache with eAccelerator.
later the signal Segmentation fault error not created but instead of above error the following error has been creating. And also again recompailed the without eAccelerator still the below error creating.
[notice] child pid 763 exit signal Bus error (7)
Server Details.
Apache 2.2.9
PHP 5.2.5
MySQL 5.0.5 (Runing Seprate box)
eAccelerator 0.9.5
I have seen 1 vps provider having very poor ping results in few online ping sites and they have a very cluster slow loading pages as well.
One of my friend has a package with them, the ping results are very poor even for him as well. Just made an traceroute found its on some node1.vpsprovider.com
Ya, one more major important similarity noticed was, the vps provider emails weren't set properly to yahoo mail, and my friend's emails sent from the server to yahoo weren't delivered as well.
So if the vps provider has poor content may be due to firewall or internal settings do the systems under the node also be affected?
I have no problems with my host so nothing to be worried about, but need to help him as he is just starting it out with a cheaper vps
"Too many open files: couldn't spawn child process"
"Too many open files: couldn't spawn child process". This is what I see in the logs.
I have a CGI application which I can't run anymore because it is giving me a 500 Internal Server Error.
Would disabling the logging on all domains solve this issue? (thinking that too many log files are opened)
I did: cat /proc/sys/fs/file-max and in return I got: 372235.
Then I did echo 350000 > /proc/sys/fs/file-max*, because I thought I have file system limitation or something like that.
The problem starts even right after I reboot the server.
If I hit lsof a lot of files are shown.
The server load is tiny so nothing actually happens.
Would editing the httpd.conf or my.cnf files help ? Should I post the prefork and MPM settings here?
What else should I do to have my CGI application running again?
i have 1 VPS with 512 MB Of Ram with direct admin there is 1 website on this server
but much times it is in hang!
i see apache logs there are much look like this error on this :
"child process 28339 still did not exit, sending a SIGKILL"
i had downgraded apache from 2.2 to 1.3 but probelem not solved!
i think there is a problem on Apache!
the title says it all per month. just curious since this is something in europe that gets a price bump when it comes to power and space. yes i got two colo servers in different locations but never have i asked.
View 8 Replies View RelatedI'm running CentOS with Paralells Plesk bundled Paralellls Premium Antivirus (Dr Web). After the latest yum updates DrWeb continously seems to crash and be restarted by the Parallells watchdog. By default there were no logs for DrWeb, but when I enable logging to a file it gets spammed continously with the following error:
Cannot create pipe for communication with scanning childs (Too many open files)and the Drweb process runs at 99% CPU for long periods. This totally fills the disk with logs and I've now disabled logging again and Drweb is back to continously being restarted by the watchdog.
My company ordered server on 23/10/2009 from LimeStone, and they asked for verification hence i given them all the ID that they need. After that, they told me they couldn't accept AMEX and have to pay through PayPal, i have paid since and NOW again they asked for ID again.
Why i have to verify for second time since i have posted all my ID and utility bill for first time? I don't think i need to give so much verification since PayPal has already verified my account as i owned PayPal Verified Business Account, this kind of "repeating" or double verification never happen when i order from ThePlanet/EV1 where i am looking to migrate to Limestone Networks.
This is very unpleasant experience and I don't want to wait for more than a week to get a server.
what happened not sure really, but asked volumedrive to reboot the machine like an hour ago and it has still not come online, anyone know or has experience with VD and how long it usually takes them to reboot machine? Or is there any outages anyone is aware that VD is facing right now ?
Damn i got some upset users already ready to shoot the bunny, baah this proxy hosting is kinda harder than one would imagine
my previous experience has been shared, all operations are done by panels and ftp.
as some budget unmanaged vps is in my budget range (sub $10), i would really like to give it a go. but as i have no experience at all, i m not sure if i can manage it well. so i m here to ask, what do i need to know, what is the skill set to be an unmanage linux vps admin.
i am not a linux expert, but i've been using suse at work for java development, and trying out ubuntu at home. i know some basic skills like getting packages and install them. use pico to edit configurations files.
what i don't know is: what packages to get, and what config files to edit and how.
but as long as there's some tutorial for me to follow, i think i should be able to do it.
I am running phproxy on my dedicated server.
Sometime some process hangs for long.
I want to kill processes automatically when it takes more than 5 minutes.
And is there any tip you will share to optimize my server best for phproxy?
I've seen many posts in the past few months about people under attack who were not able to handle things themselves, and who made statements along the lines of DDoS mitigation services that one has to pay for are too expensive.
First, I will state that my company does offer those services, and they are not cheap. We offer DDoS mitigation services for hosting/colocation/internet providers who can then resell it to their customers. I state this so that you know that I do have a bias here, though everything I state below is fact.
1. There are free open source tools that can help. Apache modules, IPTables scripts that extract info from netstat or syslog, and I know one guy who is puting together a kernel module. Most of these can stop small scale attacks, and are quite interesting to set up - if you like the technical end of things.
2. Most botnets have more than enough zombies to overpower #1 above.
3. If you have a 100 MBPS pipe to the internet, it doesn't take 100 MBPS of traffic to saturate the pipe and take it down. Enough small packets can overload a router's ability to process, and 10-20 MBPS of traffic can take out the router.
4. There may be a few ways to deal with this, though the best in my experience has been to place an intrusion prevention system (IPS) in front of the router. I have a number of friends in the industry who work at companies where malware is analyzed, and where they work with law enforcement to try and identify the attacking parties. This can be a lengthy process and will not often get a site / router back up quickly, though can be very nice in the long term.
5. Not all IPS are equal. I'm not going to name brands, but I've seen one $50,000 box that had gigabit links die after about 80 MBPS of DDoS traffic. If you're looking into IPS, make sure you compare what they actually do, and talk to people who have implemented them.
6. The majority of the IPS that we manage for our customers and that we implement when we have a new customer under attack are from TopLayer. There are three reasons for this; Their IPS actually works the way you would expect it to (the gigabit model can handle a gigabit of DDoS traffic); If there is something that the IPS can't block, we call their dev team who will work with us to figure out a way to block it; And they give us the best deals.
7. Implementing an IPS is not cheap. The suggested retail price for a gigabit level IPS is about $80,000 USD. Consider that a hosting/colo/service provider who has a two gigabit pipe will need two of these.
8. Managing an IPS takes a special skill set. The people with this skill set are usually expensive to hire as employees, and while I've known a few service providers where the chief technical guy (often a partner in the company) has been the one to manage the IPS, this guy has a lot of other important things to do, and doesn't usually want to be woken up at 2am every few days when there's a significant alert from the IPS.
9. Contracting out IPS management and monitoring can run anywhere between $1,000 and $2,000 per month depending on service options, response times, and contract length. This will usually include remote monitoring of the IPS from a security operations center (SOC), and a lot of escalation options on how to deal with attacks.
10. If an attack is using mechanisms that can get past IPS protections (I will not list them here to give people ideas on how to get around IPS protections, though if anyone is in the field and would like to talk about this I'd be more than happy to do so), then there will need to be escalation options at additional fees from other companies who specialize in that particular area. If subscribing to managed services such as #9 above, then these options should be listed with pricing knowledge available to the customer beforehand. In fact, the company offering the managed IPS service should manage the interface between their customer and the escalation company (we certainly do, and that's one of the things that our customers have been very happy about).
11. Considering #7 through #10 above; the cost of buying/leasing an IPS, managing the IPS and/or paying service fees, and escalating technical work in the event that there is something outside the scope of what can be mitigated using the standard tools, it is more than reasonable for a service provider to charge a significant amount of money to their customers for protection readiness, attack mitigation, and emergency setup fees in the event that there is a situation where a customer is being attacked, needs the service immediately, and has not been paying for protection.
12. A service provider will turn off (null route) their customer when the impact of an attack affects the rest of their customer base. If an attack takes out a full 2 GBPS pipe that they have for all their customers, and null routing one customer is the way to keep the other 1,000 customers up, then that one customer will be null routed. It is a sound business decision. In cases like this, there are options for how to deal with that customer, and that customer will have to decide if they are willing / able to pay for said options.
I changed my sites IP address via cpanel, but now it doesn't show up in a web broswer, I check out cpanel's website for information about this issue, and they posted this:
"It may take up to 48 hours for DNS servers to register the change and the site may not be associated with the new IP address during that time period."
is that true? change the sites IP address will take 48 hours to show up? I figured since it was all local IP stuff, it'll be a instant change and working instantly. since the name servers are all the same, ect.
anyone shed some light. I just wanted to know if I should give it a day to start working? or as of right now, I feel like I need to fix a cpanel error... but if its true, then everything is ok
I am running CentOS 5.10 (Final) with Plesk 11.5.30 Update #44 and since a few days have the strangest problem with incoming emails from certain hosts.
The email from known senders produces a relaylock message like:
/var/qmail/bin/relaylock [12345]: mail from 80.80.80.80:59595 (mail.hostname.com)
From what I was able to gather this is regular behaviour with plesk and qmail, indicating that the senders' mailserver has made contact.
Then nothing happens. For hours. 7-10 to be exact. Then the mail is being processed and delivered.
Panel-Version11.5.30 Update #38
BSDebian 7.4
Save Web Hosting Settings takes a long time. The Domains (Apache Server) are not reachable for 36 seconds.
Host will only update DNS in batches, takes several hours for a DNS change, is that normal?
Well today my mail stopped working and it turns out the A record for mail was deleted. How it got deleted I don't know.
I called my host and after speaking to 2 techs, they said that have added my request to a batch, and that would update in a few hours.
I said to him, batch? What can't you do it instantly?
He said thats not the way their DNS works and any DNS change would affect thousands of other sites they are hosting.
Does anyone know what kind of dns system these people could possibly have where they can not reload a single zone?
And by affected, I think he means the sites will go down for a minute or so, while it relaods every zone. Is this really the case, or are these people just idiots?
We had many backups stored on our Plesk 12 server about 51 GB in total.
After reducing the backups back to 10 GB by removing old backups through the Backup Manager.
The issue is that Health Monitor still reported that there was low diskspace displayed by color yellow. And it look Health Monitor over 16 minutes to change the alarm level from Yellow to Green
However the statistics at the specific subscription still present the Backup usage of 51 GB's whilest they are no longer there.
What can i do about this in order to speed up the synchronization?
I'm more of an application programming guy than network/internet guy so excuse any ignorance on my part.
I am currently using shared hosting on an IIS server.
I running SMF Forums and a business on the IIS server. I have a payment system that I've programmed tied into the MySQL SMF database. This payment system uses ASP.net. I'm a .Net programmer. SMF is also being currently ran on the IIS server, and it does ok.
I'm really wanting to start running my forums on a separate Unix Server.
So my question is, can i run my ASP.net scripts on my IIS server that access the MySQL database on a separate Apache server? The Apache server and IIS server would have different domains (I'm guessing that'd be required).
I think this is possible, but wondering what others think. All I should need to do is change some connection strings on my Web.config on my IIS server to point to the new SQL databases.
we plan to buy one server for apashe and one for mysql.
So first server will handle apache + cpanel + exim
Dual Processor Dual Core Xeon 5140 - 2.33GHz (Woodcrest) - 2 x 4MB cache
8 GB FB-DIMM
3 x 73GB 15k RPM SA-SCSI
Linux RHE 4 ES
second will handle only mysql
Dual Processor Quad Core Xeon 5345 - 2.33GHz (Clovertown) - 2 x 8MB cache
8 GB FB-DIMM
3 x 73GB 15k RPM SA-SCSI
Linux RHE 4 ES
Server will be used for forums about 4000, 5000 Simultaneous users.