I'm having an issue with what I believe is Apache. Things will be running nicely, then all of a sudden things will just 'blow up'. Apache seems to fork a bunch of new processes, and it starts ripping into the memory. I've had it essentially eat through all the free memory and swap, and the system will then freeze up as there's no more memory. This is a CentOS server running CPanel, the configuration had been up and running for a long time without problems then this suddenly started happening.
I have a handful of machines running Rails on CentOS 5, each with 8GB RAM. They're monitored with Nagios, so I get paged if we exceed 80% memory usage usage or 2% swap. FWIW, they're primarily dual x. dual-core Opterons, and we're using PAE.
The problem is that I keep getting paged in the middle of the night for situations where we're using 2-20% swap yet there are still several gigs of RAM free.
On some level, this makes sense -- if nothing is using a big chunk of RAM, we might as well move it out to disk. On the other hand, when there's still several gigs of RAM free, I don't see any point in bothering, especially when it causes me to get paged at 2am.
So my question is threefold:
Is there any easy way to see what's using swap? top can be made to show a swap column, but it's not what you'd think it is. (For example, most processes show as using lots of swap, and the sum vastly exceeds the actual size of our swap partition.) Can I get a list of what processes are actually swapped out to disk?
Is this a problem other people have? Are others just less aggressive in getting alerted at swap usage, or does this not happen to other people? Our setup is probably not too common (Opterons, PAE, adn 8GB on an older CentOS release?), which makes me think there's a small chance a bona fide bug plays into this somehow.
Is this a case for playing with the swappiness sysctl to make the machine less obsessed with swapping things out to disk? And if so, is there any good documentation on what the 0-100 value actually means, beyond the one sentence explaining that 100 is most likely to swap and 0 is least? Does the current setting of 60 "mean" anything, for example? 98% of the stuff online is just a mirror of the e-mail thread in which Andrew Morton and others go back and forth about what setting is ideal, which doesn't help and is tiring to read over and over.
I have a server that I have had for a little while now that was runnning perfectly fine. All of the sudden it started just using up a ton of the memory and started using up swap. This has caused the server to slow down and e-mail to stop working. I would upgrade to more memory but I suspect it is someone on the server doing something they shouldn't be. My only reasoning for this is due to the fact that this problem just suddenly arised. I cannot for the life of me determine where this ram usage is coming from.
Here are a few of the errors I have recently got just while logging in..
RIGHT after logging in... id: cannot find name for group ID 0 id: cannot find name for user ID 0 [root@www root]# top top: Unknown terminal "xterm" in $TERM [root@www root]# killall -9 spamd bash: /usr/bin/killall: Too many open files in system
I have added 2 screen shots as well. First one (top1) is what happens once I can get the command "top" to actually run (normally takes quite a few tries). Second attachement (top2) is after I sorted by mem usage...
I have a linux server that is only hosting one site , and although the httpd and mysql are working well , then the server load is high and when I restart the server the server load is high till the swap accelerates to 100% and then it uses the memory alot.
I have a dedicated server running CentOS which has been been running fine for a while now.
On the odd day Apache used to hang, I presume due to load so I installed SIM (System Integrity Manager) to check the status of httpd and to re-start when necessary.
Every now and again SIM would email saying that apache was re-started, no big deal.
Then this morning I got a email from SIM saying apache was down but couldn't be re-started. I then logged into the shell and tried to do a: service httpd restart
I then got a message saying something like: could not bind to port 80. I then tried to kill all httpd processes and re-start apache but the problem persisted. I then just rebooted my server and apache started fine.
Since then I have had hundreds of emails from SIM saying that it has had to re-start apache. Somehow it is detecting it as down and re-starting it. Apache seems to be up and fine. Re-installing SIM does not help.
The weird thing is when I logged in using Putty I got the "server's host key did not match the signature supplied" message like something had changed.
Rkhunter does not return anything just a couple of warnings.
Does anybody know what would cause sim to constantly detect Apache as down in this situation and also how this could happen with no changes being made?
For the last week, my vps (directadmin on centos on xen) has become unresponsive every three days at approximately 5 AM, and I'm baffled about why. It doesn't crash, but it stops accepting incoming connections until it's rebooted.
I'm not sure how to troubleshoot this. /var/log/messages gives no useful information. There are no unusual cronjobs running at that time.
Anyone heard of anything like this before or have any advice about how to troubleshoot?
We had a Coldfusion site running MySQL without issues. We are in the process of doing some upgrades and as part of that we recently switched over to MSSQL 5. Ever since that happened, we will have random page hangs. This usually happens on pages with more complex queries, but it is very random. There is no timeout error on the server, no page cannot be displayed errors, no errors of any kind, the page will just hang. We have been monitoring our server resource monitor and we can see when the page 'gives up', we will go from 100% CPU down to almost 0, and then the page simply never recovers. Does anyone have any idea what might be going on? When there is no error message of sorts it makes it very problematic to troubleshoot.
I installed latest kernel ( 2.6.24.4 or 2.6.24.3 ) with grsec which either panics or hang on reboot before reaching boot prompt, just when /sbin/init runs. Experienced the same with grsec patched kernel 2.6.22.9 on several servers with the same hardware.
On server with AMD Operton, I was able to successfully compile 2.6.24.4 with grsec and boot into.
Server hardware details --------------------------
[root@server kernels]# cat /proc/cpuinfo processor : 0 vendor_id : GenuineIntel cpu family : 6 model : 15 model name : Intel(R) Xeon(R) CPU 5148 @ 2.33GHz stepping : 6 cpu MHz : 2333.469 cache size : 4096 KB physical id : 0 siblings : 2 core id : 0 cpu cores : 2 fdiv_bug : no hlt_bug : no f00f_bug : no coma_bug : no fpu : yes fpu_exception : yes cpuid level : 10 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe nx lm constant_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr dca lahf_lm bogomips : 4670.51
processor : 1 vendor_id : GenuineIntel cpu family : 6 model : 15 model name : Intel(R) Xeon(R) CPU 5148 @ 2.33GHz stepping : 6 cpu MHz : 2333.469 cache size : 4096 KB physical id : 0 siblings : 2 core id : 1 cpu cores : 2 fdiv_bug : no hlt_bug : no f00f_bug : no coma_bug : no fpu : yes fpu_exception : yes cpuid level : 10 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe nx lm constant_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr dca lahf_lm bogomips : 4667.16
[root@server kernels]#
[root@server ~]# uname -a Linux server.name 2.6.19.2-grsec #1 SMP Tue Jan 15 07:08:50 CST 2008 i686 i686 i386 GNU/Linux [root@server ~]#
[root@server ~]# cat /etc/modprobe.conf alias scsi_hostadapter usb-storage alias eth0 e1000 alias eth1 e1000 alias scsi_hostadapter1 aacraid alias scsi_hostadapter2 ata_piix alias scsi_hostadapter3 usb-storage alias usb-controller ehci-hcd alias usb-controller1 uhci-hcd [root@server ~]#
I am using apache proxypass to proxy to 4 different ports (java/jsp applications) with the following configurations (for each):
<VirtualHost *:80> ProxyPreserveHost On ProxyRequests Off ServerName *****.org ProxyPass / http://*****.org:8080/ retry=0 ProxyPassReverse / http://*****.org:8080/ </VirtualHost>
Everything seems work fine until an awkward delay / connection timeout occurs. The web application hangs for about 5 to 15 seconds and starts working normally after that period. It's certainly not the application's issue because everything is fine when using IP address to access it.
When I enabled the debug log in apache, I found out that every time the server seems to hang, the following error appears in the apache error log and the number of apache child processes keep on increasing (ps aux | grep apache).
[Thu Apr 11 10:20:41 2013] [debug] proxy_util.c(1818): proxy: grabbed scoreboard slot 4 in child 13208 for worker proxy:reverse [Thu Apr 11 10:20:41 2013] [debug] proxy_util.c(1837): proxy: worker proxy:reverse already initialized [Thu Apr 11 10:20:41 2013] [debug] proxy_util.c(1934): proxy: initialized single connection worker 4 in child 13208 for (*) [Thu Apr 11 10:20:41 2013] [debug] proxy_util.c(1818): proxy: grabbed scoreboard slot 3 in child 13208 for worker http://*****.org:8080/ ..... Also, the access log (access.log) completely stops during that hang period.
This debug message keeps on repeating as long as Apache/web-application is not responding. The application starts to work normally after that duration and the number of apache child process decrease to the normal number.
Apache version : 2.2.16 (Debian) OS : Debian GNU/Linux 6.0 root@server:~# apache2 -V Server version: Apache/2.2.16 (Debian) Server built: Nov 30 2012 08:58:36
The buffer has 2375 MB of free ram. However this is a heavy mysql server. I want the empty ram be used by mysql so it becomes faster.
We are currently moving to a 8GB ram server, and the free ram will be around 7 GB if we use the exact same configuration. I want the free ram to be 1 GB at most, why pay for 8 GB ram if I'm going to use a fraction of it?
What optimizations should I attempt?
I know about harddisk I/O bottlenecks, I have two SATA drives in the system, and a SAS drive dedicated to mysql, that's all we can afford currently.
All mysql databases are for SMF forums, and currently using MyISAM tables. Switching to another storage engine is a possibility, if it won't create problems in restarts and hot backups.
I run a web design business in South Florida. I currently have an account at ThePlanet to allow me to provide web hosting services to my customers... this is what happened last night.
I received a phone call from an old customer asking me if I was considering changing my computer, i said "well, maybe". My customer then asked me to go an pickup a present. The present was a DELL PowerEdge 2950 server with 6 SCSI (SAS) hard drives 133 gigs each, 2 64-bit Intel-Xeon Quad-Core processors, 4 Gigs FBD RAM, server include an Integrated Intelligent Platform Management Interface (IPMI) 2.0 management controller (with SMASH Command Line Protocol), plus an optional DRAC 5 remote management card with continuous video and Active Directory integration for robust remote server management, Red Hat Enterprise edition with 3 years of support, mounting rack, wheels and cables.
What a present huhh? OK, I'm really excited cause I know this is a top of the line server but i also know that this isn't that easy. I guess if i plug the server to the Internet it will load the IPMI management Controller and try to connect to DELL to activate the product registration or something. My customer said that he was paying a favor he owed me and that the server was too much for him. He got it on an erroneous delivery along with a bunch of boxes and he wanted me to have it since he doesn't know what to do with it. This is not like he did something wrong, he actually asked the delivery guy and his answer was to wait two weeks to see if someone claims it... after the two weeks he said that the package was already paid up to that point by the insurance and if nobody has claimed it means that the package was his possession.
Since i manage remotely my host through a control panel i don't have enough experience to build the infrastructure to provide the connection, so here are my questions...
I guess (i don't know you can correct me) that all software and licenses are pretty much unusable, right?
I cannot even think in keeping the Redhat, right?
What else should I need besides the broadband connection in order to run it at home?... i mean router type-brand-specs, firewall type-brands-specs, etc.
Do you know of any good-cheap-quality High speed connection that i can use on south Florida to provide connection to the server?
Simplehelix offers a 128/256 bit RapidSSL Certificate for just $99.95/year. In addition, this certificate comes with a year of dedicated IP address with free installation.
im looking a free hosting server, let me explain what im looking for, i have some free softwares on my hard drive, i want to host them so other people also downloads these and its give me also a web location to store them in one place, for this im looking a free hosting server which support a great amount of web space and bandwidth.
ive created myself a little test site and i mananged to host it with a server thats sitting behind a nat router by forwarding port 80 to it, this site can be accessed from outside my home network by entering the ip address of my router into internet explorer.
then i create myself a free host at no-ip.com and i used the .servegame.com option so its MySite.servegame.com.
it succesfully resolves to my ip becuese when i ping it from the wndows command line it pings the IP of my router.
but for some reason it cant be used to acces my site through a web browser.
the only thing i can think of is that it has somehhing to do with the "domain name" and "server name" fields that you have to fill in while installing apache although no matter what i put into those fields i can stil access my site using my routers IP.
What would cause a Linux server to run out of swap? Would it be a memory leak? This happened today to my server and it had to be forcefully rebooted by the data center.
