I installed latest kernel ( 2.6.24.4 or 2.6.24.3 ) with grsec which either panics or hang on reboot before reaching boot prompt, just when /sbin/init runs. Experienced the same with grsec patched kernel 2.6.22.9 on several servers with the same hardware.
On server with AMD Operton, I was able to successfully compile 2.6.24.4 with grsec and boot into.
Server hardware details
--------------------------
[root@server kernels]# cat /proc/cpuinfo
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 15
model name : Intel(R) Xeon(R) CPU 5148 @ 2.33GHz
stepping : 6
cpu MHz : 2333.469
cache size : 4096 KB
physical id : 0
siblings : 2
core id : 0
cpu cores : 2
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 10
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe nx lm constant_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr dca lahf_lm
bogomips : 4670.51
processor : 1
vendor_id : GenuineIntel
cpu family : 6
model : 15
model name : Intel(R) Xeon(R) CPU 5148 @ 2.33GHz
stepping : 6
cpu MHz : 2333.469
cache size : 4096 KB
physical id : 0
siblings : 2
core id : 1
cpu cores : 2
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 10
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe nx lm constant_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr dca lahf_lm
bogomips : 4667.16
[root@server kernels]#
[root@server ~]# uname -a
Linux server.name 2.6.19.2-grsec #1 SMP Tue Jan 15 07:08:50 CST 2008 i686 i686 i386 GNU/Linux
[root@server ~]#
[root@server ~]# cat /etc/modprobe.conf
alias scsi_hostadapter usb-storage
alias eth0 e1000
alias eth1 e1000
alias scsi_hostadapter1 aacraid
alias scsi_hostadapter2 ata_piix
alias scsi_hostadapter3 usb-storage
alias usb-controller ehci-hcd
alias usb-controller1 uhci-hcd
[root@server ~]#
I see the following error, while trying to compile kernel with grsec.
WARNING: No module mptscsi found for kernel 2.6.22.9-grsec, continuing anyway
how to fix this or is it ok to reboot?
More details ################# [root@server3 linux-2.6.22.9]# make install sh /usr/src/kernels/linux-2.6.22.9/arch/i386/boot/install.sh 2.6.22.9-grsec arch/i386/boot/bzImage System.map "/boot" WARNING: No module mptscsi found for kernel 2.6.22.9-grsec, continuing anyway
[root@server3 linux-2.6.22.9]# cat /etc/modprobe.conf alias eth0 tg3 alias eth1 tg3 alias scsi_hostadapter mptbase alias scsi_hostadapter1 mptscsi alias scsi_hostadapter2 mptspi alias scsi_hostadapter3 mptsas alias scsi_hostadapter4 mptscsih alias scsi_hostadapter5 sata_svw alias usb-controller ehci-hcd alias usb-controller1 ohci-hcd
Tried building a grsec patched kernel as I don't fancy getting that rootkit thats going around for a second time
it was linux-2.6.23.9 which was the latest one there was a grsec patch for.
compiled ok as a monolithic kernel however on reboot I was not able to get anything but a ping from the server, Figured that the kernel had paniced and not booted.
However the tech who rebooted the server for me and selected the other kernel (a standard centos kernel) said that the system had booted but was not responding to tcp traffic.
So its either a case of
1) I missed something important when configuring the kernel.
2) perhaps a problem with APF it does warn enabling Monolithic kernel support is unsupported in the config file (although it seems to work fine with the centos stock kernel)...
I'm wandering what the best next course of action is i'd quite like to be able to run a kernel that doesn't allow write access to /dev/kmem if possible.
this server is crashing after a few hours... it just got frozen... and after rebooted the server, i was looking at the /var/log/message logs and saw this ( you will see when system restart after the crash ):
Code: Feb 20 17:35:04 server kernel: grsec: signal 11 sent to /usr/www/htdocs/mywebsite.com_cgi/script/out.cgi[out.cgi:13280] uid/euid:48/48 gid/egid:48/48, parent /us r/sbin/httpd[httpd:6180] uid/euid:48/48 gid/egid:48/48 Feb 20 17:41:40 server kernel: grsec: From 190.73.138.68: signal 11 sent to /usr/www/htdocs/mywebsite.com_cgi/script/out.cgi[out.cgi:27459] uid/euid:48/48 gid/eg id:48/48, parent /usr/sbin/httpd[httpd:20166] uid/euid:48/48 gid/egid:48/48 Feb 20 17:45:03 server kernel: grsec: signal 7 sent to /usr/bin/php[php:31710] uid/euid:0/0 gid/egid:0/0, parent /bin/bash[sh:8963] uid/euid:0/0 gid/egid:0/0 Feb 20 17:48:41 server kernel: grsec: From 87.219.205.218: signal 11 sent to /usr/www/htdocs/mywebsite.com_cgi/script/out.cgi[out.cgi:11897] uid/euid:48/48 gid/e gid:48/48, parent /usr/sbin/httpd[httpd:8152] uid/euid:48/48 gid/egid:48/48 Feb 20 17:51:04 server kernel: grsec: From 85.58.139.135: signal 11 sent to /usr/www/htdocs/mywebsite.com_cgi/script/out.cgi[out.cgi:28508] uid/euid:48/48 gid/eg id:48/48, parent /usr/sbin/httpd[httpd:19918] uid/euid:48/48 gid/egid:48/48 Feb 20 17:51:58 server kernel: grsec: signal 11 sent to /usr/www/htdocs/mywebsite.com_cgi/script/out.cgi[out.cgi:15615] uid/euid:48/48 gid/egid:48/48, parent /us r/sbin/httpd[httpd:2482] uid/euid:48/48 gid/egid:48/48 Feb 20 17:52:08 server kernel: grsec: From 166.114.104.42: signal 11 sent to /usr/www/htdocs/mywebsite.com_cgi/script/out.cgi[out.cgi:4662] uid/euid:48/48 gid/eg id:48/48, parent /usr/sbin/httpd[httpd:24468] uid/euid:48/48 gid/egid:48/48 Feb 20 17:52:38 server kernel: grsec: From 189.175.50.103: signal 11 sent to /usr/www/htdocs/mywebsite.com_cgi/script/out.cgi[out.cgi:12497] uid/euid:48/48 gid/e gid:48/48, parent /usr/sbin/httpd[httpd:32213] uid/euid:48/48 gid/egid:48/48 Feb 20 17:54:32 server kernel: grsec: From 83.53.142.7: signal 11 sent to /usr/www/htdocs/mywebsite.com_cgi/script/out.cgi[out.cgi:18556] uid/euid:48/48 gid/egid :48/48, parent /usr/sbin/httpd[httpd:22809] uid/euid:48/48 gid/egid:48/48 Feb 20 17:55:04 server kernel: grsec: signal 7 sent to /usr/bin/php[php:29694] uid/euid:502/502 gid/egid:502/502, parent /bin/bash[sh:30003] uid/euid:502/502 gid /egid:502/502 Feb 20 18:00:54 server kernel: grsec: From 189.141.26.82: signal 11 sent to /usr/www/htdocs/mywebsite.com_cgi/script/out.cgi[out.cgi:10817] uid/euid:48/48 gid/eg id:48/48, parent /usr/sbin/httpd[httpd:13549] uid/euid:48/48 gid/egid:48/48 Feb 20 18:01:07 server kernel: grsec: signal 7 sent to /usr/bin/php[php:20901] uid/euid:0/0 gid/egid:0/0, parent /bin/bash[sh:12242] uid/euid:0/0 gid/egid:0/0 Feb 20 18:03:06 server kernel: grsec: signal 7 sent to /usr/bin/php[php:9696] uid/euid:502/502 gid/egid:502/502, parent /bin/bash[sh:23721] uid/euid:502/502 gid/ egid:502/502 Feb 20 18:03:29 server kernel: grsec: From 68.26.197.159: signal 11 sent to /usr/www/htdocs/mywebsite.com_cgi/script/out.cgi[out.cgi:917] uid/euid:48/48 gid/egid :48/48, parent /usr/sbin/httpd[httpd:20771] uid/euid:48/48 gid/egid:48/48 Feb 20 18:04:43 server kernel: grsec: From 87.219.88.132: signal 11 sent to /usr/www/htdocs/mywebsite.com_cgi/script/out.cgi[out.cgi:10750] uid/euid:48/48 gid/eg id:48/48, parent /usr/sbin/httpd[httpd:4130] uid/euid:48/48 gid/egid:48/48 Feb 20 18:05:04 server kernel: grsec: From 189.167.128.26: signal 11 sent to /usr/www/htdocs/mywebsite.com_cgi/script/out.cgi[out.cgi:14515] uid/euid:48/48 gid/e gid:48/48, parent /usr/sbin/httpd[httpd:2598] uid/euid:48/48 gid/egid:48/48 Feb 20 18:07:05 server kernel: grsec: signal 7 sent to /usr/bin/php[php:29589] uid/euid:0/0 gid/egid:0/0, parent /bin/bash[sh:7958] uid/euid:0/0 gid/egid:0/0 Feb 20 18:08:31 server kernel: grsec: From 88.64.181.89: signal 11 sent to /usr/www/htdocs/mywebsite.com_cgi/script/out.cgi[out.cgi:15335] uid/euid:48/48 gid/egi d:48/48, parent /usr/sbin/httpd[httpd:27788] uid/euid:48/48 gid/egid:48/48 Feb 20 18:08:43 server kernel: grsec: From 201.244.116.46: signal 11 sent to /usr/www/htdocs/mywebsite.com_cgi/script/out.cgi[out.cgi:15217] uid/euid:48/48 gid/e gid:48/48, parent /usr/sbin/httpd[httpd:29545] uid/euid:48/48 gid/egid:48/48 Feb 20 18:17:34 server syslogd x.x.x: restart. Feb 20 18:17:34 server syslog: Iniciaci� de syslogd succeeded Feb 20 18:17:34 server kernel: klogd x.x.x, log source = /proc/kmsg started. Feb 20 18:17:34 server kernel: Linux version 2.x.xxgrs-bipiv-ipv4 (root@kernel.myserver.net) (gcc version xxxx) #1 SMP Tue Jan 31 17:34:40 CET 2006 Feb 20 18:17:34 server kernel: BIOS-provided physical RAM map: Feb 20 18:17:34 server kernel: BIOS-e820: 0000000000000000 - 000000000009c400 (usable) Feb 20 18:17:34 server kernel: BIOS-e820: 000000000009c400 - 00000000000a0000 (reserved) Feb 20 18:17:34 server kernel: BIOS-e820: 00000000000ea070 - 0000000000100000 (reserved) Feb 20 18:17:34 server kernel: BIOS-e820: 0000000000100000 - 000000007 Bold date are ( I think ) the crash, and the system booting... Any ideas about what can be causing the crash.. ? Is this kernel compiled with GRSecurity ? may that affect cgis ?
This is a fedora core server, xeon 3.2 GHZ x 4 procs using about 25 MBits per day.
I have a dedicated server running CentOS which has been been running fine for a while now.
On the odd day Apache used to hang, I presume due to load so I installed SIM (System Integrity Manager) to check the status of httpd and to re-start when necessary.
Every now and again SIM would email saying that apache was re-started, no big deal.
Then this morning I got a email from SIM saying apache was down but couldn't be re-started. I then logged into the shell and tried to do a: service httpd restart
I then got a message saying something like: could not bind to port 80. I then tried to kill all httpd processes and re-start apache but the problem persisted. I then just rebooted my server and apache started fine.
Since then I have had hundreds of emails from SIM saying that it has had to re-start apache. Somehow it is detecting it as down and re-starting it. Apache seems to be up and fine. Re-installing SIM does not help.
The weird thing is when I logged in using Putty I got the "server's host key did not match the signature supplied" message like something had changed.
Rkhunter does not return anything just a couple of warnings.
Does anybody know what would cause sim to constantly detect Apache as down in this situation and also how this could happen with no changes being made?
For the last week, my vps (directadmin on centos on xen) has become unresponsive every three days at approximately 5 AM, and I'm baffled about why. It doesn't crash, but it stops accepting incoming connections until it's rebooted.
I'm not sure how to troubleshoot this. /var/log/messages gives no useful information. There are no unusual cronjobs running at that time.
Anyone heard of anything like this before or have any advice about how to troubleshoot?
we have one box in hivelocity.net that has been down so many times this month that we were forced to remove links to siteuptime where we were once so proud of having a 99.7% uptime for 3 years in theplanet.
syslog shows that just before crashing, these entries were made:
kernel: kernel BUG at mm/rmap.c:479 kernel: invalid operand:0000 [#1]
dmesg also shows this:
... Brought up 2 CPUs zapping low mappings. checking if image is initramfs... it is Freeing initrd memory: 482k freed NET: Registered protocol family 16 PCI: PCI BIOS revision 2.10 entry at 0xf9f20, last bus=1 PCI: Using configuration type 1 mtrr: v2.0 (20020519) mtrr: your CPUs had inconsistent fixed MTRR settings mtrr: probably your BIOS does not setup all CPUs. mtrr: corrected configuration. ...
i've googled these messages and they point to ram problems.
hivelocity.net claims to have done diagnostics on the box and that there were no problems reported.
they said this is a result of a sys configuration problem made by us.
Last year I ordered a new server with Centos 4.3 and it had the kernel kernel 2.6.9-34.0.2ELsmp installed. It runned fine and I didn't update any packages since then.
Today I started getting a problem where both mysqld and kswapd0 uses very high amounts of CPU, spiking up to 100% and my memory usage is at 99% all the time. The problem seems exactly the same as the one mentioned in this thread.
In that thread the exact same kernel is said to be insecure and to cause this problem. I also came across a centOS bug that reports this problem with high cpu, mem usage and mysql & kswapd0 consuming all resources.
In the linked thread the person solved the problem by upgrading to kernel 2.6.9-42 using rpms but others recommended a newer kernel or a custom compiled kernel for CentOS.
Apparently when they used yum it said 34.0.2 was the latest kernel.
What should I do to upgrade the kernel, which version should i upgrade to, and where do I get it from? I won't be able to compile a custom kernel and I've only installed basic rpm packages before.
We had a Coldfusion site running MySQL without issues. We are in the process of doing some upgrades and as part of that we recently switched over to MSSQL 5. Ever since that happened, we will have random page hangs. This usually happens on pages with more complex queries, but it is very random. There is no timeout error on the server, no page cannot be displayed errors, no errors of any kind, the page will just hang. We have been monitoring our server resource monitor and we can see when the page 'gives up', we will go from 100% CPU down to almost 0, and then the page simply never recovers. Does anyone have any idea what might be going on? When there is no error message of sorts it makes it very problematic to troubleshoot.
I am using apache proxypass to proxy to 4 different ports (java/jsp applications) with the following configurations (for each):
<VirtualHost *:80> ProxyPreserveHost On ProxyRequests Off ServerName *****.org ProxyPass / http://*****.org:8080/ retry=0 ProxyPassReverse / http://*****.org:8080/ </VirtualHost>
Everything seems work fine until an awkward delay / connection timeout occurs. The web application hangs for about 5 to 15 seconds and starts working normally after that period. It's certainly not the application's issue because everything is fine when using IP address to access it.
When I enabled the debug log in apache, I found out that every time the server seems to hang, the following error appears in the apache error log and the number of apache child processes keep on increasing (ps aux | grep apache).
[Thu Apr 11 10:20:41 2013] [debug] proxy_util.c(1818): proxy: grabbed scoreboard slot 4 in child 13208 for worker proxy:reverse [Thu Apr 11 10:20:41 2013] [debug] proxy_util.c(1837): proxy: worker proxy:reverse already initialized [Thu Apr 11 10:20:41 2013] [debug] proxy_util.c(1934): proxy: initialized single connection worker 4 in child 13208 for (*) [Thu Apr 11 10:20:41 2013] [debug] proxy_util.c(1818): proxy: grabbed scoreboard slot 3 in child 13208 for worker http://*****.org:8080/ ..... Also, the access log (access.log) completely stops during that hang period.
This debug message keeps on repeating as long as Apache/web-application is not responding. The application starts to work normally after that duration and the number of apache child process decrease to the normal number.
Apache version : 2.2.16 (Debian) OS : Debian GNU/Linux 6.0 root@server:~# apache2 -V Server version: Apache/2.2.16 (Debian) Server built: Nov 30 2012 08:58:36
here is what I seen when I installed kernel-2.6.20-1.2948.fc6.src.rpm
rpm -ivh kernel-2.6.20-1.2948.fc6.src.rpm 1:kernel warning: user brewbuilder does not exist - using root warning: group brewbuilder does not exist - using root warning: user brewbuilder does not exist - using root ########################################### [100%] warning: user brewbuilder does not exist - using root warning: group brewbuilder does not exist - using root
then when I ran: rpmbuild -bp --target=$(uname -m) /usr/src/redhat/SPECS/kernel-2.6.spec
I seen this error: + Arch=x86_64 + make ARCH=x86_64 nonint_oldconfig In file included from /usr/include/sys/socket.h:35, from /usr/include/netinet/in.h:24, from /usr/include/arpa/inet.h:23, from scripts/basic/fixdep.c:117: /usr/include/bits/socket.h:310:24: error: asm/socket.h: No such file or directory make[1]: *** [scripts/basic/fixdep] Error 1 make: *** [scripts_basic] Error 2 error: Bad exit status from /var/tmp/rpm-tmp.93770 (%prep)
I need to have this installed to get a app installed etc... suggestions or ideas? thanks
When I type a command with putty, the connection is closed immediately. I tried shutdown-r now and reboot, halt, do nothing to console closes and nothing happens.
After a hundred connection, I can use ls, su and kill.
I think it's the fact that the partition is corrupted. I can not Hardware reboot the server because CTN1 is "out of business".
VPS isn't rebooting by itself when it goes down. Anyone has any program/script that monitors heartbeat of the server? Like when it goes down, the program will automatically reboots the system. I know there's such a script out there but I forgot what it called.
I had to reboot my server and about 20 minutes later I tried to access the web site but the page was not found... I am able to login to SSH. However, I am not familiar with *nix or the workings of CPanel... What should I do to get the sites back online?
so my server dies every day and requires human intervention to fully restart all service to have my site work properly. i suspect sigterm issues as it fails to restart all service as website is still down so i always have to reboot it.
Tried recompile apache with no success [Tue Mar 18 06:51:27 2008] [error] [client 203.160.1.39] request failed: erroneous characters after protocol string: If-Modified-Since: Wed, 21 Nov 2007 06:16:52 GMT [Tue Mar 18 10:03:18 2008] [error] Bad pid (7465) in scoreboard slot 16 [Tue Mar 18 10:03:18 2008] [error] Bad pid (27848) in scoreboard slot 17 [Tue Mar 18 10:03:18 2008] [error] Bad pid (27434) in scoreboard slot 18 [Tue Mar 18 10:03:18 2008] [error] Bad pid (30782) in scoreboard slot 19 [Tue Mar 18 10:03:18 2008] [error] Bad pid (7465) in scoreboard slot 16 [Tue Mar 18 10:03:18 2008] [error] Bad pid (27848) in scoreboard slot 17 [Tue Mar 18 10:03:18 2008] [error] Bad pid (27434) in scoreboard slot 18 [Tue Mar 18 10:03:18 2008] [error] Bad pid (30782) in scoreboard slot 19 [Tue Mar 18 10:03:18 2008] [error] Bad pid (7465) in scoreboard slot 16 [Tue Mar 18 10:03:18 2008] [error] Bad pid (27848) in scoreboard slot 17 [Tue Mar 18 10:03:18 2008] [error] Bad pid (27434) in scoreboard slot 18 [Tue Mar 18 10:03:18 2008] [error] Bad pid (30782) in scoreboard slot 19 [Tue Mar 18 10:03:18 2008] [notice] caught SIGTERM, shutting down [Tue Mar 18 10:03:20 2008] [notice] mod_security/1.9.5 configured - Apache/1.3.39 (Unix) PHP/5.2.5 [Tue Mar 18 10:03:20 2008] [notice] Any You Like mod_ssl/2.8.30 OpenSSL/0.9.8g mod_perl/1.29 FrontPage/5.0.2.2510 configured -- resuming normal operations [Tue Mar 18 10:03:20 2008] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec) [Tue Mar 18 10:03:20 2008] [notice] Accept mutex: sysvsem (Default: sysvsem)
I'm finding that my server doesn't like to reboot gracefully. Either selecting "graceful server reboot" in WHM or actually typing "reboot" in SSH, which then tells me the server is shutting down. My server is then incommunicato indefinitely until I actually do a hard reset remotely.
Is this common? Is there some way to find out why this is happening?