My server /var/log/message last night showed this below (see the kernel message)
My SSH was idling, it shows:
root@server1 [/backup/w1/cpbackup/daily]#
Message from syslogd@ at Sun Aug 10 18:29:31 2008 ...
server1 kernel: journal commit I/O error
Message from syslogd@ at Sun Aug 10 18:35:00 2008 ...
server1 kernel: journal commit I/O error
The server was down. So I had to reboot and run e2fsck to fix it. I/O error could it mean HDD problem? (because I just change this server to new HDD 3 weeks ago)
> Aug 10 18:15:08 server1 kernel: Firewall: *UDP_IN Blocked* IN=eth0 OUT=
> MAC=ff:ff:ff:ff:ff:ff:00:1d:09:6d:cd:bb:08:00 SRC=2$
> Aug 10 18:15:11 server1 kernel: 0000:00:1c.5: eth0: Detected Tx Unit Hang:
> Aug 10 18:15:11 server1 kernel: TDH <8>
> Aug 10 18:15:11 server1 kernel: TDT <30>
> Aug 10 18:15:11 server1 kernel: next_to_use <30>
> Aug 10 18:15:11 server1 kernel: next_to_clean <4>
> Aug 10 18:15:11 server1 kernel: buffer_info[next_to_clean]:
> Aug 10 18:15:11 server1 kernel: time_stamp <32dfbee6>
> Aug 10 18:15:11 server1 kernel: next_to_watch <8>
> Aug 10 18:15:11 server1 kernel: jiffies <32dfc9d6>
> Aug 10 18:15:11 server1 kernel: next_to_watch.status <0>
> Aug 10 18:15:13 server1 kernel: 0000:00:1c.5: eth0: Detected Tx Unit Hang:
> Aug 10 18:15:13 server1 kernel: TDH <8>
> Aug 10 18:15:13 server1 kernel: TDT <30>
> Aug 10 18:15:13 server1 kernel: next_to_use <30>
> Aug 10 18:15:13 server1 kernel: next_to_clean <4>
> Aug 10 18:15:13 server1 kernel: buffer_info[next_to_clean]:
> Aug 10 18:15:13 server1 kernel: time_stamp <32dfbee6>
> Aug 10 18:15:13 server1 kernel: next_to_watch <8>
> Aug 10 18:15:13 server1 kernel: jiffies <32dfd1a5>
> Aug 10 18:15:13 server1 kernel: next_to_watch.status <0>
> Aug 10 18:15:15 server1 kernel: 0000:00:1c.5: eth0: Detected Tx Unit Hang:
> Aug 10 18:15:15 server1 kernel: TDH <8>
> Aug 10 18:15:15 server1 kernel: TDT <30>
> Aug 10 18:15:15 server1 kernel: next_to_use <30>
> Aug 10 18:15:15 server1 kernel: next_to_clean <4>
> Aug 10 18:15:15 server1 kernel: buffer_info[next_to_clean]:
> Aug 10 18:15:15 server1 kernel: time_stamp <32dfbee6>
> Aug 10 18:15:15 server1 kernel: next_to_watch <8>
> Aug 10 18:15:15 server1 kernel: jiffies <32dfd976>
> Aug 10 18:15:15 server1 kernel: next_to_watch.status <0>
> Aug 10 18:15:17 server1 kernel: 0000:00:1c.5: eth0: Detected Tx Unit Hang:
> Aug 10 18:15:17 server1 kernel: TDH <8>
> Aug 10 18:15:17 server1 kernel: TDT <30>
> Aug 10 18:15:17 server1 kernel: next_to_use <30>
> Aug 10 18:15:17 server1 kernel: next_to_clean <4>
> Aug 10 18:15:17 server1 kernel: buffer_info[next_to_clean]:
> Aug 10 18:15:17 server1 kernel: time_stamp <32dfbee6>
> Aug 10 18:15:17 server1 kernel: next_to_watch <8>
> Aug 10 18:15:17 server1 kernel: jiffies <32dfe145>
> Aug 10 18:15:17 server1 kernel: next_to_watch.status <0>
> Aug 10 18:15:19 server1 kernel: 0000:00:1c.5: eth0: Detected Tx Unit Hang:
> Aug 10 18:15:19 server1 kernel: TDH <8>
> Aug 10 18:15:19 server1 kernel: TDT <30>
> Aug 10 18:15:19 server1 kernel: next_to_use <30>
> Aug 10 18:15:19 server1 kernel: next_to_clean <4>
> Aug 10 18:15:19 server1 kernel: buffer_info[next_to_clean]:
> Aug 10 18:15:19 server1 kernel: time_stamp <32dfbee6>
> Aug 10 18:15:19 server1 kernel: next_to_watch <8>
> Aug 10 18:15:19 server1 kernel: jiffies <32dfe915>
> Aug 10 18:15:19 server1 kernel: next_to_watch.status <0>
> Aug 10 18:15:20 server1 kernel: NETDEV WATCHDOG: eth0: transmit timed out
> Aug 10 18:15:23 server1 kernel: 0000:00:1c.5: eth0: Link is Up 1000 Mbps
> Full Duplex, Flow Control: RX
> Aug 10 18:15:23 server1 kernel: Firewall: *UDP_IN Blocked* IN=eth0 OUT=
> MAC=ff:ff:ff:ff:ff:ff:00:1d:09:6d:cd:bb:08:00 SRC=2$
> Aug 10 18:15:24 server1 kernel: Firewall: *UDP_IN Blocked* IN=eth0 OUT=
> MAC=ff:ff:ff:ff:ff:ff:00:1d:09:6d:cd:bb:08:00 SRC=2$
Code: Setting up Install Process Setting up repositories update 100% |=========================| 951 B 00:00 base 100% |=========================| 1.1 kB 00:00 addons 100% |=========================| 951 B 00:00 extras 100% |=========================| 1.1 kB 00:00 Reading repository metadata in from local files primary.xml.gz 100% |=========================| 21 kB 00:00 update : ################################################## 94/94 Traceback (most recent call last): File "/usr/bin/yum", line 29, in ? yummain.main(sys.argv[1:]) File "/usr/share/yum-cli/yummain.py", line 97, in main result, resultmsgs = do() File "/usr/share/yum-cli/cli.py", line 470, in doCommands return self.installPkgs() File "/usr/share/yum-cli/cli.py", line 836, in installPkgs self.doRepoSetup() File "/usr/share/yum-cli/cli.py", line 75, in doRepoSetup self.doSackSetup(thisrepo=thisrepo) File "__init__.py", line 260, in doSackSetup File "repos.py", line 287, in populateSack File "sqlitecache.py", line 96, in getPrimary File "sqlitecache.py", line 89, in _getbase File "sqlitecache.py", line 373, in updateSqliteCache File "/var/tmp/python-sqlite-root//usr/lib/python2.3/site-packages/sqlite/main.py", line 244, in execute _sqlite.DatabaseError: unable to open database file root@server [~]# rpm -q yum yum-2.4.3-1.c4
I just try to restart csf in my Centos server with Direct Admin and i got this error:
Quote:
*WARNING* Since the Virtuozzo VPS iptables ip_conntrack_ftp kernel module is currently broken you have to open a PASV port hole in iptables for incoming FTP connections to work correctly. See the csf readme.txt under 'A note about FTP Connection Issues' on how to do this if you have not already done so.
we have one box in hivelocity.net that has been down so many times this month that we were forced to remove links to siteuptime where we were once so proud of having a 99.7% uptime for 3 years in theplanet.
syslog shows that just before crashing, these entries were made:
kernel: kernel BUG at mm/rmap.c:479 kernel: invalid operand:0000 [#1]
dmesg also shows this:
... Brought up 2 CPUs zapping low mappings. checking if image is initramfs... it is Freeing initrd memory: 482k freed NET: Registered protocol family 16 PCI: PCI BIOS revision 2.10 entry at 0xf9f20, last bus=1 PCI: Using configuration type 1 mtrr: v2.0 (20020519) mtrr: your CPUs had inconsistent fixed MTRR settings mtrr: probably your BIOS does not setup all CPUs. mtrr: corrected configuration. ...
i've googled these messages and they point to ram problems.
hivelocity.net claims to have done diagnostics on the box and that there were no problems reported.
they said this is a result of a sys configuration problem made by us.
Last year I ordered a new server with Centos 4.3 and it had the kernel kernel 2.6.9-34.0.2ELsmp installed. It runned fine and I didn't update any packages since then.
Today I started getting a problem where both mysqld and kswapd0 uses very high amounts of CPU, spiking up to 100% and my memory usage is at 99% all the time. The problem seems exactly the same as the one mentioned in this thread.
In that thread the exact same kernel is said to be insecure and to cause this problem. I also came across a centOS bug that reports this problem with high cpu, mem usage and mysql & kswapd0 consuming all resources.
In the linked thread the person solved the problem by upgrading to kernel 2.6.9-42 using rpms but others recommended a newer kernel or a custom compiled kernel for CentOS.
Apparently when they used yum it said 34.0.2 was the latest kernel.
What should I do to upgrade the kernel, which version should i upgrade to, and where do I get it from? I won't be able to compile a custom kernel and I've only installed basic rpm packages before.
here is what I seen when I installed kernel-2.6.20-1.2948.fc6.src.rpm
rpm -ivh kernel-2.6.20-1.2948.fc6.src.rpm 1:kernel warning: user brewbuilder does not exist - using root warning: group brewbuilder does not exist - using root warning: user brewbuilder does not exist - using root ########################################### [100%] warning: user brewbuilder does not exist - using root warning: group brewbuilder does not exist - using root
then when I ran: rpmbuild -bp --target=$(uname -m) /usr/src/redhat/SPECS/kernel-2.6.spec
I seen this error: + Arch=x86_64 + make ARCH=x86_64 nonint_oldconfig In file included from /usr/include/sys/socket.h:35, from /usr/include/netinet/in.h:24, from /usr/include/arpa/inet.h:23, from scripts/basic/fixdep.c:117: /usr/include/bits/socket.h:310:24: error: asm/socket.h: No such file or directory make[1]: *** [scripts/basic/fixdep] Error 1 make: *** [scripts_basic] Error 2 error: Bad exit status from /var/tmp/rpm-tmp.93770 (%prep)
I need to have this installed to get a app installed etc... suggestions or ideas? thanks
I have a Xen VPS. I started with a Debian 4 image and have since upgraded to Debian 5. Firstly was this advisable? Secondly what Kernel version should I be running, or rather is it set by my installation or by the Xen server?
as part of a project I have lately been looking into various aspects of kernel tuning. Most notably lately tuning the TCP stack for more efficient memory usage/throughput.
Thought I would start this thread to mention some of the tools I'd found for doing testing and see what anyone else had to recommend.
So far my favorite of the bunch is nuttcp. Its easy to use and gives a very good idea of how much of your bandwidth you are able to utilize.
A few interesting web pages are as follows for anyone interested in the topic:
[url]- Tuning TCP for High Bandwidth Delay networks
[url]- TCP Tuning Cook book, some interesting information in there as well
[url]...formanceTuning - Performance Tuning TWiki. Has a list of useful tools, flags for existing tools and ways to monitor network performance from a system level, along with some suggestions of things to correct
What is the best way to find out which filesystems and harddrive drivers you can remove? Obviously, i need ext2,3 but how do you find which HD you only need?
i can not find any important info from /var/log/messages
but i find some records many time on it,those like ---------------------------------- Jun 15 05:30:40 server kernel: ata1.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0 Jun 15 05:30:40 server kernel: ata1.00: (irq_stat 0x40000001) Jun 15 05:30:40 server kernel: ata1.00: cmd 25/00:08:42:23:d2/00:00:2c:00:00/e0 tag 0 cdb 0x0 data 4096 in Jun 15 05:30:40 server kernel: res 51/40:00:42:23:d2/00:00:2c:00:00/e0 Emask 0x9 (media error) Jun 15 05:30:40 server kernel: ata1.00: configured for UDMA/133 Jun 15 05:30:40 server kernel: ata1: EH complete Jun 15 05:30:42 server kernel: ata1.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0 Jun 15 05:30:42 server kernel: ata1.00: (irq_stat 0x40000001) Jun 15 05:30:42 server kernel: ata1.00: cmd 25/00:08:42:23:d2/00:00:2c:00:00/e0 tag 0 cdb 0x0 data 4096 in Jun 15 05:30:42 server kernel: res 51/40:00:42:23:d2/00:00:2c:00:00/e0 Emask 0x9 (media error) Jun 15 05:30:42 server kernel: ata1.00: configured for UDMA/133 Jun 15 05:30:42 server kernel: ata1: EH complete Jun 15 05:30:44 server kernel: ata1.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0 Jun 15 05:30:51 server kernel: ata1.00: (irq_stat 0x40000001) Jun 15 05:30:51 server kernel: ata1.00: cmd 25/00:08:42:23:d2/00:00:2c:00:00/e0 tag 0 cdb 0x0 data 4096 in Jun 15 05:30:51 server kernel: res 51/40:00:42:23:d2/00:00:2c:00:00/e0 Emask 0x9 (media error) Jun 15 05:30:51 server kernel: ata1.00: configured for UDMA/133 Jun 15 05:30:51 server kernel: ata1: EH complete Jun 15 05:30:51 server kernel: ata1.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0 Jun 15 05:30:51 server kernel: ata1.00: (irq_stat 0x40000001) Jun 15 05:30:51 server kernel: ata1.00: cmd 25/00:08:42:23:d2/00:00:2c:00:00/e0 tag 0 cdb 0x0 data 4096 in Jun 15 05:30:51 server kernel: res 51/40:00:42:23:d2/00:00:2c:00:00/e0 Emask 0x9 (media error) Jun 15 05:30:51 server kernel: ata1.00: configured for UDMA/133 Jun 15 05:30:51 server kernel: ata1: EH complete Jun 15 05:30:51 server kernel: ata1.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0 Jun 15 05:30:51 server kernel: ata1.00: (irq_stat 0x40000001) Jun 15 05:30:51 server kernel: ata1.00: cmd 25/00:08:42:23:d2/00:00:2c:00:00/e0 tag 0 cdb 0x0 data 4096 in Jun 15 05:30:51 server kernel: res 51/40:00:42:23:d2/00:00:2c:00:00/e0 Emask 0x9 (media error) Jun 15 05:30:51 server kernel: ata1.00: configured for UDMA/133 Jun 15 05:30:52 server kernel: ata1: EH complete
Jun 15 05:31:26 server kernel: ata1.00: configured for UDMA/133 Jun 15 05:31:30 server kernel: sd 0:0:0:0: SCSI error: return code = 0x08000002 Jun 15 05:31:33 server kernel: sda: Current [descriptor]: sense key: Medium Error Jun 15 05:31:36 server kernel: Add. Sense: Unrecovered read error - auto reallocate failed Jun 15 05:31:36 server kernel: Jun 15 05:31:39 server kernel: Descriptor sense data with sense descriptors (in hex): Jun 15 05:31:46 server kernel: 72 03 11 04 00 00 00 0c 00 0a 80 00 00 00 00 00 Jun 15 05:31:51 server kernel: 2c d2 23 42 Jun 15 05:31:56 server kernel: end_request: I/O error, dev sda, sector 751969090 Jun 15 05:31:57 server kernel: ata1: EH complete Jun 15 05:31:57 server kernel: SCSI device sda: 976773168 512-byte hdwr sectors (500108 MB) Jun 15 05:31:58 server kernel: sda: Write Protect is off Jun 15 05:31:58 server kernel: SCSI device sda: drive cache: write back Jun 15 05:31:59 server kernel: SCSI device sda: 976773168 512-byte hdwr sectors (500108 MB) Jun 15 05:32:03 server kernel: sda: Write Protect is off Jun 15 05:32:04 server kernel: SCSI device sda: drive cache: write back -------------------
I copied the default config file and renamed it as .config but I get this:
Code: WARNING: No module dm-mem-cache found for kernel 2.6.27.10-grsec, continuing anyway WARNING: No module dm-region_hash found for kernel 2.6.27.10-grsec, continuing anyway WARNING: No module dm-message found for kernel 2.6.27.10-grsec, continuing anyway WARNING: No module dm-raid45 found for kernel 2.6.27.10-grsec, continuing anyway
My current kernel version is "2.6.9-42.0.10.ELsmp #1 SMP Fri Feb 16 17:17:21 EST 2007 i686 athlon i386 GNU/Linux". I want it to be upgraded since it is old. I have been told by our server management company that the latest kernel distributed from yum is kernel.i686 0:2.6.9-78.0.22.E. Can anyone tell me if this version is safe and secure enough? It is a CentOS release 4.7 (Final) server with cPanel installed.
when doing 2.6.26+ or w/e it is, how do you enable conntrack, what options do i need to enable under make menuconfig?
net.netfilter.nf_conntrack_acct = 1 net.netfilter.nf_conntrack_generic_timeout = 120 error: "net.netfilter.nf_conntrack_icmp_timeout" is an unknown key error: "net.netfilter.nf_conntrack_tcp_timeout_close" is an unknown key error: "net.netfilter.nf_conntrack_tcp_timeout_time_wait" is an unknown key error: "net.netfilter.nf_conntrack_tcp_timeout_last_ack" is an unknown key error: "net.netfilter.nf_conntrack_tcp_timeout_close_wait" is an unknown key error: "net.netfilter.nf_conntrack_tcp_timeout_fin_wait" is an unknown key error: "net.netfilter.nf_conntrack_tcp_timeout_established" is an unknown key error: "net.netfilter.nf_conntrack_tcp_timeout_syn_recv" is an unknown key error: "net.netfilter.nf_conntrack_tcp_timeout_syn_sent" is an unknown key error: "net.netfilter.nf_conntrack_udp_timeout" is an unknown key error: "net.netfilter.nf_conntrack_udp_timeout_stream" is an unknown key net.netfilter.nf_conntrack_max = 262144
and how do i know which hardware/devices that i can remove?
I have always used RPM kernels from the RedHat Network, in this time I need to compile an SMP kernel in a server that is used to serve common webhosting traffic (web, ftp, email, etc) with cPanel.
I have compiled kernels from source in desktops stations, but never in remote servers.. do you have any recommendations to do this at a Dual Xeon 3.2 RHE 5 + cPanel server? Oh.. almost forgot.. I will also compile it with GRsecurity.
Should I use a src kernel from RedHat, or a kernel.org latest kernel?
I installed CentOS 5 on a server with 2 quad core CPU's and 4GB of RAM. After the installation is done and during boot I see a warning that says "Warning only 3GB will be used".
So, just to check, I installed CentOS 4.4 and it sees all 4GB of RAM.
Are there any memory limitations on the distro? or could it be that the desktop kernels made their way into the iso's from upstream?
I am recompiling the kernel with the src rpm. I haven't been able to get any kernel to boot properly since i got this server.
I really could use some help, below i posted lspci, lsmod and cat interrupts. What I really need help with is which kernel modules are necessary for my server.
lspci: [url] lsmod: [url] interrupts: [url]
I have already tried a kernel with ext3, and sata support compiled in but it never boots.
