Bottlenecks (iowait Or CPU)
Oct 24, 2007
I have a server with two cPanel accounts, each running gallery2. Load has been high:
[url]
[url]
[url]
Sar output (done at a less busy time):
[url]
Hardware:
1 Intel 2.4 GHz 1066FSB - Conroe Xeon 3060 (Dual Core)
2 Generic 1024 MB DDR2 667 ECC
1 Maxtor 250GB SATA 7200RPM MaxLine Plus II
1 Unknown Onboard SATA
1 Dell Single socket 1067FSB - Quad Core Capable PowerEdge 840
Howmuch %iowait is too much?
Since it's using less than 1GB Swap, there's enough RAM?
How do I get more specific than httpd as the cause of the load. The gallery script is the only thing running so is the execution time of PHP the bottleneck? If PHP and apache are single threaded would a quad core reduce load? Does lighttpd run PHP faster?
View 5 Replies
ADVERTISEMENT
Feb 22, 2007
Code:
19:28:57 up 22 min, 2 users, load average: 2.99, 3.48, 3.52
105 processes: 103 sleeping, 1 running, 0 zombie, 1 stopped
CPU states: cpu user nice system irq softirq iowait idle
total 0.0% 0.0% 0.4% 0.0% 0.2% 99.2% 0.0%
cpu00 0.0% 0.0% 0.4% 0.0% 0.4% 99.0% 0.0%
cpu01 0.0% 0.0% 0.4% 0.0% 0.0% 99.5% 0.0%
Mem: 2047220k av, 1020368k used, 1026852k free, 0k shrd, 88516k buff
515212k active, 316448k inactive
Swap: 2040244k av, 0k used, 2040244k free 672648k cached
PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME CPU COMMAND
7775 root 15 0 1344 1344 916 S 0.2 0.0 0:02 0 top
14220 root 15 0 1216 1216 920 R 0.2 0.0 0:00 1 top -c
19768 root 25 0 3764 3764 2216 S 0.4 0.1 0:00 0 /usr/local/cpanel/bin/dcpumon
1 root 15 0 544 544 472 S 0.0 0.0 0:03 0 init
2 root RT 0 0 0 0 SW 0.0 0.0 0:00 0 migration/0
3 root RT 0 0 0 0 SW 0.0 0.0 0:00 1 migration/1
4 root 15 0 0 0 0 SW 0.0 0.0 0:00 1 keventd
5 root 34 19 0 0 0 SWN 0.0 0.0 0:00 0 ksoftirqd/0
6 root 34 19 0 0 0 SWN 0.0 0.0 0:00 1 ksoftirqd/1
9 root 25 0 0 0 0 SW 0.0 0.0 0:00 1 bdflush
7 root 15 0 0 0 0 SW 0.0 0.0 0:00 1 kswapd
Was sorted by processor usage.
Code:
root@dual [/var/log]# uname -a
Linux 2.4.21-47.0.1.ELsmp #1 SMP Thu Oct 19 11:33:45 EDT 2006 i686 i686 i386 GNU/Linux
Code:
Processor Information
Processor #1 Vendor: GenuineIntel
Processor #1 Name: Intel(R) Pentium(R) 4 CPU 3.80GHz
Processor #1 speed: 3790.694 MHz
Processor #1 cache size: 2048 KB
Processor #2 Vendor: GenuineIntel
Processor #2 Name: Intel(R) Pentium(R) 4 CPU 3.80GHz
Processor #2 speed: 3790.694 MHz
Processor #2 cache size: 2048 KB
View 2 Replies
View Related
Oct 27, 2008
Recently I've noticed named causing high iowait.
If I stop named, iowait reduces to around 40%.
Code:
10:05:55 up 1:59, 2 users, load average: 5.09, 6.73, 6.23
192 processes: 185 sleeping, 7 running, 0 zombie, 0 stopped
CPU states: cpu user nice system irq softirq iowait idle
total 4.3% 0.0% 5.0% 0.3% 0.0% 89.2% 0.9%
cpu00 4.8% 0.0% 4.8% 0.0% 0.1% 89.3% 0.8%
cpu01 3.9% 0.0% 5.2% 0.7% 0.0% 89.0% 1.0%
Mem: 1023624k av, 966688k used, 56936k free, 0k shrd, 119500k buff
658244k actv, 130464k in_d, 16084k in_c
Swap: 1052248k av, 72096k used, 980152k free 496268k cached
PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME CPU COMMAND
15 root 15 0 0 0 0 SW 0.8 0.0 2:23 1 kjournald
9182 named 15 0 3504 3504 2184 S 0.6 0.3 0:01 0 named
8301 apache 15 0 14564 11M 6832 S 0.5 1.1 0:00 0 httpd
1043 qmails 24 0 1196 1064 384 R 0.4 0.1 1:10 0 qmail-send
1047 qmailr 25 0 372 364 280 S 0.3 0.0 0:35 1 qmail-rspawn
30825 root 15 0 1140 1088 744 R 0.2 0.1 0:09 1 top
7025 apache 15 0 13536 10M 6820 S 0.2 1.0 0:00 0 httpd
7076 apache 15 0 13868 11M 6820 S 0.2 1.1 0:00 1 httpd
759 root 15 0 568 548 488 R 0.1 0.0 1:30 1 syslogd
I believe this isn't normal.
View 2 Replies
View Related
Dec 17, 2007
I was wondering if anyone has had luck with 3ware 9650's in a RAID 10 to help bring down iowait?
I currently have a VPS node that see's spikes in iowait and i'd like to bring it down. I've been considering getting a 9650 in a JBOD setup (so then I don't need to reinstall the whole node) with a battery backup to enable the cache.
My two questions are:
- In JBOD, does the write cache enable at all?
- Would the write cache help bring down the iowait much?
Drive wise we have 4 x 500GB drives in a raid10 (soft for now).
View 8 Replies
View Related
May 30, 2009
Few weeks ago my customer began to notice problem with server load.
I look to munin graphs and see this:
http://dl.getdropbox.com/u/252944/iowait/iowait-hr.png
iowait creates high performance degradation. There was no changes in server software & configuration. Sometimes server work very slow, count of apache processes is high - there is no any processes and scripts who can use a lot of hdd resources.
Sites located on customer server does not create high load, and not subjected to any attacks.
I received response from DC:
I did some research for this and it seems that other people has experienced similar issues and is reporting that it is a kernel bug have you ran yum update to receive the latest kernel if not you can try and update and see if that may resolve your issue.
CENTOS 5.2
Cpanel / WHM
Linux *** 2.6.18-92.1.22.el5PAE #1 SMP Tue Dec 16 12:36:25 EST 2008 i686 i686 i386 GNU/Linux
I try to run "yum update kernel" but it seems installed kernel is up to date.
View 14 Replies
View Related
Jul 14, 2009
In my server iowait is causing high load, how do we find out which are all users taking memory that could cause my server to use swap memory.
server : linux centos/Cpanel
I have tried the below commands, but how do we locate user specific,
ps aux --sort -vsz | head -25
vmstat 1
View 7 Replies
View Related
Apr 12, 2008
Server is Xeon 3060, 3GB Ram, 250GB SATA-II HDD. I followed the recommendations made by mysqlprimer tuner script.
top - 11:54:33 up 237 days, 15:01, 1 user, load average: 6.06, 2.80, 1.65
Tasks: 133 total, 2 running, 131 sleeping, 0 stopped, 0 zombie
Cpu(s): 13.0% us, 4.9% sy, 0.0% ni, 24.3% id, 57.8% wa, 0.0% hi, 0.0% si
Mem: 3114144k total, 3096468k used, 17676k free, 30220k buffers
Swap: 2096440k total, 32348k used, 2064092k free, 2515412k cached
PID USER PR NI %CPU TIME+ %MEM VIRT RES SHR S COMMAND
8610 nobody 16 0 28 1:37.73 0.7 78908 20m 16m S lsphp4
10520 nobody 16 0 3 0:01.97 0.4 78788 10m 6884 S lsphp4
26474 nobody 15 0 2 15:04.07 0.4 14196 11m 3216 S lshttpd
28275 mysql 22 6 1 131:28.83 7.4 733m 224m 2700 S /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --pi
Code:
[mysqld]
max_connections=500
max_user_connections=300
skip-locking
skip-innodb
query_cache_limit=1M
query_cache_size=64M
query_cache_type=1
interactive_timeout=60
wait_timeout=60
connect_timeout=6
thread_cache_size=128
key_buffer_size=640M
max_allowed_packet=16M
table_cache=2048
join_buffer=1M
sort_buffer=2M
record_buffer=1M
sort_buffer_size=2M
read_buffer_size=2M
safe-show-database
max_connect_errors=10
thread_concurrency=2
myisam_sort_buffer_size=64M
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
#log_slow_queries=/var/log/mysql/mysql_slow-queries.log
long_query_time=3
tmp_table_size=32M
max_heap_table_size=32M
#log-queries-not-using-indexes=/var/log/mysql/mysql_queries_no-indexes.log
join_buffer_size=2M
low_priority_updates=1
read_rnd_buffer_size=2M
[mysql.server]
user=mysql
basedir=/var/lib
[safe_mysqld]
err-log=/var/log/mysqld.log
var/lib/mysql/mysql.pid
open_files_limit=8192
[mysqldump]
quick
max_allowed_packet=16M
[mysql]
no-auto-rehash
[isamchk]
key_buffer=32M
sort_buffer=32M
read_buffer=16M
write_buffer=16M
[myisamchk]
key_buffer=32M
sort_buffer=32M
read_buffer=16M
write_buffer=16M
[mysqlhotcopy]
interactive-timeout
Whenever there are a lot of members online on my forum, the iowait shoot up, >30% and often times hovering at 60%. It'll eventually drop down to normal levels. However, during the high iowait, there are over 2.5GB of free memory according to the command, free -m
is it due to mysql not optimized or other processes? I don't think the bottleneck is on the cpu or ram.
View 12 Replies
View Related
Nov 22, 2008
diagnosing my server which currently experiencing high iowait for weeks, I'm still figuring the root cause.
total used free shared buffers cached
Mem: 4051 3653 398 0 341 2847
-/+ buffers/cache: 464 3587
Swap: 8191 0 8191
The server spec as follow :
HP DL 380 G5
4G RAM
2.6.18-53.el5PAE
the load is around 1 or 2 , but iowait have been on the high side even the server currently is under utilized
I'm running LVM and hardware RAID 1 on the server.
View 5 Replies
View Related
Jul 2, 2008
I am running in a High load problem lately. I have one of those cheap 1and1 servers which was running fine until 2 weeks ago. Once I rebooted accidentaly, it did not come back with some unrepairable kernel errors and I had to re-image it.
I chose to reimage the server with CentOS 5, for better support. The new re-image worked fine for some days, at least so I thought and now I am having high loads. The server crashes if not monitored every moment as the load is unpredictable.
Just a restart of the Apache will bring the server back to normality, but I am not sure if it is apache or some other script to be blamed.
I have beeing monitoring through apache server-status, but I cannot organize something unusual in the high load moments.
At this moment for example:
top - 09:00:36 up 15:43, 3 users, load average: 3.57, 4.60, 5.32
Tasks: 144 total, 2 running, 142 sleeping, 0 stopped, 0 zombie
Cpu(s): 10.6%us, 6.5%sy, 0.0%ni, 18.7%id, 63.5%wa, 0.2%hi, 0.5%si, 0.0%st
12:00:29 AM CPU %user %nice %system %iowait %steal %idle
12:10:01 AM all 9.14 0.00 5.52 44.66 0.00 40.68
12:20:14 AM all 6.83 0.00 3.98 27.88 0.00 61.32
12:30:10 AM all 6.44 0.00 4.20 81.25 0.00 8.11
12:40:09 AM all 5.25 0.00 4.09 81.93 0.00 8.73
12:50:15 AM all 5.11 0.00 3.79 90.74 0.00 0.36
01:00:07 AM all 7.22 0.00 4.52 57.11 0.00 31.15
01:10:13 AM all 6.89 0.00 4.01 55.38 0.00 33.71
01:20:14 AM all 4.37 0.00 3.27 41.88 0.00 50.48
01:30:25 AM all 4.26 0.00 3.29 63.42 0.00 29.03
01:40:06 AM all 27.18 0.00 4.75 58.27 0.00 9.80
01:50:03 AM all 29.64 0.00 6.61 51.50 0.00 12.25
02:00:07 AM all 27.00 0.00 8.48 55.49 0.00 9.03
02:10:10 AM all 19.29 0.00 4.97 73.80 0.00 1.94
02:20:04 AM all 37.85 0.00 6.78 40.70 0.00 14.67
02:30:05 AM all 15.65 0.00 4.80 68.47 0.00 11.08
02:40:08 AM all 9.06 0.00 5.60 37.49 0.00 47.86
02:50:07 AM all 5.36 0.00 3.62 42.29 0.00 48.73
03:00:02 AM all 6.05 0.00 4.08 47.27 0.00 42.60
03:10:02 AM all 4.22 0.00 3.68 38.17 0.00 53.93
03:20:02 AM all 4.06 0.00 3.75 41.37 0.00 50.82
03:30:22 AM all 4.42 0.00 3.93 45.25 0.00 46.41
03:40:11 AM all 4.34 0.00 3.95 39.58 0.00 52.13
03:50:02 AM all 4.67 0.00 4.01 32.53 0.00 58.80
04:00:08 AM all 3.72 0.00 3.87 28.40 0.00 64.02
04:10:02 AM all 13.49 0.00 6.58 20.82 0.00 59.10
04:20:01 AM all 6.70 0.00 4.63 6.06 0.00 82.61
04:30:02 AM all 1.44 0.00 1.21 4.75 0.00 92.59
04:40:01 AM all 12.42 0.00 8.12 7.65 0.00 71.81
04:50:02 AM all 1.43 0.00 1.07 4.02 0.00 93.47
05:00:02 AM all 1.60 0.00 1.40 8.62 0.00 88.38
05:10:10 AM all 3.80 0.00 3.02 17.86 0.00 75.32
05:20:06 AM all 5.10 0.00 4.22 23.34 0.00 67.34
05:30:02 AM all 1.54 0.00 1.40 11.22 0.00 85.85
05:40:05 AM all 1.75 0.00 1.89 13.12 0.00 83.23
05:50:12 AM all 2.15 0.00 2.22 18.92 0.00 76.72
06:00:02 AM all 1.92 0.00 2.01 12.87 0.00 83.20
06:10:02 AM all 2.27 0.00 2.16 11.53 0.00 84.04
06:20:03 AM all 3.56 0.00 3.02 25.26 0.00 68.16
06:30:10 AM all 2.66 0.00 2.05 18.13 0.00 77.16
06:40:02 AM all 2.58 0.00 2.25 22.87 0.00 72.30
06:50:02 AM all 2.68 0.00 1.92 15.77 0.00 79.63
07:00:03 AM all 3.06 0.00 2.48 26.01 0.00 68.46
07:10:03 AM all 3.65 0.00 3.20 36.54 0.00 56.61
07:10:03 AM CPU %user %nice %system %iowait %steal %idle
07:20:03 AM all 4.40 0.00 3.28 43.86 0.00 48.46
07:30:02 AM all 4.10 0.00 3.17 31.30 0.00 61.43
07:40:06 AM all 7.67 0.00 3.95 50.79 0.00 37.59
07:50:02 AM all 4.72 0.00 3.11 44.30 0.00 47.86
08:00:03 AM all 5.57 0.00 3.72 47.15 0.00 43.56
08:10:07 AM all 10.66 0.00 3.59 71.62 0.00 14.13
08:20:17 AM all 5.67 0.00 3.42 58.81 0.00 32.10
08:30:10 AM all 11.12 0.00 3.49 76.71 0.00 8.67
08:40:03 AM all 7.00 0.00 3.36 47.94 0.00 41.71
Average: all 7.53 0.00 3.76 38.90 0.00 49.81
Some configurations:
The reimage partittioning looks like this:
/dev/sda1 950M 138M 765M 16% /
/dev/sda5 4.7G 1.9G 2.9G 40% /usr
/dev/sda7 63G 58G 5.0G 93% /var
/dev/sda6 4.7G 23M 4.7G 1% /home
none 1.0G 500K 1.0G 1% /tmp
KeepAlive On
MaxKeepAliveRequests 300
KeepAliveTimeout 15
<IfModule prefork.c>
StartServers 10
MinSpareServers 20
MaxSpareServers 50
ServerLimit 256
MaxClients 180
MaxRequestsPerChild 2000
</IfModule>
<IfModule worker.c>
StartServers 2
MaxClients 150
MinSpareThreads 25
MaxSpareThreads 75
ThreadsPerChild 25
MaxRequestsPerChild 0
</IfModule>
[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
skip-locking
query_cache_limit=1M
query_cache_size=32M
query_cache_type=1
max_connections=250
interactive_timeout=100
wait_timeout=20
connect_timeout=10
thread_cache_size=80
key_buffer=64M
join_buffer=1M
max_allowed_packet=16M
table_cache=512
record_buffer=1M
sort_buffer_size=1M
read_buffer_size=1M
max_connect_errors=10
thread_concurrency=2
read_rnd_buffer_size=1M
myisam_sort_buffer_size=64M
server-id=1
processor : 1
vendor_id : GenuineIntel
cpu family : 15
model : 3
model name : Intel(R) Pentium(R) 4 CPU 2.80GHz
stepping : 4
cpu MHz : 2793.324
cache size : 1024 KB
View 10 Replies
View Related
