I got a dedicated server which hosted 45 sites, one of those sites contains a vbulletin forum application since january 2007 , but from two weeks i found that there is a high amount of Queries from the this database which leads the server load to be always in RED and exceeds 200 or 300! and the memory is totally consumed!
i tried alot to solve this problem as :
- reinstall the vb.
- change the vb path to another ( considering it as attack)
- installing BFD , APF , ( considering it as attack )
- upgrading to apache 2.2.6 , PHP 5.2.5 , MySQL 5.0.24 ( considering it as a bug)
this is the sample of queries.
Code:
46978hakawyc_SKJI8jjlocalhosthakawyc_froumvbSleep40
NULL46980hakawyc_SKJI8jjlocalhosthakawyc_froumvbSleep38
NULL46990hakawyc_SKJI8jjlocalhosthakawyc_froumvbSleep38
NULL47034hakawyc_SKJI8jjlocalhosthakawyc_froumvbSleep30
NULL47086hakawyc_SKJI8jjlocalhosthakawyc_froumvbSleep23
NULL47092hakawyc_SKJI8jjlocalhosthakawyc_froumvbSleep19
NULL47095hakawyc_SKJI8jjlocalhosthakawyc_froumvbSleep23
NULL47099hakawyc_SKJI8jjlocalhosthakawyc_froumvbSleep23
NULL47122hakawyc_SKJI8jjlocalhosthakawyc_froumvbSleep24
NULL47124hakawyc_SKJI8jjlocalhosthakawyc_froumvbSleep24
NULL47126hakawyc_SKJI8jjlocalhosthakawyc_froumvbSleep24
NULL47127hakawyc_SKJI8jjlocalhosthakawyc_froumvbSleep24
NULL47142hakawyc_SKJI8jjlocalhosthakawyc_froumvbQuery22Sending dataSELECT *
FROM datastore
WHERE title IN ('','options','bitfields','attachmentcache','forumcache47143hakawyc_SKJI8jjlocalhosthakawyc_froumvbQuery0Writing to netSELECT moderator.*, user.username,
IF(user.displaygroupid = 0, user.usergroupid, user.displaygroup47144hakawyc_SKJI8jjlocalhosthakawyc_froumvbQuery22LockedSELECT *
FROM datastore
WHERE title IN ('','options','bitfields','attachmentcache','forumcache47145hakawyc_SKJI8jjlocalhosthakawyc_froumvbQuery22LockedSELECT *
FROM datastore
WHERE title IN ('','options','bitfields','attachmentcache','forumcache47147hakawyc_SKJI8jjlocalhosthakawyc_froumvbQuery22Writing to netSELECT *
FROM datastore
WHERE title IN ('','options','bitfields','attachmentcache','forumcache47151hakawyc_SKJI8jjlocalhosthakawyc_froumvbSleep16
NULL47159hakawyc_SKJI8jjlocalhosthakawyc_froumvbSleep17
NULL47166hakawyc_SKJI8jjlocalhosthakawyc_froumvbSleep15
NULL47177hakawyc_SKJI8jjlocalhosthakawyc_froumvbSleep0
NULL47180hakawyc_SKJI8jjlocalhosthakawyc_froumvbSleep16
NULL47211hakawyc_SKJI8jjlocalhosthakawyc_froumvbSleep10
NULL47251hakawyc_SKJI8jjlocalhosthakawyc_froumvbSleep0
NULL47283rootlocalhostNULLQuery0NULLSHOW PROCESSLIST
Once i disabled the database the server load became normal again
another example for Queries :
Code:
117445 hakawyc_SKJI8jj localhost hakawyc_froumvb Sleep 49 NULL
117497 hakawyc_SKJI8jj localhost hakawyc_froumvb Sleep 42 NULL
117667 hakawyc_SKJI8jj localhost hakawyc_froumvb Query 24 NULL SELECT *ntttFROM datastorentttWHERE title IN ('','options','bitfields','attachmentcache','forumcache
117790 hakawyc_SKJI8jj localhost hakawyc_froumvb Sleep 7 NULL
117791 jarabhar_vb localhost jarabhar_vb Sleep 7 NULL
117793 hakawyc_SKJI8jj localhost hakawyc_froumvb Sleep 7 NULL
117795 hakawyc_SKJI8jj localhost hakawyc_froumvb Sleep 8 NULL
117798 hakawyc_SKJI8jj localhost hakawyc_froumvb Sleep 8 NULL
117799 hakawyc_SKJI8jj localhost hakawyc_froumvb Sleep 8 NULL
117802 unauthenticated user localhost NULL Connect NULL Reading from net NULL
117803 unauthenticated user localhost NULL Connect NULL Reading from net NULL
117804 unauthenticated user localhost NULL Connect NULL Reading from net NULL
117805 unauthenticated user localhost NULL Connect NULL Reading from net NULL
117806 unauthenticated user localhost NULL Connect NULL Reading from net NULL
117807 unauthenticated user localhost NULL Connect NULL Reading from net NULL
117808 hakawyc_SKJI8jj localhost hakawyc_froumvb Query 9 Sending data SELECT *ntttFROM datastorentttWHERE title IN ('','options','bitfields','attachmentcache','forumcache
117818 jarabhar_vb localhost jarabhar_vb Query 8 Sorting result SELECT post.postidntttFROM post AS postntttntttWHERE post.threadid = 52572nttttAND post.visible = 1n
117821 hakawyc_SKJI8jj localhost hakawyc_froumvb Query 8 Sending data SELECT *ntttFROM datastorentttWHERE title IN ('','options','bitfields','attachmentcache','forumcache
117829 jarabhar_vb localhost jarabhar_vb Query 8 Sending data SELECT *ntttFROM datastorentttWHERE title IN ('','options','bitfields','attachmentcache','forumcache
117832 hakawyc_SKJI8jj localhost hakawyc_froumvb Query 7 statistics SELECT post.*,ntttIF(visible = 2, 1, 0) AS isdeleted,ntttnnttteditlog.userid AS edit_userid, editlog
117833 jarabhar_vb localhost jarabhar_vb Query 7 statistics SELECT IF(visible = 2, 1, 0) AS isdeleted,ntttnntttNOT ISNULL(subscribethread.subscribethreadid) AS
117834 hakawyc_SKJI8jj localhost hakawyc_froumvb Query 7 statistics SELECT languageid,ntttphrasegroup_holiday AS phrasegroup_holiday,ntttphrasegroup_global AS phrasegro
117835 hakawyc_SKJI8jj localhost hakawyc_froumvb Query 7 Writing to net SELECT *ntttFROM datastorentttWHERE title IN ('','options','bitfields','attachmentcache','forumcache
117836 hakawyc_SKJI8jj localhost hakawyc_froumvb Query 7 Sending data SELECT *ntttFROM datastorentttWHERE title IN ('','options','bitfields','attachmentcache','forumcache
117840 hakawyc_SKJI8jj localhost hakawyc_froumvb Query 7 Sending data SELECT forum.forumid, lastpost, lastposter, lastthread, lastthreadid, lasticonid, threadcount, reply
117841 hakawyc_SKJI8jj localhost hakawyc_froumvb Query 7 Sending data SELECT *ntttFROM datastorentttWHERE title IN ('','options','bitfields','attachmentcache','forumcache
117842 hakawyc_SKJI8jj localhost hakawyc_froumvb Query 7 Sending data SELECT *ntttFROM datastorentttWHERE title IN ('','options','bitfields','attachmentcache','forumcache
117843 hakawyc_SKJI8jj localhost hakawyc_froumvb Query 7 Sending data SELECT *ntttFROM datastorentttWHERE title IN ('','options','bitfields','attachmentcache','forumcache
117845 hakawyc_SKJI8jj localhost hakawyc_froumvb Sleep 2 NULL
117846 hakawyc_SKJI8jj localhost hakawyc_froumvb Killed 0 NULL NULL
117847 unauthenticated user localhost NULL Connect NULL Reading from net NULL
117848 hakawyc_SKJI8jj localhost NULL Connect NULL Writing to net NULL
117849 unauthenticated user connecting host NULL Connect NULL login NULL
117850 unauthenticated user connecting host NULL Connect NULL login NULL
117851 unauthenticated user connecting host NULL Connect NULL login NULL
117852 unauthenticated user connecting host NULL Connect NULL login NULL
117853 unauthenticated user connecting host NULL Connect NULL login NULL
117854 unauthenticated user connecting host NULL Connect NULL login NULL
117855 unauthenticated user connecting host NULL Connect NULL login NULL
117856 unauthenticated user connecting host NULL Connect NULL login NULL
117857 unauthenticated user connecting host NULL Connect NULL login NULL
I recently started hosting a large vBulletin forum (14,014 registered users... 30-60 online at a time) on one of my dedicated servers. When there's a lot of activity on the forums, things start getting really slow. There will be like 50 php-cgi processes running. Some users get blank pages, others get this message:
Notice: Undefined variable: errfile in [path]/forums/includes/functions_log_error.php on line 163
Specs of the server are: AMD Athlon 64 3000+ 160GB SATA HDD 2GB DDR2 RAM FreeBSD 6.2 (Hosted at AwkNet)
Could there be any Apache modifications I can make to increase performance, or anything else I could try?
We host our DNS with DNSmadeeasy.com. We noticed that the number of DNS requests is too high. So we decided to contact their support and ask them for log file, so we can determine
what's going on.
Reply we got from them is the following:
We researched your queries and we found that many of them were coming from the SoftLayer resolving name servers. We phone their admins to report the abuse and they said that the queries were infact coming from your own server with them.
We will not charge for any overage on the queries this month so you can research what on your server is causing these queries. But we will provide you with logs of about 5% of our your traffic.
here is the example of the line from log file sent by them:
May 10 17:15:48 t145 named[29945]: client 66.228.118.72#52529: query: host-196.218.57.145.tedata.net.mydomain.com IN AAAA -E May 10 17:15:48 t145 named[29945]: client 66.228.118.72#52529: query: afontenayssb-151-1-40-153.w83-114.abo.wanadoo.fr.mydomain.com IN AAAA -E
I changed the real domain name to "mydomain.com"
I checked the apache access log and found these hosts there in exact order as they are in log file sent by dnsmadeeasy.
What is strange is that why is my server making back these requests ? Also these requests are bad because these are not valid domain names:
e.g. should be "host-196.218.57.145.tedata.net" instead of "host-196.218.57.145.tedata.net.mydomain.com"
Seems that it includes ".mydomain.com" to the end of every request.
Is there a way to discover what is making those DNS queries?
Is it possible, through some script or otherwise, to send the traffic to certain/chosen domains to a completely different set of nameservers.? Preferably without having too much server load?
I already use a domain catchall, so no NS records are actually registered on the server.
How will i know if my account is allocated with the ram i bought.
How will i know if my host is over selling?
My sites are going down frequently, I have to restart apache to get them back.I checked the cpu and memory usage logs on whm, and they are well with in limits.
But my hosting support fellow is saying other wise.
Quote:
Our logs indicate that from time to time your VPS is going above the RAM Guarantee, this puts your services in danger of going offline until the RAM usage is back under the guarantee. If you continue to see these problems, we recommend either reducing the RAM usage by disabling any unused services, purchasing a guaranteed RAM upgrade, or upgrading to the next plan, to help alleviate the problem.
I've been with zone.net for a couple months now, and I have a guaranteed 512MB of memory, which I seem to constantly be hitting, which seems to result in processes being killed and http access vanishing. Growing quite annoying.
I'm looking into moving onto a new provider that can provide more guaranteed RAM for about the same price.
Space isn't a huge deal, I'd do fine with a meager 5GB. Bandwidth I need at least 200GB, but wouldn't mind more.
I'd like to stay managed if possible, as I'm not as well versed in server workings as I should be. Also am in need of cPanel, which I know is a spendy sucker.
My budget is something around $70 a month, and I don't really want to go much higher than that. Still a poor college boy :/
Can anyone suggest such a provider? I've browsed around a lot of the VPS hosts but can't seem to find one that has as much RAM as I need for a decent price. All the ones that seem to have 512MB+ are pretty expensive, and offer a lot more other stuff (space/bandwidth) than I need.
As a final note, the line speed isn't that big of a deal. I'm currently on a 3mbit and am surviving, but going back to a higher speed line would be great
I am checking DNS Report for a domain and in nameservers records I am getting following:
Recursive Queries
I could use the nameservers listed below to performe recursive queries. It may be that I am wrong but the chances of that are low. You should not have nameservers that allow recursive queries as this will allow almost anyone to use your nameservers and can cause problems. Problem record(s) are: xxx.xxx.xxx.xxx
I am using Windows 2003 Standard and Microsoft DNS Server. Any ideas what is this all about and how to fix it?
I'm currently hosting all of my stuff with my own Wild West Reselling Account, i.e., InnoDomains.com.
Problem is I'm running multiple Wordpress Installations and a few websites that have some pretty intense MySQL work going on in the background, and I use about 250GB per month in bandwidth.
My Account gets the 503 downtime errors because according to Customer Service I cannot run more than 50 MySQL queries in one second. Well, that's not much of a challenge because a lengthy wordpress page runs about 90 queries and loads in 2-3 seconds. Time this by three including the other websites and a couple of shopping stores, and I'm easily doing more than 100-150 queries a second more than several times a day, thereby seeing that nasty, irritating error.
My other option is to split up all the websites over basic $ 3.95 per month accounts. I don't want to switch to a dedicated server yet because I'm not really making a whole lot of money (about $ 30 per month). Another thing I could do is move my MySQL testing to a local machine, and that may actually reduce the load on the damn server.
Are there any hosts out there that will allow me to do more than 100 MySQL queries per second? If so, are there any that are any good?
I have just upgraded my website, and set 301 redirects for all old pages.
But on the old website I used a search engine, and in Google Webmaster there are loads of 404 pages with search queries. They look like this: [URL] ....
Simply redirecting search.php doesn't work, so I guess I need to use a wildcard?
I started tracking and collecting information about every page load on my site, which made the DB grow very fast. In 24 hours I had over 1.4 million INSERTs, so the DB will grow by over 170MB/day. The inserts so far didnt affect the performance of the server (dual clovertown, 4gb ram, SCSI HD), but at this rate the DB will grow by over 5GB per month. Is it a good idea to let the DB grow to those sizes? Would it affect performance if a table had several 10s of millions of rows? Public doesnt run any SELECT/UPDATE queries on that table, only admin does (me).
Code: tar -cpvf backup-`date '+%d-%B-%Y'`.tar --exclude=/var/www/html/tempjunk /var/www/html To make some backups. It works fine, but here's the problem: it really takes a toll on server load. Server load seems to go up at by around +3 or more. I'm wondering if there is a way to use less CPU usage on that command, or is there another command that uses less usage. I'm not concerned if it backs up slower by starving the amount of CPU usage it has, so if there's a way to do this any method?
This has happened twice now. I have my wait_timeout variable set to 60 seconds, but occasionally my database gets overloaded with "locked" queries. Here's an example of my processlist:
This continues on with locked queries all the way up to my limit of 450 connections, then the whole server goes down. Note the time spent processing these queries, 35483 seconds and counting! Surely the wait_timeout is supposed to prevent this from happening?!
Also, this appears to start with queries in the State "statistics". Then the State for the following queries becomes "locked". Does anyone know what these mean? Can I get them to timeout normally? Queries which never timeout are guaranteed to bring your server down eventually, surely this is not the desired behaviour.
I've been pondering something for a long time here, but never really knew the right question to ask, or how to word it... so I never even know what to search for. So I honestly have no idea what I'm asking or if this is even under the right topic. What I'm looking for is answers, links, tutorials, or just some words I can search for to help me do the following:
I have a MySQL database filled with accounts. People have to pay to get these accounts. All the accounts are set to expire on different days. I need to send an e-mail out to accounts, for example, to let them know they're expiring, say, a month before this will occur. Right now the only way I know to do this is to have MySQL queries on a PHP page that, every time the page is loaded, runs through all the accounts, checks their expirations, and sends emails to those that need it. I know there has to be a way to have this happen automatically without having to load a PHP page every day, for example.
I know about the general log but want to just log queries of a specific user. Our database does 400 queries/s on a slow day, so I think it would be a pain to grep though a huge log like that. And, im not sure if it would cause any preformance issues.
I'm currently hosting my website on core2duo / 2GB RAM dedicated server. Thing is, programmer did a VERY good job optimizing scripts and this server is way too much than I need. The average server load is usually 0.00 - 0.20. I'm thinking of switching to either a cheaper dedicated server or VPS.
We use 300GB-1TB of bandwidth per month though. 90% of our traffic is from US. Do you know any good VPS solution for this?
I am kind of new to the idea of colo and am curious how you know the amount of amps to purchase for your server? I want my machine to have enough to run efficiently without causing hardware failures.
I wasn't sure if this would be better posted in the resellers forum, but that seemed to generally be reviews of companies.
Anyway, I work for a company who is trying to move a large amount of sites from one server to a new one. I was looking for the best way to handle this, since you run into issues with mySQL databases, file permission getting lost in ftp, ect.
I am running a SQL test server here that is piping HUGE amounts of data for a test project I am running.
Approximately 30 queries a second - constantly. That's over 2.5M queries a day -- so we are talking big (I think?)
I am currently running this test on a
AMD Phenom 9300, 4GB RAM - SATA 500GB HDD and I am running MySQL 5.0.51a i386 on CentOS
I have programmed a process control for our applications purpose - basically it controls the launching of our SQL intense applications, and stops launching when the Load is greater than 2.5
I have plans to optimize the number of queries (I will build in a cache to some of the applications - and run INSERT statements all together) however I am looking for SQL tweaks that will improve performance. Would running the 64bit version work better?
What I did here was create /var/log/mysqld.slow.log for queries that are taking one second or longer. After around 36 hours, here is what I came up with. Since the file size of /var/log/mysqld.slow.log was 219KB, posting it in its entirety it not practical. Therefore, I used a program, the perl script known as mysql_slow_log_parser found on www.retards.org, to parse the mysqld.slow.log to make it easier to interpret and read. The result was that the original filesize of 219KB was converted to the smaller size of 14KB....
I have fairly a large web site that has a forum and a torrent tracker.
Currently MySQL server is handling about 150 queries an avarage per second. Here is the server spec:
Core2Duo 2.66Ghz 4Gb RAM 320GB SATA 7200RPM (Server provider does not have 1.5K RPM nor 1.0k RPM) 100Mbit Connection (servers on the same switch and the switch does not have 1Gbit port) MySQL Version: 5.0.51a
I had Master-Master Replication setup with forum running on one and the tracker running on the other. Although this has been working for about few days, we started seeing lags in the replication process. After a week, there is a major lag and the changes made on one of the servers takes about 5 hours to appear on the another. So, this doesn't work.
What would be the other ways of splitting MySQL queries concerning the same database?
While I was researching, I read about MySQL Cluster with database storage engine being NDB.
But, let's say that there is a power failure on both the nodes at the same time, then I would lose the whole database as the database is stored on the memory correct? I would not like to take that chance either, but if this is faster then replication method then maybe I will concider.
I thought about editing the forum coding to make all queries that concerns the tracker to go in to, say server B (with forum's primary MySQL server being Server A), and make the tracker use server B as MySQL backend, but it seemed like a heavy work so that will be the last choice.
How do you limit the amount of processes a user can create? I'm running suEXEc mode and I want to limit the processes to 5 to prevent abuse and resource drainage.
