Limiting MySQL Queries
Jun 30, 2007Is there any way to set max MySQL queries per hour to all MySQL users at once?
(lets say 20000 queries for each user?)
(cPanel/SuSE 10.0/MySQL 5.0.X).
Is there any way to set max MySQL queries per hour to all MySQL users at once?
(lets say 20000 queries for each user?)
(cPanel/SuSE 10.0/MySQL 5.0.X).
I'm currently hosting all of my stuff with my own Wild West Reselling Account, i.e., InnoDomains.com.
Problem is I'm running multiple Wordpress Installations and a few websites that have some pretty intense MySQL work going on in the background, and I use about 250GB per month in bandwidth.
My Account gets the 503 downtime errors because according to Customer Service I cannot run more than 50 MySQL queries in one second. Well, that's not much of a challenge because a lengthy wordpress page runs about 90 queries and loads in 2-3 seconds. Time this by three including the other websites and a couple of shopping stores, and I'm easily doing more than 100-150 queries a second more than several times a day, thereby seeing that nasty, irritating error.
My other option is to split up all the websites over basic $ 3.95 per month accounts. I don't want to switch to a dedicated server yet because I'm not really making a whole lot of money (about $ 30 per month). Another thing I could do is move my MySQL testing to a local machine, and that may actually reduce the load on the damn server.
Are there any hosts out there that will allow me to do more than 100 MySQL queries per second? If so, are there any that are any good?
I added the following to my.cnf and restarted mysql
log_slow_queries=/var/log/mysql/mysql_slow-queries.log
I saw at leat 53 slow queries, however, there is no such file in that folder.
This has happened twice now. I have my wait_timeout variable set to 60 seconds, but occasionally my database gets overloaded with "locked" queries. Here's an example of my processlist:
Code:
| 14773848 | db_name | localhost | db_name | Query | 35483 | statistics | SELECT COUNT... |
| 14776290 | db_name | localhost | db_name | Query | 35144 | statistics | SELECT COUNT... |
| 14778877 | db_name | localhost | db_name | Query | 34841 | statistics | SELECT COUNT... |
| 14790744 | db_name | localhost | db_name | Query | 33056 | Locked | UPDATE... |
| 14791275 | db_name | localhost | db_name | Query | 32977 | Locked | SELECT... |
| 14791395 | db_name | localhost | db_name | Query | 32960 | Locked | SELECT... |
| 14791556 | db_name | localhost | db_name | Query | 32929 | Locked | SELECT... |
| 14791603 | db_name | localhost | db_name | Query | 32924 | Locked | SELECT... |
| 14791757 | db_name | localhost | db_name | Query | 32900 | Locked | SELECT... |
| 14792209 | db_name | localhost | db_name | Query | 32815 | Locked | SELECT... |
| 14792355 | db_name | localhost | db_name | Query | 32786 | Locked | SELECT... |
This continues on with locked queries all the way up to my limit of 450 connections, then the whole server goes down. Note the time spent processing these queries, 35483 seconds and counting! Surely the wait_timeout is supposed to prevent this from happening?!
Also, this appears to start with queries in the State "statistics". Then the State for the following queries becomes "locked". Does anyone know what these mean? Can I get them to timeout normally? Queries which never timeout are guaranteed to bring your server down eventually, surely this is not the desired behaviour.
I've been pondering something for a long time here, but never really knew the right question to ask, or how to word it... so I never even know what to search for. So I honestly have no idea what I'm asking or if this is even under the right topic. What I'm looking for is answers, links, tutorials, or just some words I can search for to help me do the following:
I have a MySQL database filled with accounts. People have to pay to get these accounts. All the accounts are set to expire on different days. I need to send an e-mail out to accounts, for example, to let them know they're expiring, say, a month before this will occur. Right now the only way I know to do this is to have MySQL queries on a PHP page that, every time the page is loaded, runs through all the accounts, checks their expirations, and sends emails to those that need it. I know there has to be a way to have this happen automatically without having to load a PHP page every day, for example.
I know about the general log but want to just log queries of a specific user. Our database does 400 queries/s on a slow day, so I think it would be a pain to grep though a huge log like that. And, im not sure if it would cause any preformance issues.
View 0 Replies View RelatedWhat I did here was create /var/log/mysqld.slow.log for queries that are taking one second or longer. After around 36 hours, here is what I came up with. Since the file size of /var/log/mysqld.slow.log was 219KB, posting it in its entirety it not practical. Therefore, I used a program, the perl script known as mysql_slow_log_parser found on www.retards.org, to parse the mysqld.slow.log to make it easier to interpret and read. The result was that the original filesize of 219KB was converted to the smaller size of 14KB....
View 7 Replies View RelatedI have fairly a large web site that has a forum and a torrent tracker.
Currently MySQL server is handling about 150 queries an avarage per second.
Here is the server spec:
Core2Duo 2.66Ghz
4Gb RAM
320GB SATA 7200RPM (Server provider does not have 1.5K RPM nor 1.0k RPM)
100Mbit Connection (servers on the same switch and the switch does not have 1Gbit port)
MySQL Version: 5.0.51a
I had Master-Master Replication setup with forum running on one and the tracker running on the other.
Although this has been working for about few days, we started seeing lags in the replication process.
After a week, there is a major lag and the changes made on one of the servers takes about 5 hours to appear on the another.
So, this doesn't work.
What would be the other ways of splitting MySQL queries concerning the same database?
While I was researching, I read about MySQL Cluster with database storage engine being NDB.
But, let's say that there is a power failure on both the nodes at the same time, then I would lose the whole database as the database is stored on the memory correct?
I would not like to take that chance either, but if this is faster then replication method then maybe I will concider.
I thought about editing the forum coding to make all queries that concerns the tracker to go in to, say server B (with forum's primary MySQL server being Server A), and make the tracker use server B as MySQL backend, but it seemed like a heavy work so that will be the last choice.
I have a VPS with 256MB ram. i want to limit mysql memory access. suppose i want that mysql should not use more than 128MB ram.
is this possible? if yes how?
is this possible for all other services too?
I've got 25 domains on a Virtuozzo/Plesk8.6/CentOS5 VPS. Each domain has one up-to-date install of WordPress, most have very little traffic (average 200mb per month), maybe 2 domains get 5-7gb traffic per month.
I monitor port 80 connections and rarely see more than 10 at a time.
That should in my opinion be no problem at all for a VPS with 768mb guaranteed ram and 2.4ghz cpu. I've got 30gb hard drive spare too.
But.... about 8 or 10 times a day it grinds to a complete halt: server load at 500-1000%, sites timing out, plesk takes 3mins to load, often I can't even connect with SSH, and the plesk web server, apache
INSERT INTO module_watchdog_sys_stat (time, type, value, service_id) VALUES(FROM_UNIXTIME(1226404705), 'MAINMEM_USAGE', 17472, 11);
80 seconds sounds like a huge amount of time for a MySQL insert to me! Does anyone know if this is likely to be the cause of my trouble? Some problem with Plesk and the database? Or could it be something else?
We host our DNS with DNSmadeeasy.com. We noticed that the number of DNS requests is too high. So we decided to contact their support and ask them for log file, so we can determine
what's going on.
Reply we got from them is the following:
We researched your queries and we found that many of them were coming from the SoftLayer resolving name servers. We phone their admins to report the abuse and they said that the queries were infact coming from your own server with them.
We will not charge for any overage on the queries this month so you can research what on your server is causing these queries. But we will provide you with logs of about 5% of our your traffic.
here is the example of the line from log file sent by them:
May 10 17:15:48 t145 named[29945]: client 66.228.118.72#52529: query: host-196.218.57.145.tedata.net.mydomain.com IN AAAA -E
May 10 17:15:48 t145 named[29945]: client 66.228.118.72#52529: query: afontenayssb-151-1-40-153.w83-114.abo.wanadoo.fr.mydomain.com IN AAAA -E
I changed the real domain name to "mydomain.com"
I checked the apache access log and found these hosts there in exact order as they are in log file sent by dnsmadeeasy.
What is strange is that why is my server making back these requests ? Also these requests are bad because these are not valid domain names:
e.g. should be "host-196.218.57.145.tedata.net" instead of "host-196.218.57.145.tedata.net.mydomain.com"
Seems that it includes ".mydomain.com" to the end of every request.
Is there a way to discover what is making those DNS queries?
how can i run queries in dnstuff.com
for examlpe we can run following query for DNS Lookup MX record
Code:
nslookup -q=MX www.webhostingtalk.com
ISP Cached DNS Lookup =?
E-mail Test = ?
ISP Cached DNS Lookup =?
I have 2000 domains set to
NS1.anydomain.com NS2 etc
Is it possible, through some script or otherwise, to send the traffic to certain/chosen domains to a completely different set of nameservers.? Preferably without having too much server load?
I already use a domain catchall, so no NS records are actually registered on the server.
I am checking DNS Report for a domain and in nameservers records I am getting following:
Recursive Queries
I could use the nameservers listed below to performe recursive queries. It may be that I am wrong but the chances of that are low. You should not have nameservers that allow recursive queries as this will allow almost anyone to use your nameservers and can cause problems. Problem record(s) are:
xxx.xxx.xxx.xxx
I am using Windows 2003 Standard and Microsoft DNS Server. Any ideas what is this all about and how to fix it?
I have just upgraded my website, and set 301 redirects for all old pages.
But on the old website I used a search engine, and in Google Webmaster there are loads of 404 pages with search queries. They look like this: [URL] ....
Simply redirecting search.php doesn't work, so I guess I need to use a wildcard?
I got a dedicated server which hosted 45 sites, one of those sites contains a vbulletin forum application since january 2007 , but from two weeks i found that there is a high amount of Queries from the this database which leads the server load to be always in RED and exceeds 200 or 300! and the memory is totally consumed!
i tried alot to solve this problem as :
- reinstall the vb.
- change the vb path to another ( considering it as attack)
- installing BFD , APF , ( considering it as attack )
- upgrading to apache 2.2.6 , PHP 5.2.5 , MySQL 5.0.24 ( considering it as a bug)
this is the sample of queries.
Code:
46978hakawyc_SKJI8jjlocalhosthakawyc_froumvbSleep40
NULL46980hakawyc_SKJI8jjlocalhosthakawyc_froumvbSleep38
NULL46990hakawyc_SKJI8jjlocalhosthakawyc_froumvbSleep38
NULL47034hakawyc_SKJI8jjlocalhosthakawyc_froumvbSleep30
NULL47086hakawyc_SKJI8jjlocalhosthakawyc_froumvbSleep23
NULL47092hakawyc_SKJI8jjlocalhosthakawyc_froumvbSleep19
NULL47095hakawyc_SKJI8jjlocalhosthakawyc_froumvbSleep23
NULL47099hakawyc_SKJI8jjlocalhosthakawyc_froumvbSleep23
NULL47122hakawyc_SKJI8jjlocalhosthakawyc_froumvbSleep24
NULL47124hakawyc_SKJI8jjlocalhosthakawyc_froumvbSleep24
NULL47126hakawyc_SKJI8jjlocalhosthakawyc_froumvbSleep24
NULL47127hakawyc_SKJI8jjlocalhosthakawyc_froumvbSleep24
NULL47142hakawyc_SKJI8jjlocalhosthakawyc_froumvbQuery22Sending dataSELECT *
FROM datastore
WHERE title IN ('','options','bitfields','attachmentcache','forumcache47143hakawyc_SKJI8jjlocalhosthakawyc_froumvbQuery0Writing to netSELECT moderator.*, user.username,
IF(user.displaygroupid = 0, user.usergroupid, user.displaygroup47144hakawyc_SKJI8jjlocalhosthakawyc_froumvbQuery22LockedSELECT *
FROM datastore
WHERE title IN ('','options','bitfields','attachmentcache','forumcache47145hakawyc_SKJI8jjlocalhosthakawyc_froumvbQuery22LockedSELECT *
FROM datastore
WHERE title IN ('','options','bitfields','attachmentcache','forumcache47147hakawyc_SKJI8jjlocalhosthakawyc_froumvbQuery22Writing to netSELECT *
FROM datastore
WHERE title IN ('','options','bitfields','attachmentcache','forumcache47151hakawyc_SKJI8jjlocalhosthakawyc_froumvbSleep16
NULL47159hakawyc_SKJI8jjlocalhosthakawyc_froumvbSleep17
NULL47166hakawyc_SKJI8jjlocalhosthakawyc_froumvbSleep15
NULL47177hakawyc_SKJI8jjlocalhosthakawyc_froumvbSleep0
NULL47180hakawyc_SKJI8jjlocalhosthakawyc_froumvbSleep16
NULL47211hakawyc_SKJI8jjlocalhosthakawyc_froumvbSleep10
NULL47251hakawyc_SKJI8jjlocalhosthakawyc_froumvbSleep0
NULL47283rootlocalhostNULLQuery0NULLSHOW PROCESSLIST
Once i disabled the database the server load became normal again
another example for Queries :
Code:
117445 hakawyc_SKJI8jj localhost hakawyc_froumvb Sleep 49 NULL
117497 hakawyc_SKJI8jj localhost hakawyc_froumvb Sleep 42 NULL
117667 hakawyc_SKJI8jj localhost hakawyc_froumvb Query 24 NULL SELECT *ntttFROM datastorentttWHERE title IN ('','options','bitfields','attachmentcache','forumcache
117790 hakawyc_SKJI8jj localhost hakawyc_froumvb Sleep 7 NULL
117791 jarabhar_vb localhost jarabhar_vb Sleep 7 NULL
117793 hakawyc_SKJI8jj localhost hakawyc_froumvb Sleep 7 NULL
117795 hakawyc_SKJI8jj localhost hakawyc_froumvb Sleep 8 NULL
117798 hakawyc_SKJI8jj localhost hakawyc_froumvb Sleep 8 NULL
117799 hakawyc_SKJI8jj localhost hakawyc_froumvb Sleep 8 NULL
117802 unauthenticated user localhost NULL Connect NULL Reading from net NULL
117803 unauthenticated user localhost NULL Connect NULL Reading from net NULL
117804 unauthenticated user localhost NULL Connect NULL Reading from net NULL
117805 unauthenticated user localhost NULL Connect NULL Reading from net NULL
117806 unauthenticated user localhost NULL Connect NULL Reading from net NULL
117807 unauthenticated user localhost NULL Connect NULL Reading from net NULL
117808 hakawyc_SKJI8jj localhost hakawyc_froumvb Query 9 Sending data SELECT *ntttFROM datastorentttWHERE title IN ('','options','bitfields','attachmentcache','forumcache
117818 jarabhar_vb localhost jarabhar_vb Query 8 Sorting result SELECT post.postidntttFROM post AS postntttntttWHERE post.threadid = 52572nttttAND post.visible = 1n
117821 hakawyc_SKJI8jj localhost hakawyc_froumvb Query 8 Sending data SELECT *ntttFROM datastorentttWHERE title IN ('','options','bitfields','attachmentcache','forumcache
117829 jarabhar_vb localhost jarabhar_vb Query 8 Sending data SELECT *ntttFROM datastorentttWHERE title IN ('','options','bitfields','attachmentcache','forumcache
117832 hakawyc_SKJI8jj localhost hakawyc_froumvb Query 7 statistics SELECT post.*,ntttIF(visible = 2, 1, 0) AS isdeleted,ntttnnttteditlog.userid AS edit_userid, editlog
117833 jarabhar_vb localhost jarabhar_vb Query 7 statistics SELECT IF(visible = 2, 1, 0) AS isdeleted,ntttnntttNOT ISNULL(subscribethread.subscribethreadid) AS
117834 hakawyc_SKJI8jj localhost hakawyc_froumvb Query 7 statistics SELECT languageid,ntttphrasegroup_holiday AS phrasegroup_holiday,ntttphrasegroup_global AS phrasegro
117835 hakawyc_SKJI8jj localhost hakawyc_froumvb Query 7 Writing to net SELECT *ntttFROM datastorentttWHERE title IN ('','options','bitfields','attachmentcache','forumcache
117836 hakawyc_SKJI8jj localhost hakawyc_froumvb Query 7 Sending data SELECT *ntttFROM datastorentttWHERE title IN ('','options','bitfields','attachmentcache','forumcache
117840 hakawyc_SKJI8jj localhost hakawyc_froumvb Query 7 Sending data SELECT forum.forumid, lastpost, lastposter, lastthread, lastthreadid, lasticonid, threadcount, reply
117841 hakawyc_SKJI8jj localhost hakawyc_froumvb Query 7 Sending data SELECT *ntttFROM datastorentttWHERE title IN ('','options','bitfields','attachmentcache','forumcache
117842 hakawyc_SKJI8jj localhost hakawyc_froumvb Query 7 Sending data SELECT *ntttFROM datastorentttWHERE title IN ('','options','bitfields','attachmentcache','forumcache
117843 hakawyc_SKJI8jj localhost hakawyc_froumvb Query 7 Sending data SELECT *ntttFROM datastorentttWHERE title IN ('','options','bitfields','attachmentcache','forumcache
117845 hakawyc_SKJI8jj localhost hakawyc_froumvb Sleep 2 NULL
117846 hakawyc_SKJI8jj localhost hakawyc_froumvb Killed 0 NULL NULL
117847 unauthenticated user localhost NULL Connect NULL Reading from net NULL
117848 hakawyc_SKJI8jj localhost NULL Connect NULL Writing to net NULL
117849 unauthenticated user connecting host NULL Connect NULL login NULL
117850 unauthenticated user connecting host NULL Connect NULL login NULL
117851 unauthenticated user connecting host NULL Connect NULL login NULL
117852 unauthenticated user connecting host NULL Connect NULL login NULL
117853 unauthenticated user connecting host NULL Connect NULL login NULL
117854 unauthenticated user connecting host NULL Connect NULL login NULL
117855 unauthenticated user connecting host NULL Connect NULL login NULL
117856 unauthenticated user connecting host NULL Connect NULL login NULL
117857 unauthenticated user connecting host NULL Connect NULL login NULL
I am running a SQL test server here that is piping HUGE amounts of data for a test project I am running.
Approximately 30 queries a second - constantly. That's over 2.5M queries a day -- so we are talking big (I think?)
I am currently running this test on a
AMD Phenom 9300, 4GB RAM - SATA 500GB HDD and I am running MySQL 5.0.51a i386 on CentOS
I have programmed a process control for our applications purpose - basically it controls the launching of our SQL intense applications, and stops launching when the Load is greater than 2.5
I have plans to optimize the number of queries (I will build in a cache to some of the applications - and run INSERT statements all together) however I am looking for SQL tweaks that will improve performance. Would running the 64bit version work better?
im running a few blogs on a shared hosting account (steadfast) and i noticed the 18 sql queries being generated by a fresh install of wordpress 2.7 are taking 0.246 seconds to execute.
<!-- 18 queries. 0.246 seconds. -->
considering this is a shared hosting server with over 800+ more domains on it, is the performance better or worse when compared to other shared hosting providers ?
archim3des
For those who run their own DNS servers I am interested in what DNS software your running on BSD or Linux platforms and on average how many DNS queries per second it serves on what hardware specification?
View 3 Replies View RelatedI have a vps at VPSVille, which I use for tunneling out onto the internet (behind a nazi firewall). Unfortunately, my vps keeps getting disabled because I cross the "20GB of traffic per month" allowed to any vps for tunneling - even though I have 300 GBs of bandwidth.
I'm now looking for another cheap vps host ($20/mo or less) that will let me utilize all the bandwidth I paid for.
I am just playing around with a server of mine, and I wanted to know is there anyway to limit the amount of inodes ( files and folders) a user can have? I am using cpanel/whm. Or is there a way I can receive an " alert" once a user has reached the specified limit?
View 3 Replies View RelatedI've recently hired a new server with the aim of offering FTP backups.
I have Plesk CP but I don't want to use that to create the accounts for these users - i want to use the Plesk license for webhosting customers.
My box is running CentOS with ProFTPd.
The problem i'm having, is locking down SSH access to the Backup customers - if i add an account using "useradd", it is enabled in ProFTPd as I want, and I have set ProFTPd to only allow access to their home dir. However, the users can still log in via SSH and have full access to the box. Ideally, I want to be able to remove these users from SSH access, or if that's not easily possible, not allow them to "cd" above their home dir.
I know it's possible to add their usernames to the SSHd config file under Deny, but is there a more elegant solution (bearing in mind i'm planning to have quite a few of these users)?
bandwidth limiting on my home network, to some specified pc's, as they are taking up to much download bandwidth. Im leaning towards windows more but i am still unsure, would any one be able to guide me or give me a few tips on what type of program will do this, what operating system and maybe some tutorials if anyone knows any good ones?
View 6 Replies View RelatedIP limiting on switch port.
Where a switch is set to only allow a specific IPs to be streamed from the servers connected to it.
Any server set with another IP even with a same C class range will be dropped or ignored.
eg.
Switch : with a set IP of 111.111.111.1 - 10
Any server that tries to set an IP of 111.111.111.11 and above will not get connected.
Currently I am using 3com SuperStack III 3250 switches. but after reading the manual and also searching on the net, I am a little lost on ideas....
Is it possible to limit internationally traffic? National would be Australian bandwidth.
For example;
Offering Unmetered national bandwidth and 10 GB international bandwidth.
OR
Offering 50 GB national bandwidth and 5 GB international bandwidth.
How would you go about doing this, tracking it, managing it, etc. Is there any software that does it already? For example Xen or Virtuozzo?
On an cPanel + RHEL 5.3 box at WHM - Tweak Settings, I activated "The maximum each domain can send out per hour (0 is unlimited)" and set that value to "300".
But, it seems that this limit is only if the user is sending using webmail or an email client, right now a joomla website is sending much more than 300 mails per hour, but it's using php to send the mails.
My question, how can I limit emails per hour on each domain while sending from php?
Code:
29457 username 20 0 29724 13m 5848 R 10.2 0.2 0:00.10 /usr/bin/php /home/username/public_html/index.php
We have Linux and windows shared hosting on our dedicated servers using Plesk are we are interested in putting in connection limits and bandwidth limiting in place but not sure what to use - we don't want to set them to be too restrictive but we don't want it to be free reign either.
I was wondering if anyone else was doing this, and if so what their levels were.
Additionally, does anyone use anything for IIS 6 on 2003 that supports bandwidth throttling of dynamic content? I know there are things for IIS 7 in 2008 but that doesn't help much
Is it possible to setup ftp accounts that have the ability for the users to view files on the server and upload their own files, but not download anything from the server? If so, how would this be done, if not, then as a 2nd option, do most hosting companies provide the option to disallow downloads on an ftp account if you need to do so, but allow uploads?
View 4 Replies View RelatedWhich brand / model would you recommend for those:
1 - bw limiting on specific port (like 3mbits/sec on port4)
2 - creating vlans by ports (like vlan1 = port1 + port2)
3 - creating vlans by ips (like vlan1 =x.x.x.1 to 8, vlan2 = x.x.x.9 to 27)
4 - managing the switch from GUI
I've been searching a while for a box that I can throw in front of my servers that will basically look at inbound connections and if there are more than X amount of connections from IP:XXX it will block it for X minutes.
I thought a standard load balancer would do it, but according to Barracuda, their load balancers don't. They do DDOS prevention, but that's based on packet inspection. What I'm looking for is something that will block legitimate traffic when there's just been too much in a short amount of time. Basically I've got people using autosurfing programs and refreshers on my site and I'm getting tired of analyzing log files to find them.
I am interested in setup user activity limits to avoid peaks on the server load, I have readen a lot about PAM and limits.conf but still have no idea on how set this limits. Most of the examples are similar to this page http://www.seifried.org/lasg/users/ but they are still confusing to me
>> I would like to setup rules like this:
Customers may not use more than 2% CPU daily, 3% memory daily, run more than 10 simultaneous processes per user, allow any process to run for longer than 30 CPU seconds, run any process that consumes more than 20% of available CPU at any time, or run any process that consumes more than 16 MB of memory.