MySQL just released an update including "scalability improvements" -- how badly were these needed?
"An update has been released for initial preview release of MySQL 5.4. The release contains scalability improvements and additional DTrace probes for diagnostic troubleshooting on Solaris."
after enduring not one but now 2 price hikes and being promised better service, etc...we still had to wait 25 minutes for a reboot last night.
Yet one more reason we are moving ALL our servers away from them!
I have used this forum before to find good suggestions for VPSs, but now I'm in need of something that I have no idea where to go to to get the solution, or even what that solution may be.
First, my situation is this...
I've built a web application in PHP for an emerging company and it's primary function is to crawl remote websites that we are provided API access to (via lib_curl_multi). The company's clients login to their account and initiate the crawler on a domain of their choice that has API access. They are then added to a queue (to prevent abuse/server overloading) which performs the crawl. The crawl process takes anywhere from 1 minute to an hour and uses anywhere from 50KB of bandwidth to 30MB depending on the remote domain.
The client anticipates their first 3 months needs to be 1,000 crawls every 24 hour period, obviously meaning crawls would have to be done in simultaneous 'groups' to ensure all 1,000 in the queue can be done throughout the day.
That means for their first 3 months, they need bandwidth of about 10-15GB per day and I have NO CLUE what kind of hardware setup they'll need.
That's not really the issue though as I'm sure most dedicated server setups found here can support that. The problem is that after that 3 months they anticipate a ten-fold increase (obviously this would be a gradual build-up) in the number of crawls needed to be done daily, meaning 10,000.
Now that's a huge increase in bandwidth as well as CPU and memory needs. What kind of setup and/or host could accomodate this constant need for scaling without charging ridiculous prices?
My theory is there needs to be one dedicated server or VPS to serve up the website and its content, whereas there can be one or many dedicated servers (expand as they grow kind of deal) that process the crawler queue in the background (hopefully geographically dispersed as their clients are worldwide). EDIT: I forgot to mention that if the website server is separate from the others, they MUST share the same MySQL database as that is where the queue is stored.
I hope I didn't confuse anyone. I'm great with programming, but hardware and hosting's not my strong point so please let me know if you need clarification.
I'm just curious as to what kind of things the huge sites--Youtube, Myspace, etc.--are doing to try to keep scalable. What sites do you guys just hate for failing in this regard, and perhaps most importantly, what are some ways we can prevent downtime?
View 4 Replies View RelatedDoes support matter if there was 100% uptime and scalability?
Our team has been developing scalable sites since 2004. We started renting servers from Layeredtech then, since they had good reviews and they were still good until we migrated away from dedicated server land. Although we have systems administration backgrounds, it still took time away from developing software in order to administer the servers (look over logs, backups/restores, performance graphs, hardware failure, etc). Having said that, one thing I've noticed is that customers are usually happy if servers are always running and running fast.
To get rid of the systems administration part we tried Mosso (they had just released, great support but a lot of problems), we tried mediatemple's grid (also had a lot of problems), couldnt try EC2 because of persistent storage, and lastly we are currently using thegridlayer (it lags, the initial request takes about a second to display a page with no load on the server).
The next things to try were VPS then managed dedicated servers. We decided to try VPSes so we can isolate sites from each other and add VPSes as needed for specific sites. So I got a zone.net and they were running fine until they had a problem mentioned here. People recommended them because they had fast servers, now is the opposite because of this one downtime.
So finally, my questions:
1) how much do you think support is needed if your hosts provides fast servers and 100% uptime?
2) What measures do you take (if any) to verify the host's procedures such as backups, company size, profitability, etc?
3) How do you verify that a host is not overselling before buying a hosting package (assuming shared or VPS)?
Since my /var partition is full, so I moved /var/lib/mysql to /backup/mysql/.
Seems all the files are copied and I changed my.cnf reboot mysql, but all the web sites using db is not working anymore..
I installed the MySQL binary packages in /usr/local/mysql/ after removing the MySQL RPM package. MySQL is functioning when I executed /usr/local/mysql/bin/safe_mysqld. I reinstalled MySQL before I installed PHP. When I used a PHP script to access a MySQL database, it outputs an error:
Code:
Warning: mysqli::mysqli() [function.mysqli-mysqli]: (HY000/2002): can't connect to local mysql server through socket /var/lib/mysql/mysql.sock in index.php on line 2
However, I installed MySQL in /usr/local/mysql, not in /var/lib/mysql. How do I fix MySQL?
For some reason mysql wont start, i have tried restarting mysql but it wont, it says FAILED. The mysql.sock file seems to have disappeared and i cannot find it anywhere.
View 5 Replies View RelatedI recently had a harddrive failure and luckliy I can still access certain directories on this failed drive. I can still access the /var/lib/mysql/ directory which holds all the users databases and have backed all these up separately using tar.
Now what I need to know is how do you restore these database files to another server? I tried simply untar'ing one of these to the new servers /var/lib/mysql/ direcotry and it stuffed Mysql up - it went offline. I had to get a cpanel tech to bring Mysql back online.
how can I get these database files to fully work on a new server?
Can someone recommended me some one with knowledge of mysql exploit or mysql injection, it seem to our VB forum have issue with database load..
View 5 Replies View RelatedI've been racking my brains with this problem for the last couple of months and have made zero progress. I've asked a lot of people if they know what might be wrong here, but none of them have any idea.
Basically, the problem is that any sort of service monitor I put on my server shows MySQL as being down/offline, even when I know for sure it's up and running perfectly fine. All other services report a green light.
The mysqld service is running on port 3306, which is open both inbound and outbound on my APF config, so as far as I can tell it's nothing to do with the firewall (I won't rule that out though).
The other odd thing is that MySQL shows a green light within WHM, but not anywhere else.
Does anyone know what might be wrong here? Am I missing something entirely fundamental and obvious?
my /var partition is full,
im not sure if any files i can remove to get more free space on /var partition.
because there are more free space on /home,
i think if i can move all the /var/lib/mysql to /home/mysql,
if yes,howcan i move it and do any change,
can let the sql data do not lose,
andrun well in the feature ?
I've rented a dedicated server that comes with Red Hat Enterprise Linux 4 and MySQL 3.23 but I need at least MySQL 4.1 or higher. My knowledge about GNU/Linux is very limited, so I've read some documents at mysql.com and now I think I have to follow these steps:
1. Uninstall MySQL 3.23 with this command line:
shell> rpm -e mysql-*.rpm
2. Install the server and client RPMs of MySQL 5.0:
shell> rpm -i MySQL-server-VERSION.i386.rpm
shell> rpm -i MySQL-client-VERSION.i386.rpm
I have a number of web sites on a dedicated server. Some of these web sites are a few years old. Are there any issues upgrading from MySQL 4.1 to 5.x? Are there web sites that may have compatibility issues?
View 6 Replies View RelatedI keep getting this error :
MySQL Error : MySQL server has gone away
How do I fix it?
my.cnf contents
Quote:
[mysqld]
safe-show-database
old_passwords
back_log = 75
skip-innodb
max_connections = 800
key_buffer = 48M
myisam_sort_buffer_size = 64M
join_buffer_size = 1M
read_buffer_size = 1M
sort_buffer_size = 2M
table_cache = 3000
thread_cache_size = 128
wait_timeout = 90
connect_timeout = 30
tmp_table_size = 128M
max_heap_table_size = 64M
max_allowed_packet = 256M
net_buffer_length = 16384
max_connect_errors = 10
thread_concurrency = 4
read_rnd_buffer_size = 786432
bulk_insert_buffer_size = 8M
query_cache_limit = 3M
query_cache_size = 48M
query_cache_type = 1
query_prealloc_size = 262144
query_alloc_block_size = 65536
transaction_alloc_block_size = 8192
transaction_prealloc_size = 4096
default-storage-engine = MyISAM
max_user_connections = 500
[mysqld_safe]
nice = -5
open_files_limit = 8192
[mysqldump]
quick
max_allowed_packet = 16M
[myisamchk]
key_buffer = 64M
sort_buffer = 64M
read_buffer = 16M
write_buffer = 16M
[mysqlhotcopy]
interactive-timeout
I have recently upgraded mysql server from 5.0.75 to 5.1.31 on my Ubuntu server 9.04 32 bit. After that when I am running phpmyadmin it is printing a warning :
Your PHP MySQL library version 5.0.75 differs from your MySQL server version 5.1.31.
I'm trying to upgrade from MySQL 4 to MySQL 5 for performance reasons. MySQL 4 was working great so I decided to just go ahead and upgrade to MySQL 5 via CPanel. The upgrade appeared to go fine however I now get this error when trying to start MySQL.
/usr/bin/mysqladmin: connect to server at 'localhost' failed error: 'Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)' Check that mysqld is running and that the socket: '/var/lib/mysql/mysql.sock' exists! mysql has failed, please contact the sysadmin (result was "Warning, no valid mysql.sock file found.mysql has failed").
I went ahead and did a forced re-install of MySQL 5, but that didn't help. Apache has been updated as well since then, but that did no good, and the MySQL process isn't starting at all.
Its CENTOS with PHP5.
I am not sure what difference between --with-mysql and --with-mysql=/usr to compile php 4.4.7
because I get error message when I use --with-mysql=/usr
the error message is like Mysql header can not be found or similar thing.
it ok when I use --with-mysql
so, I am afraid that can I use php and mysql without problem by using --with-mysql?
I'd like to upgrade my current mysql 4.1 version to mysl 5, but i can't find a good tutorial or explenation! I've been searching for hours now without any good results.
I've downloaded the lastest stable mysql 5 .rpm, and tried to install it with rpm -i mysql.version.rpm, but it says the following:
# rpm -i MySQL-server-standard-5.0.27-0.rhel4.i386.rpm
warning: MySQL-server-standard-5.0.27-0.rhel4.i386.rpm: V3 DSA signature: NOKEY, key ID 5072e1f5
error: Failed dependencies:
MySQL conflicts with mysql-4.1.20-1.RHEL4.1.i386
MySQL-server conflicts with mysql-server-4.1.20-1.RHEL4.1.i386
I guess that means i have to uninstall mysql 4.1 first.
ok...
# rpm -e mysql-server-4.1.20-1.RHEL4.1.i386
warning: /var/log/mysqld.log saved as /var/log/mysqld.log.rpmsave
Let's try to install it now.
# rpm -i MySQL-server-standard-5.0.27-0.rhel4.i386.rpm
warning: MySQL-server-standard-5.0.27-0.rhel4.i386.rpm: V3 DSA signature: NOKEY, key ID 5072e1f5
error: Failed dependencies:
MySQL conflicts with mysql-4.1.20-1.RHEL4.1.i386
Ok, let's remove that.
# rpm -e mysql-4.1.20-1.RHEL4.1.i386
error: Failed dependencies:
libmysqlclient.so.14 is needed by (installed) perl-DBD-MySQL-2.9004-3.1.i386
libmysqlclient.so.14 is needed by (installed) dovecot-0.99.11-4.EL4.i386
libmysqlclient.so.14 is needed by (installed) php-mysql-4.3.9-3.22.i386
libmysqlclient.so.14(libmysqlclient_14) is needed by (installed) dovecot-0.99.11-4.EL4.i386
libmysqlclient.so.14(libmysqlclient_14) is needed by (installed) php-mysql-4.3.9-3.22.i386
mysql = 4.1.20-1.RHEL4.1 is needed by (installed) mysql-devel-4.1.20-1.RHEL4.1.i386
Does upgrade work?
# rpm -Uvh MySQL-server-standard-5.0.27-0.rhel4.i386.rpm warning: MySQL-server-standard-5.0.27-0.rhel4.i386.rpm: V3 DSA signature: NOKEY, key ID 5072e1f5
error: Failed dependencies:
libmysqlclient.so.14 is needed by (installed) perl-DBD-MySQL-2.9004-3.1.i386
libmysqlclient.so.14 is needed by (installed) dovecot-0.99.11-4.EL4.i386
libmysqlclient.so.14 is needed by (installed) php-mysql-4.3.9-3.22.i386
libmysqlclient.so.14(libmysqlclient_14) is needed by (installed) dovecot-0.99.11-4.EL4.i386
libmysqlclient.so.14(libmysqlclient_14) is needed by (installed) php-mysql-4.3.9-3.22.i386
Clearly not.
How do i upgrade mysql?
I have a big issue importing 4.0 databases with mysql 5.0.27.
I did a mysqldump -pxxxxx --force --all-databases > file.sql on server with mysql 4.0
But when I tried to import the file on server with mysql 5.0, this is what I got:
ERROR 1064 (42000) at line 2: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'database' at line 1
I even tried to force import and I got:
ERROR 1046 (3D000) at line x: No database selected for every database.
I checked sql file and CREATE DATABASE xx and USE xx were both added.
I tried this with a single database and worked but with all not
loading a phpinfo() shows that the mysql variable "Client API version" on my server doesn't match the version of mysql installed ( 4.0.20 when mysql version is 4.1.21).
Can this cause trouble and php or mysql errors on scripts i'm running?
What are the files in this directory - /var/lib/mysql/
Files like
#sql_2a6c_0.MYD
#sql_2a6c_2.MYD
etc
etc
I have a stack of them all 200MB plus in size
I need to cleanup /var as its 99% full, what else can I clean up?
MySQL 5.1 GA
Has anyone tried to migrate already?
Mysql usage is very high now and most time server load is high.
Where is issue and how can resolve this?
I'm trying to understand, generally, on a Hybrid VPS with 1gig guaranteed/4gig burst- for small sites that might be installing stuff like Wordpress, a Forum, Joomla!, etc., how many MySQL databases this system can support.
I realize traffic would be a major factor but for discussion sake lets assume an average of 50 users for each db with a peak of 100 users.
Just trying to get a 'rule of thumb'. Wondering also, how much RAM (resources?) an idle db consumes.
i've been using 4.1 without problems but now i'm switching servers and i am wondering whether its a good idea to go for ver 5. How do they compare in terms of performance/security etc? Whats recommended for web use?
View 7 Replies View RelatedMy Server using more MYSQL and CPU
i checked every thing
1 ) i went to : Show MySQL Processes
i want to know what is NULL
root localhost NULL Query 0 NULL SHOW PROCESSLIST
2 ) i went to CPU/Memory/MySQL Usage
Top Process %CPU 122 gzip
Top Process %CPU 110 gzip
Top Process %CPU 89.0 gzip
Would enabling PDO and PDO Mysql modules in apache compile (via whm) be a bad idea? Any possible conflicts or warnings?
It says
(UNSUPPORTED EXPERIMENTAL) Includes SQLite support. Other drivers (freetds, oracle, etc) can be installed with PECL. If you use rawopts to add them during compile then you will have to have them '=shared' and *NOT* do the PHP "versioning" option. Custom opt mods for PDO extensions should also do the "versioning" logic this one does.
I'm already using mysql 5 with mysqli
I have successfully upgraded PHP 4.3.9 to PHP 5.1.6 and MySQL 4.1.20 to 5.0.48 and everything appears to working properly. Couple of questions:
1. Will the databases that were created under 4.3.9 automatically work under 5.0.48? Or do I need to do anything to them?
2. The database information appears to be fine under phpMyAdmin, but I noticed a new database file is now showing. It is named "information_schema". What is this file and is it a new part of MySQL 5.0.48?
3. Does phpMyAdmin 2.6.3 work fine with MySQL 5.0.48? Or do I need to upgrade that as well? And if so, what version works with 5.0.48?
A table in my DB got crashed and I repair it via phpmyadmin it gives me status that its repaired but when I browse it, it says table is marked crashed.
I've tried repairing several times,
started programming for web about a year ago (PHP, mysql, xml etc ) and am completely addicted. Having no prior experience in computing and have been hosting sites on shared server packages. I would, tentatively, like to start hosting my sites on a dedicated servers as i have now realised that it gives you more control over things like security, php / mysql extensions and config files.
I thought i'd start the process on a vps first and make some mistakes there, as i know it can get pricey with dedicated server. A friend reckoned that i should start with Ubuntu on linux (i have no experience with linux, but would like to get involved).
First of all does anyone now any good, cheap vps providers (i've looked at vpsland, feedback? better alternatives?)
and also, can you guys recommend good books/resources that will get me started with server administration (idiots guide!!) or just any good hints to get going! Should i set up on a web server on own pc first or jump in on a vps?
