I wasn't sure where to post this so here goes, I need to migrate a MySQL DB, in the past I have just created an SQL file and used that method (sometimes having to split the SQL file up) but now the DB is about 50 meg and 733,233 records.
Is there an easier way to migrate the Database from one server to another?
I optimized a mysql table of 2 million records and about 500MB.. it took about 15 minutes.. However, on the same DB now i have another huge table of 88 million records, it size is 2.2 GB and it has about 30 MB to optimize... my questions..
1.- How can I speed up the optimization process so it can take the less possible time? any tweaks to my.cnf?
2.- Should I repair it using phpmyadmin or just from the shell?
3.- Should I stop http traffic during this optimization?
This is a dedicated db mysql server that handles a large VB forum with 5-8 users online average:
Code:
Intel Xeon 3.2 x4 procs, 32 bits, 4 GB ram
/etc/my.cnf
Code:
[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
skip-locking
skip-innodb
skip-bdb
query_cache_limit=1M
query_cache_size=48M
query_cache_type=1
max_connections=1200
interactive_timeout=100
wait_timeout=300
connect_timeout=10
thread_cache_size=128
key_buffer=48M
join_buffer=8M
max_allowed_packet=16M
table_cache=2036
sort_buffer_size=1M
read_buffer_size=1M
read_rnd_buffer_size=2M
max_connect_errors=10
# Try number of CPU's*2 for thread_concurrency
thread_concurrency=4
myisam_sort_buffer_size=64M
# Add
max_heap_table_size = 48M
tmp_table_size = 48M
low_priority_updates=1
concurrent_insert=2
[mysqldump]
quick
max_allowed_packet=16M
[mysql.server]
user=mysql
#basedir=/var/lib
[mysqld_safe]
err-log=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid
log-slow-queries=/var/log/slow-queries.log
On the other side, i have the same hardware for the webserver..
if there was a way for me to cut up a large mysql database in shell. We're talking almost 8 gigs here.
I would like to import it in chucks, something like 1 gig per shot. Is that doable with some software that I can use in shell? I wouldn't want to use vim to splice up that sucker.
Or is there a better method of doing this? I already dumped this and I rather not do it again unless if I could perhaps dump this in chunks.
on optimizing mysql
My server specs:
Quote:
Server: Quad Processor Quad Core Opteron 8344 - 1.80GHz - 16 x 512KB cache
Ram: 8 GB DDR2 Registered 667
Disk Controller: SA-SCSI RAID 1 Disk Controller
First Hard Drive: 147GB SA-SCSI 10K RPM
Second Hard Drive: 147GB SA-SCSI 10K RPM
my.conf file:
Quote:
[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
skip-locking
skip-innodb
query_cache_limit=1M
query_cache_size=32M
query_cache_type=1
max_user_connections=25
max_connections=500
interactive_timeout=10
wait_timeout=10
connect_timeout=10
thread_cache_size=128
key_buffer=16M
join_buffer=1M
max_allowed_packet=16M
table_cache=1024
record_buffer=1M
sort_buffer_size=8M
read_buffer_size=8M
max_connect_errors=10
thread_concurrency=16
"/etc/my.cnf" 57L, 893C
[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
skip-locking
skip-innodb
query_cache_limit=1M
query_cache_size=32M
query_cache_type=1
max_user_connections=25
max_connections=500
interactive_timeout=10
wait_timeout=10
connect_timeout=10
thread_cache_size=128
key_buffer=16M
join_buffer=1M
max_allowed_packet=16M
table_cache=1024
record_buffer=1M
sort_buffer_size=8M
read_buffer_size=8M
max_connect_errors=10
thread_concurrency=16
myisam_sort_buffer_size=64M
server-id=1
#[mysql.server]
#user=mysql
#basedir=/var/lib
[safe_mysqld]
err-log=/var/log/mysqld.log
pid-file=/var/lib/mysql/mysql.pid
open_files_limit=8192
#[mysqldump]
#quick
#max_allowed_packet=16M
[mysql]
no-auto-rehash
[isamchk]
key_buffer=64M
sort_buffer=64M
read_buffer=16M
write_buffer=16M
[myisamchk]
key_buffer=64M
sort_buffer=64M
read_buffer=16M
write_buffer=16M
[mysqlhotcopy]
interactive-timeout
is this good enough or we could fine tunning it more?
Since my /var partition is full, so I moved /var/lib/mysql to /backup/mysql/.
Seems all the files are copied and I changed my.cnf reboot mysql, but all the web sites using db is not working anymore..
I'm having a lengthy issue where my databases are to large to import in phpmyadmin using plesk. Unfortunately I dont have direct access to phpmyadmin and can only access it by DB user through plesk.
I have tried to edit php.ini in the following locations:
upload_max_filesize = changed this to 64M
post_max_size = changed this to 32M
maximum_execution_time = changed this to 300
maximum_input_time = changed this to 300
Why am I still not able to import my DB's which are about 8MB each?
I installed the MySQL binary packages in /usr/local/mysql/ after removing the MySQL RPM package. MySQL is functioning when I executed /usr/local/mysql/bin/safe_mysqld. I reinstalled MySQL before I installed PHP. When I used a PHP script to access a MySQL database, it outputs an error:
Code:
Warning: mysqli::mysqli() [function.mysqli-mysqli]: (HY000/2002): can't connect to local mysql server through socket /var/lib/mysql/mysql.sock in index.php on line 2
However, I installed MySQL in /usr/local/mysql, not in /var/lib/mysql. How do I fix MySQL?
what type of hosting do use big sites like Deviantart, or any kind of site with a lot of Gb or Tb. and how much it costs?
View 7 Replies View RelatedI have a website which has about 20K users, and now I am using VPS plan at LunarPages.
However, I have encountered a trouble of out-of-memory. Although I have configured my Apache and MySQL carefully, the 512M memory is not enough. Therefore, the users' expirence is not good these days because my site is very unstable.
I contacted Lunarpages, asking them whether I can upgrade my VPS to bigger RAM, but they said the ONLY way to get a RAM bigger than 512M is to upgrade to dedicated hosting plan.
The following are some stats of my website:
Total Members: 20k
Online at the same time: max 600, average 300
The Lunarpages VPS plan:
www[dot]lunarpages[dot]com/virtual-private-server/
disk space: 20G
RAM: 512M
price: $42 / mo
Now I am not sure whether to migrate to didicated hosting plan, because currently, the main problem is just the size of RAM. Other resources e.g (CPU, network etc. ) are not my bottleneck. So I think it seems not worthwhile for me to migrate to the dedicated hosting plan with a doubled price (even more, almost 3x if I need 1G RAM), just for a larger size of RAM.
Can you guys give some suggestions to choose a VPS provider for my site?
The factors taken into my consideration include:
* RAM size: at least (1G for peak, 768M garantee). The bigger, the better. Nice if can choose larger size when needed.
* price
* bandwidth: 1T/mon?
* easy to upgrade to dedicated host: just in case that one day I will have to use dedicated.
* whether there are coupons for a lower price.
I've been with zone.net for a couple months now, and I have a guaranteed 512MB of memory, which I seem to constantly be hitting, which seems to result in processes being killed and http access vanishing. Growing quite annoying.
I'm looking into moving onto a new provider that can provide more guaranteed RAM for about the same price.
Space isn't a huge deal, I'd do fine with a meager 5GB. Bandwidth I need at least 200GB, but wouldn't mind more.
I'd like to stay managed if possible, as I'm not as well versed in server workings as I should be. Also am in need of cPanel, which I know is a spendy sucker.
My budget is something around $70 a month, and I don't really want to go much higher than that. Still a poor college boy :/
Can anyone suggest such a provider? I've browsed around a lot of the VPS hosts but can't seem to find one that has as much RAM as I need for a decent price. All the ones that seem to have 512MB+ are pretty expensive, and offer a lot more other stuff (space/bandwidth) than I need.
As a final note, the line speed isn't that big of a deal. I'm currently on a 3mbit and am surviving, but going back to a higher speed line would be great
Just had a quick question about backing up a large MySQL DB. I have a database that is 50gb with about half a billion entries in it. One table itself is about 40gb, the other 10gb consists of smaller tables.
The problem is, I want to back the database up and be able to keep it LIVE at the same time (as it will fall behind quickly if it's pulled for more than a few hours, as there are somewhere in the area of a million entries an hour, plus other deletions and queries).
I'm currently using iptables to ban IP addresses from the servers, like:
Code:
iptables -A INPUT -s xxx.xxx.xxx.xxx -j DROP
I ran a "spam trap" for the last few months and now I have over 11000 IP addresses who were trying to spam on my website (guestbooks, phpBB and forms) and I want to ban them all (pretty sure bots run from them).
My question - is iptables the way to do it? I mean does banning such a large number of addresses have any significant performance or other issues I should be aware of (except of the fact I may be banning some legitimate traffic)? Is the -A INPUT the way to ban them all or is there a more appropriate way of baning such a number of addresses?
I'm on CentOS 4.5 i686, Apache/1.3.37, Pentium D 930, 2GB RAM.
I'm trying to find a server that can offer about 7-8 TB with 6-8 GB of ram. Does anyone know a good provider?
View 14 Replies View Relatedany recommendation for a good e-mail service (IMAP) for large accounts? between 5 and 10GB each?
View 13 Replies View RelatedI'm selling downloads of music files. The zip files are quite large. I've had several people complain that they get a message that the server resets their connection before the download finishes.
What can I do to allow these large downloads?
I'm on IIS.
I have a large directory which I want to copy to another account on the same server. Its 1 folder which contains 20000+ files and its around 2GB in size.
I used:
Quote:
cp -r /home/useraccount/public_html/foldername /home/useraccount2/public_html
It worked but some of the files didn't transfer correctly,
Does anyone know any shared hosting or semi-private VPS hosting for a site with large MS SQL2005 DB? Our DB is currently 1GB already but could get up to 5-10GB when all is said and done. The site is a B2B networking platform with many users, company profiles, products, etc.
View 3 Replies View RelatedI have a website that is approx 50GB, that I essentially would like to take offline for a while. Obviously, while the site is offline, I don't want to be paying for my server.
Can anyone let me know of some options to "store" this mammoth of a site. Downloading it locally is not an option, so it needs to stay in the cloud for the lowest possible price.
to delete and reset the domlogs files from Apache server, as they have become too large, but I want to keep a record of the past few years website stats.
I could download the domlogs files (they look like MS-DOS applications) but once I have I am not sure how to read them, is there some off server application that could open these files? I currently view stats in awstats from cPanel, but the size of the files is causing problems with updating now.
The only thing I can think of doing is taking screenshots of the relevant stats that I want to keep, before deleting the files, but was thinking there must be a better solution than that.
Linux Fedora 6, Apache 2 with Mod Security, MySQL.
Our mod_sec logs get incredibly large very quickly. In the configuration for mod_security, we have specified logging options as
SecAuditEngine RelevantOnly
SecAuditLogRelevantStatus "^[45]"
but the mod_sec.log gets to almost 10 GB (in a matter of 5-6 days) before it is truncated to mod_sec.log.1 and a new one is created.
Is there a way we can specify that a max size of one log file is 1 GB, for example?
Or another question, how come it gets so huge so quickly? We thought that logging "RelevantOnly" will only display errors / requests that are deemed security risks.
I have a customer who wants to sell access to videos of conferences he runs.
Each flv vid is approx 1 - 1 1/2 hors long approx 380MB each and there will be about 12 videos per conference.
approx 4 - 8 conferences per year.
My customer suggests 10 - 20 people will buy access to watch each video.
Access to watch the videos will be through a password protected webpage.
issue - the current site hosting company only allow uploads up to 150MB per file.
Can I host the flash videos elsewhere and deliver them through the password protected web page without anyone else being able to see them via server they are hosted on?
This would also reduce the bandwidth going through his current site server.
I have several gigs worth of data which my programmer wants to put into MS SQL 2005. But the last time I check, MS SQL comes in blocks of only a few MB (eg 200MB for $25).
Now if I have 10GB of data, that yould cost me like...thousands?).
One solution is to use mysql which is free of course, but the website is done in ASP.NET. How will this affect performance?
Also, is there any other way which I can purchase SQL 2005 myself? If yes, how much for one domain name?
I would like to have around 100 Gb with room to expand if possible.
This will be hosting for a radio station, archival material.
I am unsure of where to look, the sites I have seen are pretty small in scale as far as capacity.
Further, I would also like to be able to re-sell this, instead of just setting up a straight account.
Further if at all possible I would like to bundled with some other small hosting packages (wordpress sites) on a reselling basis.
My client's website needs to hold files that are around 60 or 70 MB. The host only allows files up to 10 MB. Is that typical?
Right now I'm linking to a file storage but would rather make the files available from my site without going to a 3rd Party Site. He doesn't want to zip his files either - just to be a straight download.
I am trying to locate what large file are filling up the / on the server but I am having trouble using the find command to do this.
View 1 Replies View RelatedI couldn't find a suitable category for this topic, please feel free to move if I made a mistake.
I have a big website.
MySQL Size: 1 GB
Diskspace: 2 GB
It's over loading the server, so I am moving it to a dedicated server.
In the past, basically I would pause the live website (e.g. turn off forum). Move the data, turn everything back on in the new server and then change the nameservers.
But for this website, I don't want to pause the website nor I wish to loose any posts because of DNS propagation. (e.g. user might post data, which is stored in the current server. But then after propagation, the user won't see his post)
My server is running WHM 11.15.0 cPanel 11.18.6-S24739 /
REDHAT Enterprise 4 i686 on standard.
I am using EXIM.
Lately, users on my server have not been able to receive most attachments. E-mails with small attachments, such as an 11,000 byte file, show up just fine. An e-mail from the same person, but with a 560,000 byte attachment will simply never show up, and there is no returned "bounce" message to the sender. It is like the e-mail was never sent.
The odd thing is, e-mails sent to and from e-mail addresses hosted on my server have no trouble sending/receiving large attachments of any size (I have tested up to several megs in size). For example, tom@example.com and joe@another_example.com both have their domains hosted on my server. They can send each other large attachments with no trouble. But, if tom@yahoo.com or tom@gmail.com tries to send a larger attachment to tom@example.com, the e-mail and the attachment never show up.
Things to note:
1) The e-mail accounts are no-where near their storage limits.
2) message_size_limit is not set, so I assume the max attachment limit is the default 50mb.
I have a situation here.
I wanted to transfer a file (about 10GB) from one server to another server. Both server are on the same LAN.
Inside server1, I zipped a folder as file.tar.gz
From server2, I uses wget http://server1.com/file.tar.gz and it says "404 Not Found"
I suspect it won't allow me to download because the file is too big.
to transfer a MySQL database with way over 1 million rows to a different server.
What is the best way to do this without losing any data? I've tried using the export feature in phpMyAdmin but it doesn't seem to handle it very well.
There has to be a feature in WHM or cPanel,
I’ve to buy two new chassis for our redundant storage system. I decided to build my own system.
I looked around to find a good chassis which we can use for the next 3-5 years, the supermicro chassis seems to be the right one. I planed to use a 3Ware Raid Controller with 12x 500GB SATA-II drives in an Raid 6 Array, mirrored with DRBD to a second, identical, system. The System boots from an own Raid Array (Raid 5) with 3 x 120GB Discs on an separate 3Ware Raid Controller.
Now the question is, which chassis, I don’t know if it is better to fit the drives vertically or not.
Here the two chassis:
[url]
[url]