Recommendations For High Performance Hosting (clustering)
Jun 14, 2008
I'd like to get your recommendations on how I should approach this problem.
Problem:
How to handle large amounts of traffic with for a social network website? If a user uploads a photo or video, how does it become accessible on all of the server? If traffic is expected to be about 500,000 visitors a day, how many machines do you think I should use?
Possible Solution:
I've come up with the following possible infrastructure.
One load balancer. The load balancer has 3 PHP/Apache servers behind it. Behind each of the PHP/Apache servers is a (slave) MySQL server, from which data is read. Behind the slave MySQL servers, there is 1 master MySQL server, which handles all of the database writes. The master MySQL and slave MySQL servers are synced up, so data is up to date.
The actual photo and video files are not stored in the database, only the links to them is stored in the database (to keep the database small). The photo and video reside in a central location (like a SAN or NAS), which is accessible by all of the 3 PHP/Apache webservers.
Questions:
1. How many machines do you think will be able to handle photo and video uploads for 500,000 visitors a day?
2. Is having a SAN with Terabytes of RAIDED disk space an available option?
3. If a SAN or NAS is not an option, does anyone have any ideas on how to make sure all of the web servers have access to the same photos and videos? Is rsync a viable solution?
4. Which hosting provider do you think I should go with?
5. Is clustering what I need? What is clustering and how will it address my concerns?
I'd like to get your recommendations on how I should approach this problem. I posted this in another part of the forum, but I feel that this is a more appropriate place for it.
Problem:
How to handle large amounts of traffic with for a social network website? If a user uploads a photo or video, how does it become accessible on all of the server? If traffic is expected to be about 500,000 visitors a day, how many machines do you think I should use?
Possible Solution:
I've come up with the following possible infrastructure.
One load balancer. The load balancer has 3 PHP/Apache servers behind it. Behind each of the PHP/Apache servers is a (slave) MySQL server, from which data is read. Behind the slave MySQL servers, there is 1 master MySQL server, which handles all of the database writes. The master MySQL and slave MySQL servers are synced up, so data is up to date.
The actual photo and video files are not stored in the database, only the links to them is stored in the database (to keep the database small). The photo and video reside in a central location (like a SAN or NAS), which is accessible by all of the 3 PHP/Apache webservers.
Questions:
1. How many machines do you think will be able to handle photo and video uploads for 500,000 visitors a day?
2. Is having a SAN with Terabytes of RAIDED disk space an available option?
3. If a SAN or NAS is not an option, does anyone have any ideas on how to make sure all of the web servers have access to the same photos and videos? Is rsync a viable solution?
4. Which hosting provider do you think I should go with?
5. Is clustering what I need? What is clustering and how will it address my concerns?
We are considering a new dedicated server host for a set of 25 domains, about 5 of which are very high traffic (80 million clicks a day each).
I am told that the biggest speed boost and performance comes from memory and fast hard disk. So I'm looking for at least a 16GB RAM and SCSI 10k 300GB hard disks.
I am also told that PHP etc is okay, but MySQL is the one that hogs resources after a while. So for the database server I need a high end server.
I like WHM/Cpanel so that should be alright.
A lot of email alerts may be sent through our system (about 500,000 a day on a good day as there's user subscription to updates functionality) so we may need a separate mail server?
My question: is the following config of 4 x quadcore Dunnington Intels be good enough for the above site? Can I run a database on this config for my kind of traffic or do I need a separate server too?
I don't know where to start. I need advice about how to setup a high performance firewall for data center. What big companies use? I suppose that they use Cisco or Juniper firewalls. But I have specific needs. I want to control that firewall with php program - to pass IPs witch can pass the firewall. I think that I have to store them in .txt file. So the firewall must read that file.
In my case I think that I need iptables + IPset or maybe nf-HiPAC? Or transparent proxy like squid or Haproxy?
I'm very confused about the software? The firewall must forward 10GBit of traffic and 60 000+ connections
I am currently hosting my website on one server with the specs:
2.8ghz Dual Quad-Core processor + 8 gigs of ram + two 500 hard drives with a 50 mbps unmetered bandwidth package.
My current problem lies in high server loads and very slow server performance throughout the day.
I am considering migrating over to The Planet onto server with the specs:
3.0ghz Dual Quad-Core + 18 gigs of ram + two 50gb hard drives with 2TB of monthly bandwidth transfer.
In an attempt to have great bandwidth pricing and server performance, I plan on downgrading my current server with my current host to a lowe-end server and keeping it only to host my VIDEO and MUSIC files with the 50mbps unmetered package. The Planet will then host my database and all other web related files on their new server.
Is this a good idea as an attempt to save money in bandwidth costs and eliminating my server lag issues?
I was offered a setup of a separate web and database server at my current host but from what I have read, no one touches the performance and reliability The Planet has to offer.
For the past 3 weeks I have been trying to set up a joomla web site.
At first, I tried to host a joomla website on liquid web's shared host.
They seemed to be quite good. They had all the feature I needed, but then I couldn't upload files into my webspace via ftp.
They told me they don't know what is the problem, and that I should use what is available.
I have stopped hosting at liquid web, and tried a different web host. HostNexus.
There I had the same problem with myu ftp account, but they told me the problem is either in my firewall or ISP.
This really allowed me to look into a solution for this problem, and I partially solved it.
I believe this was the same problem I had with liquid web, but liquid web support didn't tell me the information that helped me to solve this problem.
Though my problems are not solved yet.
I need to create a ftp account into the httpdocs or public html folder so joomla could use it to update its own files.
I was able to create such ftp accounts in liquid web, but it is possible that I am unable to create such ftp account via the hostnexus control panel, and I have asked them if its possible for them to create me such an account.(Asked them topday, so I didn't get a reply yet).
Now, what should I do?
I need a ftp account in order for joomla to work properly.
Should I go back to liquid web?
Should I look for a different web host?
Maybe HostNexus will solve my problem, and all this will be irelavent.
But assuming host nexus won't solve my problem, what should I do?
My boss has recently asked me to source a new host for our servers. At the moment we're running 8 Windows boxes but are unhappy with the service we're receiving from our hosting company. We are looking for a managed hosting company, that has offices in the UK and US to look after the servers.
At the moment we haven't nailed down our exact requirements so I'm just really looking for any recommendations you guys might have. I've done a wee bit of Googling and have found a company called Rackspace but I'm ideally looking to contact around 5 companies to see who can offer us the best deal.
I am planing on running I guess you would call it a semi tube script. It pulls videos from others sites and thumbs them at your own site in categories. The viewers however do view the videos at my site in a popup video player or another page.
Hostgator looks like a good host however I just talked to them and they don't have ffmpeg....
I have accounts on site5 and hostgator (shared plan, not dedicated) and was trying setup knowledgetree a open source document management portal built on Linux/PHP/Apache/MySQL platform.
Everything worked fine except the indexing component as that required Java/JRE. Indexing is very important to me.
So all my hard work for the last 2 days came to a screeching halt.
I badly need to find a hosting service that offers LAMP + Java technologies for a decent price.
A friend of mine basically just wants to get his business on the web. He wants a website that is entirely informational (e.g. no commerce or anything like that). Basically a home page with a couple of links to download some documents and some contact info.
He does not have a domain name, so he would need that.
And the person wants an easy to use website designer for this simple site.
My initial thoughts were:
Microsoft Office Live - but that doesn't seem to be totally free anymore.
Wordpress.com (I am not sure this would work as well especially if you wanted to post documents or something)
I have been building a business around blogs and other social media items on mostly OSS technology - but unfortunately, my current reseller account is inadequate for what I need (especially with sites that query a DB) and unable unfortunately to upgrade to what I might need. I don't want to rip on them because they've provided me great service, but they're just not able to ramp up. My thought is there won't be any reseller to meet my needs after learning more info, so I'm going to make the leap up.
I'm not going to go pound the pavement for hosting business; this is only really for my clients I serve as a agency/consultant/designer and my clients need and deserve more.
I have been considering Mosso; but also weighing a dedicated box as it's around the same price and looking around here for the solutions people recommend for the past few hours. But since I sort of have a specialty application, I thought I'd ask for your recommendations.
I want to say that I came here from another forum because my provider was affected by the Planet outage and the conversation and knowledge was great; you are a very smart community. I hadn't heard of this place beforehand but am glad I found it.
Iīm thinking about using a centralized hosting solution, in order to achieve better redundance and performance while having more room to expand if necessary. In order to achieve this, I was thinking about implementing a storage server, and use a software to provide iSCSI target capabilities.
As storage server, I was thinking about using HP DL320s ( URL ), loaded with 12 147GB SAS HDīs 15K RPM. I will make some tests to understand the real difference between RAID5 and RAID10, concerning write speed. Also, Iīm not sure if the controller provided with this server is good enough to provide a reliable operation.
For switching, I will use HP 2824 or 2848 Gigabit switch, and use port trunk in order to join both NIC controllers of the storage server.
As iSCSI target software, I still donīt know wich one to use. I think FalconStor would be a good bet, however it seems to be a bit expensive. Any good alternative?
This storage server would be used to provide storage for about 10 "regular" hosting servers, that have, at the moment, regular dual 10K SATA Drives (Raptor) in RAID1. I'm afraid the 2x Gigabit ports arenīt enough, even considering that I will not have intensive sequential reads / writes, but random acceses.
I'm going to be rolling out a php/mysql driven site soon and I'm pretty much resigned to the fact that the mysql performance dreamhost has given me isn't going to cut it, its probably too oversold. Simple one table one column selects can take 30 seconds or time out depending on how badly the server is being hammered. HTTP requests are usually snappy, but the mysql performance is bogus.
What is a good host for me to launch this site with? Storage wouldn't need to be too terribly high, at least initially. I'm tempted by MediaTemple's slick marketing, but I've seen on here that some people have had poor sql performance (contrary to what some personal friends have experienced, so I'm torn). I was reading about downtownhost on here, but their load times seemed slow when I hit a couple domains listed on here that are hosted by them.
This host needs to be located in the US. Honestly, I like dreamhost and their panel, save for the sql sluggishness I'm getting.
im running a few blogs on a shared hosting account (steadfast) and i noticed the 18 sql queries being generated by a fresh install of wordpress 2.7 are taking 0.246 seconds to execute. <!-- 18 queries. 0.246 seconds. -->
considering this is a shared hosting server with over 800+ more domains on it, is the performance better or worse when compared to other shared hosting providers ? archim3des
I am using dreamhost host 3 of my web sites and 1 blog. Dreamhost is great, offers alot space and bandwidth.
but I think they are oversellling their space, sometimes it gets really slow. (overselling ? ok, I dont really know, but sometimes its really slow, and most my asian readers said need to refresh to load the page. I am wondering if theres a way to check if they are overselling or not.)
I am thinking about buying vps, even tho, I still got 5 month left with dreamhost.
I found 2 vps companies are highly recommanded on this forum, JaguarPC and LiquidWeb.
theres already a post compared both companies in terms of price and service. I say I will pick JagarPc, cuz, its basic plan just 20 USD, and htey got promotion now, its even cheaper. and basic Liquidweb vps plan is 60 bucks.
I am wondering why Jagarpc is so cheap , are they overselling? how can we check if they are overselling.
I found a few posts saying how good jaguarPc is. and they are not overselling, but those members just signed up this month, and only have 1-3 posts. I cannot really trust those new members.
Can someone share their experience with JaguarPC? compare JaguarPc performance and liquidweb performance. antoher question is switch from dreamhost to JaguarPC basic vPS plan, will performance gets better?
last question: VPS account allows 3 IP, 3ip = 3 domains? if not, how many domains can I have?
company that can offer me super fast php hosting. It will be large loads with upto 20 php requests a second, the load is so much that i am currently running mysql on a dedicated mysql server.
My company are looking for a web hosting solution, that can handle sometime spike of 4000 connections at a time and not really a lots of Data transfers. Lets say 150 GB per month.
Im currently with host gator and have nothing but amazing things to say but i have a music streaming website and its eating up bandwith like crazy but at least the cpu usage of the shared hosting is not that high but i will definitley pass my 2 terabyte mark within the month. My question is even on the dedicated hosting plans on host gator they still dont offer a lot of bandwith so does anyone know any very reliable and good companies that offer a LOT of bandwith for a moderate price. I was looking at liquid web dedicated which offers 3 terabytes as i hear their super reliable and professional
I just set up a cPanel DNS Cluster today. Followed the guide, and everything didnt according to my plan. As in, setup went fine, setting it up went fine too. But I do not think I understood how this solution works in real... or atleast I had assumed it to work in a different way.
So for my questions...
1) How and where do I edit the DNS Zones. On the local system or the DNS dedicated server?
2) How do I create nameservers for this dedicated DNS server? Not the registrar part, but the A record and NS record part.
3) Small doubt, not a big one. Does the DNS server have to be in stand alone mode or synchronized mode?
I am trying to setup a server cluster for my client and was thinking about load balancer. Currently he has 4 Servers and 1 VPS.
4 Servers are in the same datacenter while VPS isn't.
Now from my thoughts about the system load, I have decided to setup like this: VPS being the load balancer 2 Servers being the Apache 1 Server being Storage server 1 Server being MySQL
This will be hosting two forums that has very high concurrent access.
Now the problem is VPS has only 200GB/mo bandwidth while he needs about 800GB total. My question is is there anyway to load balance the servers without the traffic going
like this: LoadBalancer-->Apache Servers-->LoadBalancer-->USER If the traffic goes like that then VPS needs to have about 1600GB bandwidth and will have network bottlenecks.
I would like the traffic to go like the following: LoadBalancer-->Apache Servers-->USERS
I'm currently using Cpanel on all our web hosting machines (total of 6 machines). Each runs its own DNS but also uses the Cpanel cluster feature to sync between all machines.
This has got to the point where DNS updates will not stay, the machines get very lagged etc etc. Now I have put 2 more machines online to do nothing but DNS.
quality shared hosting solution for a WordPress-powered multi user blog. The blog currently has around 1000 posts, 5000 comments and around 2000 unique visitors every day, but those numbers will grow grow exponentially in future, so the hosting in question needs to be expandable since I will eventually have to move to a dedicated server.
Right now, however, I want a quality US-based hosting company that has good connection to EU, with fast support and basic features: PHP5, MySQL5, shell access, ~10GB HD space and ~50-100GB traffic.
First and foremost I am looking for quality and am prepared to pay as much as $25-30 per month.
I have a dedicated server currently hosted over by Aplus.NET
I have a 3000 GB Monthly Transfer limit and we have been going over this limit for the past few months. This has resulted in a large sum of overage fees.
I am looking to go to another hosting company that is just as good as Aplus.NET, if not better... with a better traffic rate. A friend told me about Choopa.com and I wanted to know how good of a company they were. What are some other top reliable hosting companies with premium servers and that specialize in unmetered bandwidth?