How Much Downtime Does It Take To Become Unacceptable
Sep 24, 2009
Recently our provider has been having unplanned long outages (along with a complete server move that took several hours). This has turned out uptime statistic from exemplary to mediocre (100% to <99%).
As we are a hosting company and normally have an extremely good amount of uptime, when is it time to give up on the provider? I am extremely concerned this will become regular and our uptime will simply continue to go down the drain, however, they have been good and have been the main attribution to our 100% uptime (along with our great technicians and hardware).
I'm not going to out the provider before I hear their explanation of it, but I've just had a very unnerving experience with a VPS provider.
Last night my server was experiencing latency issues, but I thought nothing of it until the server went down. I thought the network went down, but it appears that the server went down for emergency maintenance. That's fine, but it would have been nice if they alerted me of it. It went back down an hour later, so I sent my provider an email asking why I could not access my server and if they were aware of the problem.
I receive a response from someone asking for my root password and that they would take a look (he is referring to the latency issue I mentioned, but I was still completely unable to access the server at this poihnt, so I could have cared less about latency -- I just wanted the server up!). I explained the server was still down and asked if he knew anything about it, and this is when he told me about emergency maintenance that had to be done on the server. I also asked if he could ping my server (wondering if it was just a problem on my side), and he responded that he could if he "unpaused the server". He then asked me again to send my root password to diagnose the pinging issue. I think nothing of this email and went to sleep, expecting the issue to be fixed in the morning.
I wake up this morning and my server is still down, but now it's apparent that the host is fine, so I shoot back another email asking why my server has been paused. Fast forward 3 hours, I get a reply with him telling me that I was evasive about giving my password -- I would have gladly given a temporary root password, but I wasn't about to send over my regular password in cleartext over email. This is the part that really bothers me. He then told me that he looked around the files on my server to see if I were hiding anything. I cannot understand how me not giving my root password means I am hiding something. My server was down -- I couldn't change my root password to give to them since they had it paused. If I were abusing bandwidth, I could understand.
All this just because I didn't give him my root password? Am I in the wrong?
I have very sad experience with solarvps and their support quality these days. I'm their user for over year now. Minor problems appeared rarely, but in those cases support reacting time and quality was simply excellent.
Now something wrong happened with them. To problem ticket, which priority is defined as emergency, they answer after 4 hours. Now i'm waiting for last answer already 11 hours.No answer till now. And server is not usable, our customers ar ready to eat us alive. It's going third day as we suffer from problem.
Ok, lets say their support staff is unable to solve this complex problem. But they should at least inform customer what is done to solve problem, what means are used and what's the current situation and problem solving progress. At first problem day they used to reply godaddy-style short zero-valuable comments, that have nothing to do with problem essence and solution. They simply don't try (or unable) to understand problem deeply enough.
This message to forum has mainly two targets. Primary is to get problem solution from all of you, who may have experienced same problem as me, since i already lost my hope to get problem solution from solarvps. Second - to kick *** solarVPS a bit for poor service quality.
And here is the problem.
VPS system: windows OS, 384mb of ram (venusX package).
As soon as i start apache server, or IIS server, or whatever server that is listening to port 80, same moment TCP connection count (measured by windows 'performance' tool) rises to more than 300, usually 600. Then all http requests are rejected, website is completely down. It is notable, that problem is clearly not caused by heavy http server loads. Problem appears on even low load. Stopping http server causes connections to fall to normal level, nearly 0. Starting again suddenly causes problem again. Connection count jumps rapidly to >300.
This problem appears about 30% of all time. There are periods problem does not appear, server functions normal. No problem regularity found.
It seems that connections are stuck at OS or physical level, they are not even handled by apache. Apache logs show no or little activity when problem happens, CPU usage is very low. Only very small part of requests are passed to apache. Others are just gone, although large amount of connections are shown by system. Seems they are somehow 'hung' and stay too long not disconnected.
When trying to reach website through browser, firefox says 'connection interrupted' at same moment. This again shows that requests are not even processed by apache. In case of heavy load 'time out' would appear after several seconds only.
At early stages of problem i suspected two causes: apache server software problem or DoS attack. Both of these were denied. Apache was reinstalled (and IIS, lighthttpd 'causes' the same problem too). DoS as cause was denied too, as traffic does not seem suspicious.
At this moment i do not know what to suspect, but its almost clear it's solarvps problem, not mine. As long as they did not deny it and got full understanding of problem. I could suspect virus, making DoS in their network, router physical disorder, what else? Anyway their problem, not mine. But his becomes my problem, as i have no service for third day.
I do not blame solarVPS for problem. Problem is realy mystical and interesting, it may happen to anyone. I just say that their support reacting time and quality is pathetic in this situation. And i miss adequate work to find problem cause.
Anyway they have work around to solve problem - move my whole VPS account to another physical server. But they don't answer about this.
We’ve been with PowerVPS for approximately two years given the positive recommendations from WHT.
Few days ago, our site was hit by the MPack Malware and has affected ALL domains on the VPS as well as the cPanel/WHM core file. We have since secured the vulnerability and cleaned the important domains. Yesterday (July 13th) we requested a full VPS restore to be initiated from the July 11th nightly backup to ensure any other unknown changes are rolled back. We were quite disappointed to be told that there was no backup from the 11th available and that the latest would be the 9th. This came as a blow as they advertise nightly backups for their VPS Servers.
Here is their backup process from their support site: --- PowerVPS backs up your VPS nightly to a dedicated backup server with a raid5 or raid10 array. The nightly backups are incremental [1] for six days, and on the seventh day a weekly full backup [2] is performed. Two full backups are kept and the old ones rotated out automatically.
As many as fourteen backups are kept at any given time. The automated backup scheme allows for enough room for full recovery from the incrementals (which require the most recent full backup to exist), the other full backup, and still leaves room for customer initiated backups. ---
As a result, we have asked for the July 9th backup to be restored and it has been a full 18 hours now which they claim is to create a list of file differences and than the actual restore, which the later has yet to start. Although I do not know the exact process on their end, I cannot imagine how if the server was non-functional, that a restore that has so far taken 18 hours and counting, with no ETA, would be acceptable. Add on top, the lack of nightly backup from the night before.
If anyone could shed light on how a restore could take over 18 hours, please chime in. As for now, I am one very disappointed customer.
In your opinion how much downtime is too much downtime?
1, 2, 5 hours over the course of a month? 99.8%, 98%, 95% total uptime too little?
I mean I can understand technical difficulties and I am willing to be patient with my host especially when the service (when it is up) is good but where exactly do you draw the line, start asking questions, or canceling that contract and demanding your refund?
i have 150 domains my server I wish it to migrate to a new server I can do restore domains, mail and users and data, stats to new server. I need minimum downtime so I need your's advice step by step which service need to restore first so my user get minimum downtime.
i've recently been experiencing a lot of apache downtime on my eUKhost VPS. anyone else having this problem? it's driving me crazy and i'm thinking about moving.
It's not even been a month that I started subscribe a windows 2003 VPS hosting with them and now their website is gone and I am unable to access my VPS server
I am the webmaster for a high level court system. We run mainly .php and MySQL like most do. We are in the process of doing a transfer to 1 and 1. I want to be able to test everything (database wise) before I do the transfer. I thought the best way to do so would be to purchase another DNS name and server and transfer everything there first to test and then when that is fully operational have a redirect to there and do the DNS name transfer from the old site to 1 and 1, and when the DNS name is on 1 and 1 take the redirect off . I don’t want any downtime at all.
This might not be the most cost effective but for downtime I thought it would be the best way, especially when I can have a live test before the transfer.
In the forum here i put a story about when i was changed to vz45 from vz69 some months ago and the problem is growing. One month ago, around 1/oct, was a half day.
Yesterday i detect a problem at 08:40 am aprox, and they are working in that but something is AWFUL here.[url]
I have many domains and webservers. so it's hard to monitor everything usually. i heard there are some websites and softwares to do this.
does windows 2003 have anything default like this ? or can anyone suggest the application for my windows 2003 server? which sends alerts if any error is going on my server?
Also there any other websites which is doing this monitoring? because i have some shared accounts and i want to monitor it too.
I'm having a bit of a quabble with my current host because our server was unreachable for an extended period yesterday, and since I made a traceroute that arrived at their datacenter but not at our server, I was assuming it's a crystal clear network issue.
Today however they tell me that a traceroute means nothing. To quote them:
Quote:
Please note that having a traceroute end prematurely does not always indicate a problem with the network it ends at. Being that traffic is typically asynchronous, the return path back to you does not follow the same providers. Your traceroute can not fully take into account network connectivity issues on the return path nor will it show them. You would need a traceroute to and from the server at the time of the issue (while this may be impossible if the server is down, opening a ticket immediately and informing the technician you would like a traceroute to X.X.X.X IP would hopefully catch the return path issue).
What do I make of this? First time I've heard that, when the network is down, I need to contact my host to tell me. But without any polemic - is this what it boils down to?
It is probably acceptable for a site on shared hosting to have some downtime. However, how much is acceptable?
According to Pingdom one of my sites had a total downtime of 14 hrs for Feb. One time it was reportedly down for 4 hrs.
I had set it up to ping every 15 mts. Are these results trustworthy? And if it is should I move my host? Or at least tell them to move my site to another server?
The reason I ask is because I was wondering if this was the norm for shared hosting.
Probably been asked a million times but after spending 30 min searching since the limit is 130 seconds between searches... Anyway, i am VERY unhappy with ModVPS as my server keeps going on and off and on and off and they tell me its "fixed" but it keeps doing it to all of my staff even... anyway enough of that
I purchased a really nice dedicated server and need to know how to migrate all accounts over without having any downtime and allowing people to still use their website as normal. I have about 60 clients and would hate to see one of them upset because their data was lost during propagation when they connected to the old server instead of the new, exc.
Is the best method to setup a cluster through cpanel? Or is there a better way?
I have a server that used to be with Hostgator. There is a hundred accounts and the average load is 0.5. It has 2 gigs of RAM, a Dual-Core Pentium D and 2x Sata HDDs in Raid-1. At one point it started being very often (at least once a day) down. It would be all of the sudden down. I used to monitor it, with 0.2 load average, and all of the sudden, the connection is lost, I can't access the web sites, etc... Fed up, after several weeks pressuring over Hostgator, I cancelled the server, I contracted the same configuration at Softlayer and contracted TouchSupport to manage the new server. They migrated the hundred web sites to the new server... Exact same problem. Every single day, any time of the day the server is down. There's no specific time. And right before it's down, load average is below 1. From the logs, there's nothing unusual. Softlayer changed the CPU and RAM just to make sure. Same problem. It's been a month like this, and I just don't know what to do at this point. Moving to yet another company doesn't seem to be a solution I think. If it were a DoS, load average would be very high before going down, but it's always very low right before. Do you know what may be causing it?
I colo a 1U machine with 2-36gig drives. They're not in RAID, and I have it set to rsync backups to a remote machine on a regular schedule. I have another remote machine functioning as a secondary DNS. Neither of these 2 are on a large upstream pipe. I just bought 2-147gig drives that I'd like to replace the 36g's with. How does this sound for a scenario to accomplish this with little downtime (pre-pardon my noob'ish ways):
1. Do a complete rsync of the filesystem to my remote machine as well as sync the mysql db's (to 1 remote drive).
2. Pop that single rsync'd drive into an external enclosure.
3. Travel to datacenter, once there, plug external drive into laptop and start up a VM that boots off of that drive.
4. Sync again so external drive has the most up-to-date data.
5. Change over IP's from colo to VM on laptop.
6. Shutdown and swap out drives in colo'd box with the new ones.
7. Setup new drives as RAID 1, install OS, then rsync filesystem over from laptop to new drives in colo'd box.
8. Change back IP's.
What am I missing, or is there an easier way without a 2nd colo/dedicated server? Currently, the colo'd machine is using about 1.3Mbit/sec outbound and it's running a low load.
to migrate domains to another server with zero downtime. There is a panel (but its webminish) and I have full root access to everything so this will be a hands on job.
The thread I recalled seeing was about someone hosting shoutcast servers and they wanted to migrate and there was a way they did this without any downtime.
I can't remember the details but it had a way to redirect all traffic to the new machines. This would be great, as I also want to direct all traffic including mails (not just HTTP).
like to clarify this is not a post to bash liquidweb.
I have been a loyal customer of LW for the past 2 years (almost)
My websites have been down since Thursday 3:00 a.m due to "DDos Attack"
on Friday night, the network engineer told me the "DDos Attack" has stopped and problem should have been resolved. However, no websites were accessible.
On Saturday, I ensure you I have made many phone calls to Liquidweb dedicated server support and try to resolve this issue. (Never once did I raise my voice and they were being very polite.)
I believe the server admins that were helping was at the end of their shifts and every time I try to make a phone call to the person that was assisting me to get an update, liquidweb would tell me that he has gone home. Every time the server admin finishes his shift, the support ticket is not assigned to anyone else that is working until I make a phone call to them stating my problem.
I own quite a few large sites that generate over 3 millions hits per day and every hour of downtime means I will be losing valuable clients and revenue. That is why I am extremely frustrated with the 63 hours downtime as it seems like the results of working hard for the past 2 years is slipping away from me as the clock keeps ticking.
I am sorry, I am just trying to share this with fellow webmasters as I really need a place to express my feeling. (None of my friends will understand online business.)
Again, I am not bashing liquidweb...I must have really bad luck myself.
Ive looked for a long time now trying to find the perfect tutorial to migrate my clients (on cPanel) to my VPS without downtime. Every tutorial i found doesnt tell you how to make it so you dont have any downtime at all.
I know you can point the DNS from the old server to the new server by editing it through cpanel but i have more than 100 clients and it would be DAMN annoying to edit every single one of them one by one. Is there a faster way? Anyone have a tutorial that i can follow?
I am having a downtime issue with VPSlink, the "low-cost" VPS service from Spry, specifically they have let me out in the cold for over 24h and counting, due to a hardware problem in the node I am hosted for about 2 months now.
In all truth, the service has been great (read outstanding for low-cost) until this day.
It's Spry with another name after all, and they insist in putting that reference in each and every page footer.
I know hardware problems are neither easy nor fast to fix, and given that they refuse to give an ETA, I have patiently only asked for a situation update each 12h.
No answer to the update so far, only the initial reply to the support ticket and a thread post describing the situation which dates from yesterday (~24h ago).
The thing that really annoys me is that they are taking new orders to the very same plan, and those new orders are being placed on a new working node.. Couldn't they just move the affected costumers to the other node while they are fixing the affected one, even if just temporarily?
Everyone should know what to expect from subscribing a "low-cost" service, even if it comes from Spry. Despite being "low-cost", keep this in mind, you can't subscribe a "low-cost" service without incidentally subscribing a service of some kind.. Service that hasn't been provided for over 24h.
Don't really know what your feelings are on this one, but I for once remembered Spry to be better than this. You can't honestly expect to put everywhere "VPSLink, a subsidiary of Spry" and don't expect people to draw some parallels..
My expectations were just a very low end box, totally self managed, that simply worked in the areas that aren't my responsibility. To me, in the hosting world 24h downtime is just way to much time to go without giving an update, no matter how low each one standards may be.
There is a difference between asking for patience and leaving people in the cold. Over 24h downtime tell the difference.
I have several unmanaged Xen-based VPSes with a company that I've for the most part been very pleased with for the past couple years. I recently got an email from them saying that the physical server my VPS was on needed some components replaced and that the VPS would be "briefly unavailable" on a Saturday morning to do so.
Now, my definition of "briefly unavailable" is a few minutes, maybe 5 minutes max. I assumed that they would migrate the VPSes to a backup physical server while they fixed the other one, then migrate them back afterwards. In fact, I thought that Xen was capable of doing this in real-time with no downtime.
But the VPS was down for over an hour, so they obviously just shut down the box with the VPSes running. Because of the nature of my website, that particular Saturday was very inconvenient for me to have extended downtime. I was rather upset and contacted them to let them know, but they seemed rather indifferent and suggested that I look into upgrading to their redundant/failover services.
So my question is, am I being unreasonable to expect that *planned* downtime be limited to a couple minutes, or is it normal for companies to just take the servers down while they perform the maintenance? I thought that real-time migration to a different server was one of the benefits of virtual servers and I was surprised that they didn't take advantage of it.
Is Nick Laws here? Someone knows where to call him or managemybox.com?
I have 3 servers with them. One of them is down for almost 16 hours now, and it has been 4 hours without notice. Tech replied saying is a harddrive crash and they will replace it, I have lost a lot of dollars because all important clients on this server are running out.
Now, can someone help me contact Nick? I sent him a message a minutes ago, but his techs are nto replying and one of my clients is really in a bad mood saying he will take legal actions against me.
Anyone can say a word? This is really bad and I dont get even a reply, just a word last night saying a few hours.
This is the ticket: Dear Felipe,
ssh down too, will have a tech reboot and clear firewall, will take about 15 minutes
Thank you for choosing ManageMyServices, we appreciate your business!
Administrator: managedservices Ticket Department: Sysadmin Ticket Category: ALL Assigned Administrator: ALL Importance: Urgent Company Area: ManageMyBox-Dedicated Ticket Status: closed Reply By felweb - 26-06-2008 01:57 Any news from your end on this ticket, or at least on the other 2?
This is taking way too long and it is affecting this a lot ....
as an Internet marketer I have several servers, With several different companies and data centers. one of our main servers has been with FDC
We decided to choose them because they're pricing is fairly cheap shortly after getting services we experienced down times, almost on a daily basis ranging from 10 minutes to several hours. we debated leaving them many times but we had many files/websites/tools running on the server and really didn't have time to move every thing over to a different server.
Anyways today i wake up to find out our services were suspended for non payment. so i contact support about the issue asking what i owe and why i didn't receive notice before hand, apparently we didn't pay for 4 months. i spoke to them and informed them that we have not received any invoices and are bill is set to auto renew so this should not have happened.
i went ahead and asked if i could pay half of the invoice today and the rest in the a week, they said no i have to pay the full $1,075. Being with them for over a year i feel that that it was unreasonable for them not to help us out has it was there billing issue that cause this situation in the first place.
Over all even if i didn't experience this billing issue we were planing on move services elsewhere. anytime they do maintenance all servers go down and it happens very often.
Moral of the story is you get what you pay for stay FAR away from FDC
As the title reads, what is the best method if you using WHM to migrate accounts from one server to the other and experience the least amount of downtime?
I know WHM has it's function where you can migrate all the accounts simply. And from my view point, you should migrate the accounts, then update the DNS (nameservers) for the new server, and that would reduce the downtime?
