Disaster Recovery Plan And Backing Up One's Server
May 13, 2008
There is so much information on disaster recovery and backing up one's server, that I'm getting glassy-eyed trying to take it in. Maybe if I became an actual case study, and get some "group think" help, this thread could benefit many others in a similar situation.
Current Situation:
1. I'm a small hosting company, 5 years in existence, with about 350 clients. www.mlhi.net
2. Dedicated Linux server, PLESK CP w/unlimited domains license, fully managed at HostNexus (great guys). It does not have a RAID array (used to have that at Rackspace) but it does have a backup drive that everything is backed up to with a cron job every night.
3) In addition I have a Linux Sys Admin on retainer, www.linuxbox.co.uk (he is better than excellent). Two years of excellent server maintenance and security on top of the managed service I get at HostNexus.
4) I just bought a VPS plan at JauguarPC.com after much research (a lot of it here at WHT) and as they say "so far so good" with the ease of dealing with them. I have not setup anything there yet- just got the VPS provisioned a few days ago.
Fears and Concerns:
1. Data center destroyed/ my server burns up (including backup drive) etc etc.
2. DDOS attack (which did hit this data center a few months ago and I was down for hours)
3. If I had to FTP everything back to another server from my local, at 18 GB, it's not too cool.
Want to do this:
1. I want my Sys Admin to run a backup copy (and incrementals every night) to an identically configured VPS server at JaguarPC. Both servers are now running identical PLESK 8.4.
2. I want the fastest recovery possible without spending a ton of money. I know this means I don't get an "instant" recovery, but recovery within 24 hours is more than OK. None of my customers are ecommerce... just brochureware sites.
My "I'm not an expert" plan:
1. If primary server goes bye-bye forever, I can login to my BulkRegister/Enom account and change the child nameserver IPs to the IP's of the VPS. In 24 hours or less, every request for the nameservers would then be routed to the new server.
2. I can create an A record on every domain like www2.johndoeinsurance.com that would point to the IP at the VPS, so I can ease my mind anytime I want to make sure everything is safe and sound on the second server, and ready to go in an emergency.
How do I configure the DNS?
I control dns at Enom for about two-thirds of my customers. I have ALL domains pointed to ns.mlhi.net and ns2.mlhi.net. Here are my options??
1. I create two more child nameservers... ns3 and ns4 and have then pointed to the IPs at the new server, then update all the domains I control. The rest of the customers I can email and ask them to add the additional nameservers. I know... good luck on them doing it.
2. I change the ns2 IP to go to the new server. And I make sure when I make edits on a website during the day that I FTP to both servers.
3. I don't have any nameservers assigned to the new server. I just change the IP on the existing nameservers in the event of an emergency.
I'm trying to find a good hosting provider to host our company's website as our fallback option in case of disaster. One or two dedicated servers should do it, but it'll need the space/bandwidth to host a database of around 60-80 gigs, with the ability to rsync newer copies of the database on a regular basis. We also need to store a Tomcat website, which will take up much less space, but also need to be rsynced to be kept up to date on a regular basis, though less often than
Also, we probably need Red Hat Linux specifically, as opposed to other flavors of Linux.
Of course we need root access to install the other apps we'll need. My paramount concern is the security of our companies data, much of which not only has to be protected for our companies sake, but also for laws such as HIPAA, etc. Cost is a consideration, but security, dependability, and flexibility (root access to our machine, ability to rsync between sites) is more important.
I was wondering if anyone's got suggestions for me, hosting providers they've liked for these purposes? I'm looking at Media Temple's dpv Nitro option right now ...
I am not familiar with quick backup methods, but let me give you an example which might help you provide me with the best possible information.
1. I currently run a hosting business.
2. cPanel accounts are backedup nightly.
Question: I am worried something may happen to the server (DoS Attack) or just an internal error - what would be the quickest way to have your entire webserver back online?
I was thinking another HD with the same installation/configuration.
What procedures have other hosting organizations taken to prevent there web server from loosing downtime.
I'm putting together a disaster recovery plan and I'm trying to come up with a quick way to get new machines in place in case of catastrophic hardware failure, e.g. two servers go down at the same time (this is an HA setup and must always remain fault tolerant). At this point I'm trying to find someone who would rent a server to me, allowing me to take physical possession of it.
Why do I need physical possession of it? The problem is that I use IP-based storage on a private network and therefore can't rely on a rented server in another cabinet, even across the aisle in the same datacenter, without adding significant complexity to my setup.
So my question, in summary, is this: What would be the fastest, cheapest and simplest way to get a new server in my cabinet in a few short hours?
My servers are going to be in Colo4Dallas, so if there's a provider there who specializes in this or has made an arrangement with you or someone you know in the past, please let me know about them. If you can think of a better way to do things (the obvious solution of investing in more servers excluded),
I'm used to operating dedicated servers. This year I want to venture into getting a reseller account and / or media temple account.
With my dedicated servers, I do automated mysql dumps and I do nightly rsyncs of all my /home /etc and other important data so disaster recovery is possible.
My question is, how do you do the same thing? Sleep well at night? with these regular web hosting accounts?
I see most newbie people who pay for web hosting do not bother to backup their data. Ignorance is bliss until disaster strikes.
How do you automatically backup your data (mysql and directories) in your multiple web hosting accounts like reseller or grid service media temple?
I'm working on a plan for restoring my whole Plesk server fast in a worst case scenario. Which will never happen hopefully.
The backup is a full backup done by the Backup Manager in Plesk. It lies on a remote FTP server.
Right now I'm fetching the whole backup directory with wget while in the background the Plesk Installation is running. These actions take about 1h to complete.
Afterwards I would restore with /usr/local/psa/bin/pleskrestore.
Is there a better way? For example installing Plesk directly out of a full backup?
In light of whats happened at the Planet in Houston we are moving up our plans to have an external DNS nameserver (with failover) and emergency website to alert customers in the event of our data center (C4D in Dallas) ever goes down.
From all the forum messages reagrds this event its apparent that a lot of hosting companies are suffering the worst from having their support websites at the DC where they host their customers and have no way of notifying their customers of whats going on.
Looking to trade 1U of rack space and 1 ip address anywhere in US (Dallas preferred) with someone else who is looking for the same thing.
While we are extremely happy with C4D and have faith in their DC, you can never be 100% prepared for things beyond your control.
I like to create some service plans using the cli-tools, /usr/local/psa/bin/service_plan.I am able to create a service plan, but I'm unable to create a service plan inside a reseller plan. For example I cannot "tell" the service_plan script to add the created serviceplan to a reseller plan. Is it possible to create a serviceplan inside a reseller plan, using the cli?
My host just recently sent the hard drive with my sites to a data recovery company called Gillware. Website is [url]- but they failed and gave the following reason:
Quote:
Originally Posted by Gillware
Unfortunately, your file system was so severely damaged that no data can be recovered. We will make arrangements to return your drive via UPS. Sorry we could not help you further.
Gillware Inc.
Do you guys think there's still hope?
The hard drive is now being shipped to a more well known company, Drive Savers - [url]and I'm guessing that this is the last hope, because the more the drive gets tampered with, the more chance of permanent data loss.
So yeah.. I was just wondering what you think? If the file system is so severely damaged, do you think it STILL can be recovered?
We have a dedicated server with CPanel/WHM. It appears that the CPanel/WHM backup portion of the control panel is not functioning properly, as we just had an incident where all mySQL databases were deleted from the system and the last backup we had was from 03/31/08. This is not acceptable, as we need to have atleast at the min. weekly backups done of our system.
We are in need of a FREE 3rd Party (preferrably open source) solution where we can install on our Linux CentOS system to be able to have backups, and would like to have this software preferrably be able to backup to a remote location, but this is not required.
one of the worst things (in hosting) has happened. I received a notice this morning from lfd (configserver) that someone had logged into my server as root and it wasn’t me.
Unfortunately I didn’t notice it until eight hours later so I have no idea (yet) what happened during that period. Thankfully I don’t have any really critical data on the system that could have been stolen.
I’m in the process of restoring from a full system backup right now. After that’s done I’m going to look to see what the differences are between the files from the backup and that on the comprised drive. I’m not sure if I’ll get anything useful from the diff but hopefully I’ll find a clue as to how they got root access.
Then of course I need to get my server back up. However, I don’t want to do this until I’ve taken some steps to identify how the individual got in and take some additional preventative steps.
Here’s what I am planning on doing:
1) Check to make sure all exposed services are patched and look at some security sites to see if there are any known vulnerabilities for these services. Anyone know which sites are good to look at?
2) Change firewall to only allow ssh access from a couple specific IP addresses.
3) Disable root ssh access so I have to login via a different account and perform sudos, etc.
4)?
I’ll also look for a good server-hardening guide to see if there are some obvious things I forgot to secure.
Do any of you find folks have any other suggestions or resources that I should check out?
I'd like to back one of our servers up using S3 but I'm not sure the best way to go. Server is using CPanel. Obviously I want to keep costs down.
I need the files to be secure, backups need to be nightly incremental, and I need to be able to restore to another CPanel server fast in case of hardware failure. CPanel supports FTP for backups. I currently back up to another server, but I want to move to the cloud.
Specific questions I have:
1. Any recommendations on a cloud backup service (S3, Rackspace, etc?), and why?
2. Any recommendations on tools? I want this to be simple to set up. As easy as setting up an FTP account.
3. Is there another option I should consider? That server has about 62GB of data on it to be backed up.
we are looking for a suggestion on a best practice to help a customer recover from being hacked. One thought is that the client should be put on a new server and allowed to rebuild from there. The concern is that there are so many web app breaches that the server often gets rehacked because the bad code is still in the web app.
So, there is my example. I have two servers in different locations. First one is the main server with all the data and the second one has just simple notice like "We will be back soon".
Now, if I set the primary DNS server on main server and secondary DNS on second one (but with different A records), will users actually get the server two if main server is down?
Will this work or not? I want 100% access to the site, at least to the server with only notice (I don't need data from main server). I can set the round robin, but I don't need load balancing (actually I want access only main server), but just if main server is down to go on the second one.
Since they had disaster with their billing system it seems their support is falling apart also. I submitted a support ticket 1 month ago and still no reply!?
I am doing migration of plesk 11 from one server to another server on both server version is same but after migration I cant find any service plan which is there on old server.how can i copy / migrate service plan from one serer to another server
I heard from some technicians that SAS disk recovery is going to be much more difficult than the current generation of hard disk. Is it true? Anyone tried recovering from a spoilt SAS hard disk before?
After I create a backup of a VPS using Virtuozzo (I'm using the 1and1 VPS), where is the image stored? Is it necessary to manually save this image to a different server, or do 1and1 take of that automatically?
who you have used in the past to recover damanged hard drives that have been dropped. I got a client that damanged his HD, and it needs to be sent out to someone who is reliable is not gonna cheat and steal the data.
if someone have for mistake cancelled a cpanel account and in the same dir was the backup file of this account, should be possible to recovery at least the backup file from the deleted files? Or the entire dir cancelled?
I am trying to recover some backups on the server by rsync the home dir from BQbackup to my server. I deleted 1 file to see whether it actually works. I executing this command on the server:
I saw that it downloaded the file I deleted, however after checking on FTP if the file was recovered it actually wasn't. Is there something wrong in the command I used as it seems that it is not recovering any files.
I understand that servers can do automatic backups of information, yet I also see forum modifications that enable simple ways of doing a backup. Are there different types of backups? Why is it necessary to manually backup a forum database when its done automatically by the server? In terms of assuring the data, what is required and whats a typical procedure, what does it entail, is it manual and if so usually how often, or is it usually automatic?
I currently have 2 VPS accounts. The first one runs cPanel and hosts my website's files.
The second VPS hosts the mysql databases which the website uses. There are about 70 databases, each averaging about 120MB in size.
At current, i use sourceforge.net/projects/automysqlbackup/ to backup each database over two days. (This crashes the server briefly for about 10 minutes per day ). I then use rsync on both servers to backup everything to off-site location.
Obviously, I need to find a better solution. I was thinking about backing up the databases weekly instead. However, the 10 minute downtime period from the script will still happen.
