1and1 Has a Massive Server Failure

Every so often as I'm sure many web developers do I tried to pop into my site just to see how thing were doing. To my dismay I was unable to hit the site, I figured it was something I did wrong so I cleared internet cache, checked that I was connected and tried again. Nothing still happened. Passed a quick e-mail to my provider, 1&1, to let them know I had noticed and wanted to know what was going on. They sent me a response.
Thank you for contacting us. Due to some technical issues, the server where your websites are hosted is still offline at the moment. We sincerely apologize for this very unfortunate event, although unforeseen but which might already be causing you a lot of inconvenience. We also understand the frustration you might be feeling right now about this whole issue and how anxious you are to see your website(s) online again. Unfortunately, we still do not have an estimated time frame for a resolution however rest assured that our administrators are working as quickly as possible. We are also taking safe measures to ensure that an incident of this nature will not be able to happen again. Thank you so much for your patience and understanding as we work to resolve this issue. If you have any further questions please do not hesitate to contact us.
Fine, at least they know about it. I have had similar problems myself and I know how it must feel to have to explain they messed up. My frustration is better vented to my audience (surprisingly that's you), and it allows them to focus on resolving the problem. This morning when I checked again I saw that the site had still not been restored and so I posted a twitter and facebook update about it. Funny enough my site, all be it an older version, had been restored and was again available. About 4 hours later I thought to look again and saw the site was again not available. I thought I would give them the time needed to get the bad servers offline and update the new servers with the backup. When I got home I checked my e-mail and saw this from them.
On February 9, 2009, our technicians discovered a malfunction in the hardware that your site is hosted. We quickly resolved this malfunction with a replacement of hardware, moving your site to a new system. The new system itself, once live, started to fail with the RAID storage system resulting in our technicians setting up a brand new system to host your site. The setup of this system will take about 10 hours, with up to another 20 hours to restore the data from our backups. We are currently about 5 hours into this new system setup and restoration. The worst case scenario currently would be a maximum of another 25 hours downtime until all sites are online, however our current estimate is approximately 8 hours. When the system is back online we shall update you by e-mail with the current status. Please be assured that we have done everything within our capacity in handling this matter. Our technicians understand the priority of this, and have followed all procedures in place to ensure limited downtime for your site, which is our primary concern. We are investigating any improvements we can make to our processes for the future to ensure that an incident like this does not develop to a major outage. Sincerely, Your 1&1 Internet Team 1&1 Internet Inc.
Well that's just fan-damn-tastic. So I figured nothing else to do but write this blog about it. Depending on the state of my hosting space this blog might at times look funky because I kept the formating code or my CSS style sheet separate and stored on the hosting server of 1&1. If you are having a hard time reading this I'm sorry. I'm looking at getting some temporary space while this is being fixed. If you have had this happen to you let me know what you have done to marginalize the affects.
UPDATE: They officialy declared my site back online at 1:15 PM Feb. 11 '09
We are happy to announce that your website is back online. Our technicians have safely moved your entire site to a brand new system. All your data has been successfully migrated. We would like to apologize for the inconvenience caused by this prolonged downtime and would like to thank you for your patience while our technicians worked around the clock to get your site working again. If you or your users notice any issues, please make sure to contact us right away. Please call 1-866-99 1AND1 to speak to our friendly support agents or write to support@1and1.com and we will handle your questions as quickly as possible. Please be assured that we will be investigating the cause of the hardware failure and take measures to ensure that this will not happen again. Thank you for your understanding in this matter and thank you for choosing 1&1 Internet as hosting provider. Sincerely, Your 1&1 Internet Team 1&1 Internet Inc. http://1and1.com