Skip to main content

1and1 Has a Massive Server Failure

Every so often as I'm sure many web developers do I tried to pop into my site just to see how thing were doing. To my dismay I was unable to hit the site, I figured it was something I did wrong so I cleared internet cache, checked that I was connected and tried again. Nothing still happened. Passed a quick e-mail to my provider, 1&1, to let them know I had noticed and wanted to know what was going on. They sent me a response.
Thank you for contacting us. Due to some technical issues, the server where your websites are hosted is still offline at the moment. We sincerely apologize for this very unfortunate event, although unforeseen but which might already be causing you a lot of inconvenience. We also understand the frustration you might be feeling right now about this whole issue and how anxious you are to see your website(s) online again. Unfortunately, we still do not have an estimated time frame for a resolution however rest assured that our administrators are working as quickly as possible. We are also taking safe measures to ensure that an incident of this nature will not be able to happen again. Thank you so much for your patience and understanding as we work to resolve this issue. If you have any further questions please do not hesitate to contact us.
Fine, at least they know about it. I have had similar problems myself and I know how it must feel to have to explain they messed up. My frustration is better vented to my audience (surprisingly that's you), and it allows them to focus on resolving the problem. This morning when I checked again I saw that the site had still not been restored and so I posted a twitter and facebook update about it. Funny enough my site, all be it an older version, had been restored and was again available. About 4 hours later I thought to look again and saw the site was again not available. I thought I would give them the time needed to get the bad servers offline and update the new servers with the backup. When I got home I checked my e-mail and saw this from them.
On February 9, 2009, our technicians discovered a malfunction in the hardware that your site is hosted. We quickly resolved this malfunction with a replacement of hardware, moving your site to a new system. The new system itself, once live, started to fail with the RAID storage system resulting in our technicians setting up a brand new system to host your site. The setup of this system will take about 10 hours, with up to another 20 hours to restore the data from our backups. We are currently about 5 hours into this new system setup and restoration. The worst case scenario currently would be a maximum of another 25 hours downtime until all sites are online, however our current estimate is approximately 8 hours. When the system is back online we shall update you by e-mail with the current status. Please be assured that we have done everything within our capacity in handling this matter. Our technicians understand the priority of this, and have followed all procedures in place to ensure limited downtime for your site, which is our primary concern. We are investigating any improvements we can make to our processes for the future to ensure that an incident like this does not develop to a major outage. Sincerely, Your 1&1 Internet Team 1&1 Internet Inc.
Well that's just fan-damn-tastic. So I figured nothing else to do but write this blog about it. Depending on the state of my hosting space this blog might at times look funky because I kept the formating code or my CSS style sheet separate and stored on the hosting server of 1&1. If you are having a hard time reading this I'm sorry. I'm looking at getting some temporary space while this is being fixed. If you have had this happen to you let me know what you have done to marginalize the affects.
UPDATE: They officialy declared my site back online at 1:15 PM Feb. 11 '09
We are happy to announce that your website is back online. Our technicians have safely moved your entire site to a brand new system. All your data has been successfully migrated. We would like to apologize for the inconvenience caused by this prolonged downtime and would like to thank you for your patience while our technicians worked around the clock to get your site working again. If you or your users notice any issues, please make sure to contact us right away. Please call 1-866-99 1AND1 to speak to our friendly support agents or write to and we will handle your questions as quickly as possible. Please be assured that we will be investigating the cause of the hardware failure and take measures to ensure that this will not happen again. Thank you for your understanding in this matter and thank you for choosing 1&1 Internet as hosting provider. Sincerely, Your 1&1 Internet Team 1&1 Internet Inc.


Popular posts from this blog

Hack Trillian 3 Basic to Use Other Plugins

For some time I have been using the popular instant messaging client Trillian from Cerulean Studios. In the basic version it allows a person to connect to MSN, Yahoo, AOL / ICQ, and IRC. The four largest instant messaging communities all in one application. It saves my system memory space and removes all the crap I don't need that is installed by default on the standard IM clients. I have recently needed to add a fifth community when I had people connect to me from Google Talk. Their application was simple enough but now I had two IM clients instead of one. Fortunately Cerulean Studios has created the ability to add plug-ins to Trillian and many people have created some fun and interesting plug-ins. They are however for use with the Pro version and not the free Basic version. So I tried a few simple things to see if I could get the Jabber plug-in working with Trillian Basic. The end result was simple but it meant I lost one of my other IM connections. For me I choose the loose th

QR Codes

  So what are all these mixed up checker board looking things you're seeing everywhere? They are called QR codes, and they make your smart phone a whole lot smarter. Imagen you are walking through the mall and you see a poster of that new movie everyone is talking about. You want to discover more about it, but you're never going to remember the web address. Then you see the QR Code in the corner. You scan the code with your phone's camera and you are taken to the web page. All with a few taps of your fingers and none of it typing on a keyboard. Let's find out what else QR codes can do and what is needed to read them.   QR is the abbreviation for Quick Response, and was named so because the 2D bar code is meant to be decoded at high speed. It was developed in 1994 by Denso-Wave for automobile parts tracking. Since then the technology has evolved to allow much more. To use them you will have to have a camera phone and an app designed to read the QR codes. Google has pu

Awake from deep sleep

So it's been close to a year now since my last post and I have the need to blog again. Recently I have been adding myself to beta projects for Scrybe and Grand Central . Scrybe is an on-line calendar that allows you to quickly and easily view important dates of the year and drill down to the days events with a few simple clicks. use the link to check out a demo. The other site, Grand Central, is a kind of router for you phones. If you are like me you have more then one phone number for you, family, friends, and work to remember. You can give them all one number from Grand Central and that number will call all your numbers. You can even set it to screen calls or send telemarketers to a SPAM area. Again use the link to see more about it.