Start with a couple blogs, some websites maintained as favors, and a very small, free-lance, website business. Enter one failed hosting company cutting off service with no notice. Allow 38 hours. Mix in one vacation.

That’s exactly what happened to us earlier this month.

One Tuesday evening, just by happenstance, Pam discovered a link from a blog to post on a discussion board mentioning that our hosting company was having problems. Apparently most servers were already down, nobody could contact the company, and it was assumed the rest of the servers would go down shortly.

We immediately scrambled and started grabbing backups of all the sites and domains we worked with. We managed to obtain backups of most of the sites. We already had fairly recent backups and re-creation materials for most of the sites, but with active blogs and actively changing sites, it’s always best to get an up-to-the-minute backup and always easiest to restore from a complete .tar.gz of the site. Plus, a couple of our clients maintain their own sites with our help and advice – we’re not necessarily the only ones changing things. More on backups later, along with some advice.

Anyhow, we’re at 4:00 PM Tuesday evening, and all Hell has broken loose! I have to be at work at 5:00 AM Wednesday, I have a commitment on Wednesday evening, and we leave for vacation first thing Thursday morning.

There is no time for panic and profanity. We’ve got to start restoring websites. There are some sites I can help with, but mostly, it’s Pam has to start restoring websites while I pack for vacation.

We started with our main site, umpqua-uppers.com. Normally, one would start with the clients’ sites, but we needed to get this site hosted to restore e-mail access to communicate with clients, domain registrars, and hosting companies. Pam quickly found a new hosting company – more expensive than the old one of course – and we set up the first account. The domain was registered through a major registrar, and we quickly re-pointed the domain. We had e-mail within a few minutes and started uploading the fresh backup. Less than 90 minutes from discovery of the problem, and with no more than fifteen minutes of site outage, we had the site restored. E-mail was back. We could communicate. Maybe this wasn’t going to be as bad as we thought it was.

While Pam was working on the domain, I was busy using Bloglines, Google, and other tools to copy-paste-save some blog entries that hadn’t been backed up. We’d have to do some re-creation, but it wouldn’t be bad. Nothing more than a few entries were lost, we thought, It’s a pain. It’s going to take time, but we can do this. The earthquake was starting to look like a speed bump.

Our relief was short lived.

Pam set up the next account at the new host. We had a backup. This was going to be quick. Pam went to re-point the DNS and …

We discovered, that, for convenience, we had used our now-defunct hosting company to register the domain for us. Instead of being registered to our client with us as a contact, the domain was registered to our now-defunct hosting company. We couldn’t get access to our domains.

Faxes, e-mails, and phone calls were exchanged. Time elapsed. The registrar wanted to allow 72 hours to try and contact the now-defunct host. Meanwhile, sites were down and we were planning to leave for vacation before then.

Finally, Pam consulted the thread about our ISP and found out how others were getting access to their domains. We gained access and continued the restoration process.

Meanwhile, we were also calling those clients that maintain their own sites, telling them to take backups (we were able to take backups of some of these sites) and telling them what we were doing to recover our site.

Very little sleep was had Tuesday night.

Wednesday, I suffered sleepily through the workday. Pam suffered with websites and the registrar. New hosting was established for all sites. Eventually, the registrar released the domains and the DNS’s were re-pointed at the new host. Clients were advised of our success with the registrar and informed how to get their domains back. Headaches and fatigue abounded.

By Wednesday evening, all the client domains were back up. Some still need a little work – mostly on the back end. We slept in Thursday evening, then left for a week of vacation. Other than re-creating one database, we did little website work while on vacation.

We’ve now been back from vacation for several days. We’ve had non-website adventures that have slowed the restoration of websites. All sites except this one are back. This one, as you can see, is partly back, and we’re taking this opportunity to make some changes under the hood.

Lessons Learned

1) Keep your own backups. Some of the clients of our now-defunct hosting company had paid that same company to make and keep backups. These businesses, some of them fairly serious on-line commerce organizations, lost all their data. Keep your own backups or pay somebody other than your hosting company to make and keep your backup. (There is also the whole idea of multiple backups stored in different locations that you may need to consider, but I’m not going to touch on it here.)

2) Don’t use your hosting company to register your domains. The domains we had registered with other companies were quickly and (relatively) easily re-pointed to the new host. Those that were registered through the now-defunct hosting company were much more difficult to transfer and for a while, we thought we may have lost these domains.

3) Think about your website business. You have a hosting site, a backup, a domain registrar. You. What happens if any piece of the structure fails. By separating the pieces, of the website, you are in better position to recover should any one piece fail. We had a real problem because there is still no way to contact anybody at our former ISP. The outfit has vanished.

4) Make frequent backups. We had backups, but in some cases they were several weeks old. This isn’t really adequate on actively changing sites. Also, make an extra backup any time you make major changes in a site. We had made major changes – in once case major structural changes splitting one blog into two – in two sites since our last backup. We relied on the next scheduled backup to save these changes. The site failed before this next scheduled backup. We have the materials to re-create these changes, but re-loading from a backup is faster and easier.

5) Don’t trust the future based on the past. We started with the now-defunct host in 2001. At that time, they were state of the art with the best price available. We were thrilled with the service and support. It was great to work with an ISP that actually gave us access to change many aspects of the domain. This host was great. We recommended the host to friends and hosted all our clients with them. Time passed. Quality declined, but they’ve always been good, so we decided to allow them time to fix things. We’ve had real quality warnings since the first of the year, but the host has always been so good and we figured they’d straighten things out. Instead, they vanished and left us in the lurch. In the future, we will be continuously re-evaluating our business partners based on current service, not past service.