Yesterday we had one of our worst case business continuity issues – the good news, we recovered everything with NO data loss, the bad news, Tactile CRM was down for a few hours.

Our services are run on a load balanced web farm and backed up daily off-site and every 10 minutes on-site. Yesterday we had an issue with our main database and backups which meant that if we had had to restore from backups about 6 hours worth of users data would have been lost. Fortunately due to the excellent work of our team and hosting provider we worked through our backup and continuity process perfectly and nothing was lost. Here’s what happened:

1. Our database server hung and on reboot said there were no drives in the RAID array present.
2. We immediately brought up our backup server and started restoring the database
3. We drove the main database server direct to the supplier for diagnostics (we thought it was suspicious that all the drives had failed – we were correct)
4. Whilst the database was being restore we got the good news we wanted – it was the backpane in the server and in 10 minutes a new one was installed, tested and verified.
5. The database server was driven back to the data centre
6. The database server was installed again and everything started working

If we hadn’t taken the database off-site for diagnostics and restored from backups we’d have had the site down for about an hour less time but lost half a days worth of data. In the situation we thought the action we took was best for everyone, we hope you agree.

Here’s what we doing now to make sure this doesn’t happen again:

1. Monitoring the database server more closely for the next week
2. Moving forward our plans to implement realtime failover/replication of the database
3. Building another database server to add to our cluster with less of a lag between backups

If you have any questions at all about what happened, please drop us an email to support [at] tactile crm .com.

Photo Credit: TheAlieness GiselaGiardino²³’s