Today, during the rollout of database schema changes in connection with the forthcoming launch of our new web based file management interface, a SQL statement caused a database table to be dropped incorrectly. This caused our web based interface to become unavailable.
The incorrect change was replicated to our failover database server within seconds. Consequently, to recover the database table, it was necessary to restore the latest database backup which was taken at 7am this morning. Replaying the transaction log files for the period between the completion of the backup and the execution of erroneous statement took several hours.
This process was completed and full service was restored at 6.50pm UK time.
We stress that no data stored on humyo.com has been lost, changed or affected in any way. The affected database held meta data relating to our web based file management interface only, all of which has been restored in its entirety. Throughout the outage, users of our desktop software continued to have full access to their data.
We sincerely apologise for the inconvenience this first service outage in our two year history has caused our users. Whilst we are pleased that our backup procedures ran smoothly, we are naturally looking at ways we can reduce the likelihood of a reoccurrence and increase the speed of database restorations in the future.
