US MySQL server MYSQL01 suffered from an outage earlier this morning. While we were able to restore the server itself, the MySQL service was failing to start. After a quick RAID verification the service was able to fully start and due to the crash MySQL had to perform a recovery on itself. The recovery completed and the MySQL service is now fully restored. No data appears to have been lost or corrupted, the recovery logs show all INNODB records were recovered and processed without any errors.
At this time we are performing a full RAID verification and repair to see if we can identify the disk that is failing and proceed to replace it. If we are unable to find the failing disk we will need to take down the server to check each drive’s SMART status in order to find the failing one.
Thank you for your continued patience during this time!
