Service Outage: File Server Space
8 03 2007At 6:00 Tuesday morning we began a routine process to add disk space to the Faculty and Staff storage area. While adding the space we encountered a bug within IBM’s software on the storage server. This bug caused the existing space to become unreadable. We worked for several hours with support personnel from both IBM’s support partner and Novell’s support team to recover the data.
Late in the day Tuesday it became clear we would not be able to recover the existing data and initiated the process to restore the data from backup tapes. The total amount of information being pulled from the tapes and the recreation of the virtual server structure went well but was very time consuming.
The Student space was returned to operation on Tuesday, March 6th at 8:00 pm. There was no data loss on the student volume.
Faculty Space returned to full operation on Wednesday, March 7th at 8:00am. All files updated on the faculty volume between 2:00AM and 6:00AM Tuesday, March 6th are not in the restore.
Staff Space returned to full operation on Thursday, March 8th at 8:00am. All files updated on the straff volume between 2:00AM and 6:00AM Tuesday, March 6th are not in the restore.
Any data saved or modified on the Faculty and Staff volumes between 2:00AM and 6:00AM Tuesday, March 6th has been lost.
We will continue working with IBM over the next weeks to completely isolate the portion of the software that created the issue, and we will be updating the software on our storage server with the patched version.
Thanks for your patience and understanding during this service outage.
Vince Di Scipio
Director Networks, Telecommunications and Systems






I’d like to express my appreciation to Senior Systems Administrator Roger Dills for the work that he did to overcome this problem and to bring these resources back online. His level of effort–and the dedication to service from which it arose–were truly extraordinary. Thanks, Roger.