Power Loss Interrupts IT Services

31 01 2008

At approximately 8:15 p.m. on Tuesday, January 29, the campus lost power due to a falling tree branch. Battery units provide power to our servers under these circumstances, but they can do so only for a short period of time. (Less than two hours.) The purpose of these battery units is to provide the Networks, Telecommunications and Systems (NTS) group the opportunity to shutdown our servers in an orderly way. Should the servers lose power during normal operation they can be damaged and/or data can be lost. NTS arrived on campus shortly after the power failure and were able to shutdown the great majority of servers before the battery units were drained. Power was restored to campus shortly after 11:00 p.m. At that point NTS began to restart services. With the exception of the Datatel system, all critical services were operational by 3:15 a.m. Datatel was available before the beginning of the next business day.

At about 8:30 on the morning of Wednesday, the 30th, NTS staff began working with our support vendor to research anomalies with the storage area network (SAN) that appeared following the power outage. The SAN provides data storage for a significant number of our critical servers. The SAN became unstable while running the routine maintenance and diagnostic routines the vendor recommended. It became necessary to shutdown the SAN shortly after noon. This interrupted those services that rely upon the SAN to house data. These include the Datatel system. The problems with the SAN were corrected at approximately 5:30 p.m. and all services had returned to normal operation two hours later.

We regret the inconveniences that these outages created for the campus community and appreciate your support and understanding. IT is very grateful to NTS members Vince Di Scipio, Roger Dills and John McCreight would worked into the wee hours of Tuesday morning and throughout the following day to restore services.


Actions

Informations

Leave a comment

You must be logged in to post a comment