Transformational Education > New technologies
Stephen Laster, CIO, on system stability
Transcript
So my first day of work was early October of 2006. A beautiful day, new CIO walking in, rested, ready to go. And my finance director meets me at the front of Cotting House, and says, "Go home." And I said, "Why would I go home?" He said, "We've got an outage." I said, "Oh, I'm sorry that the building's out, but I'm sure we'll fix it." He said, "No, you don't understand. The whole campus is out." So instead of going home I called my wife and I said, "I'll see you in a few days." And that was the starting point. …
So our system instability was really caused by our growth, or our adoption of technology. And what had happened is that we had grown the use of technology so quickly that we hadn't kept tabs on how much electricity we were consuming in our data center, where all of our servers live, and, in laymen's terms, we ran out of electricity. And so when we ran out of electricity, the systems that check for fire thought there was a fire, and they shut down the server room. …
It was an interesting day. So the problems with a shutdown like that is we had never practiced restarting. And so now you've got hundred and hundreds of servers, and hundreds and hundreds of pieces of technology, and you have to figure out the right order in which to start them to get everything running again. Fortunately, we figured out the right order. Eighteen hours later we were back up and running.