When in doubt, check the plug
Yet another sad but instructive tale from the annals of the IT world.
Yesterday, our network was shit.
Email was slow, half the corporation couldn't get to any web-based applications and those that did found them unbearably slow. People whose jobs depended primarily on being able to do work online (me, for example) found ourselves sitting on our hands. Of course the application I babysit for a living was hosed.
We were able to do some stuff in our development environment, but the production environment -- forget it. I spent a large amount of time replying to emails ("Yes, we are aware of the problem and working to resolve it as quickly as possible") from our user population since there wasn't much else I could do.
Many games of Solitaire were played yesterday, I suspect.
At around 6:30 PM, after being on a conference call for about 5 hours with the vender to whom we outsourced control of infrastructure, my counterpart who's responsible for making sure the machines we use are happy decided to go home and continue sitting around listening to the call from there. I left at that point as well.
This morning I met him in the parking lot around 8 when we both arrived. I asked if they'd finally fixed things. He nodded - about 10 PM.
The final verdict?
A loose cable in the data center out in California.
The unanswered questions: how the hell did it get loose, and why did it take them 15 hours to discover the same?
The loose cable that decided to outsource has yet to be discovered.
Yesterday, our network was shit.
Email was slow, half the corporation couldn't get to any web-based applications and those that did found them unbearably slow. People whose jobs depended primarily on being able to do work online (me, for example) found ourselves sitting on our hands. Of course the application I babysit for a living was hosed.
We were able to do some stuff in our development environment, but the production environment -- forget it. I spent a large amount of time replying to emails ("Yes, we are aware of the problem and working to resolve it as quickly as possible") from our user population since there wasn't much else I could do.
Many games of Solitaire were played yesterday, I suspect.
At around 6:30 PM, after being on a conference call for about 5 hours with the vender to whom we outsourced control of infrastructure, my counterpart who's responsible for making sure the machines we use are happy decided to go home and continue sitting around listening to the call from there. I left at that point as well.
This morning I met him in the parking lot around 8 when we both arrived. I asked if they'd finally fixed things. He nodded - about 10 PM.
The final verdict?
A loose cable in the data center out in California.
The unanswered questions: how the hell did it get loose, and why did it take them 15 hours to discover the same?
The loose cable that decided to outsource has yet to be discovered.
Comments
I love the denouement, such as it is.