When in doubt, check the plug

Yet another sad but instructive tale from the annals of the IT world.

Yesterday, our network was shit.

Email was slow, half the corporation couldn't get to any web-based applications and those that did found them unbearably slow. People whose jobs depended primarily on being able to do work online (me, for example) found ourselves sitting on our hands. Of course the application I babysit for a living was hosed.

We were able to do some stuff in our development environment, but the production environment -- forget it. I spent a large amount of time replying to emails ("Yes, we are aware of the problem and working to resolve it as quickly as possible") from our user population since there wasn't much else I could do.

Many games of Solitaire were played yesterday, I suspect.

At around 6:30 PM, after being on a conference call for about 5 hours with the vender to whom we outsourced control of infrastructure, my counterpart who's responsible for making sure the machines we use are happy decided to go home and continue sitting around listening to the call from there. I left at that point as well.

This morning I met him in the parking lot around 8 when we both arrived. I asked if they'd finally fixed things. He nodded - about 10 PM.

The final verdict?

A loose cable in the data center out in California.

The unanswered questions: how the hell did it get loose, and why did it take them 15 hours to discover the same?

The loose cable that decided to outsource has yet to be discovered.

Comments

Dan Brekke said…
My question is: How does this day, and the solitaire playing, go into the timekeeping software?

I love the denouement, such as it is.
Dr Ralph said…
For me, the day was written off as "Troubleshooting and consulting." Solitaire falls under the category "unscheduled tasks." One can always find a category if willing to think creatively.

Popular posts from this blog

Contact Fedex Delivery Dept For Your Package....

Random Thoughts on AI-generated images

Cloud on the Cheap