Author Topic: To Redundant, Or Not To Redundant...  (Read 2541 times)

Offline chpalmer

  • "Loose Screw"
  • Administrator Nut
  • Certifiable Nut
  • *****
  • Posts: 2428
  • Gender: Male
To Redundant, Or Not To Redundant...
« on: January 27, 2012, 03:22:03 am »

I was in the middle of a complicated order online for my business the middle of last year when suddenly my connection to the site I was on went dark. Turns out that a truck had turned to sharp and pulled 1800 feet of pole mounted fiber with it before stopping. Needless to say they weren't going to fix that real quick.

Before I knew what was going on I began to make a serious of ping tests to see how limited my new network was. I could reach several sites that I know exist on the NOANET network whom my ISP gets their connections from, but not any further. Luckily I could still reach my office over the VPN as the ISP that I use there also uses NOANET.

What really surprised me that day was how many were affected by this fiber string 6 miles away from my home. 911 phone systems at centers across the sound along the I5 corridor  had gone quiet. And three states worth of customers of my large cable ISP were without...

But also surprising was the fact that the VOIP phones at the office which I could reach were still live. If they had any down time at all I never saw it... Remoting into the desktop at my station there I was able to reach the rest of the world.  Very weird, I remember thinking... They must have a redundant route...  I later found out they did. By some quick changes to my firewall I was routing over the VPN and through my office gateway to and out to the internet. It was slow, but I got the order finished on time to get it shipped that day.

Another half hour of the outage I could of had email coming in also. But the fiber had been restrung before I got that far and I wasn't to worried.

What I later learned is that my office ISP, a small local hole in the wall establishment had not only an OC3 into NOANET's fiber (Actually NOANET manages PUD fiber in the state) but that they had redundant routes into the Qwest network. And while this event overloaded that redundant line, they still had connectivity to the rest of the world when my large multistate cable ISP fell down. To their credit the cable company sent fiber techs and bucket trucks to assist in the rebuild of that line... and they made sure that fact was mailed to all their commercial customers via a postcard about the event... But have we become that complacent? To run mission critical connections over single vulnerable routes?  Then when they fail, we pat ourselves on the back for being Johnny on the spot?  Its really the 911 services that went down that I worry about more than anything.