On Tuesday, Feb. 26, 2008, much of Florida suffered a power outage. I'm calling it the Flagami outage since that's the name of the FPL substation where the outage started. Ironically, this substation is right next door to the FPL Headquarters at 92nd Ave and Flagler Street in Miami. At 13:08 there was a small fire on insulator at this substation. At 13:09 the power went off for 3 million customers in the State of Florida. While the fire was the trigger for the outage, it's still not clear why this caused so many generators to go off line. Also important to note, not all of Florida or even all of South Florida was impacted. In many areas, there was either no outage or only a small glitch.
I was driving on SR836 when I heard WQAM go off the air for a few seconds. At about the same time I got multiple emails on my Blackberry from UPSs around Miami Dade County. I knew there was a major power hit. Since the UPSs are backed up by generator, I got the power restored messages a few seconds later. After a call from another telecom professional, I found out that the power was out for many areas. Since I was on a highway during the day, there was no way to see that the power was out. I checked our local EAS (Emergency Alert System) Primary radio news station, WIOD, for local news updates and instead found Rush Limbaugh.
As a part of my job, I proceeded to the Miami Dade 911 Center. For the next few hours I tracked the progress of the outage, the recovery and the impact on our telecom systems. As FPL generators came back on-line, power was restored in different areas. By about 17:30, almost all areas were back up. The Miami Dade Emergency Operations Center was activated to a Level 2 (out of 3, 1 being normal operations). There were hundreds of additional calls for Fire Rescue service in Miami Dade due to mostly people stuck in elevators and false alarms from Fire Alarms that had lost power.
Lucky for us, the length of the outage was short enough to avoid failures of Remote Terminals. We only had one location fail due to the power outage. This email documents the time and duration of the T1 outage:
From a few previous outages, I knew that this Remote Terminal was located in the closet of a private aviation company. It had been out for many days after Hurricane Wilma. This same RT serves the FAA Tower at Opa Locka airport and the USCG base.
Since we know the power went off at 13:09, the batteries at this RT lasted 3 hours and 5 minutes. The outage lasted 32 minutes. I'm assuming the outage ended when FPL restored power to the Airport. Out of the 80+ locations I'm responsible for, this is the only one that had a telecom outage.
It is my opinion that we would have had many more such telecom outages if the power outage had lasted more that 4 hours. Four hours is the design goal, and now FCC requirement, for Remote Terminal run time during a power outage. This power outage could be a good drill to see the best possible scenario for RT Generator deployment. It was the middle of a typical business day so I expect ATT was fully staffed and few techs had to be called in to work. The weather was clear except for one small part of Broward County. Since this outage was not storm related, there were no unsafe areas or areas that had to be cleared by the power company for access. No curfews were invoked. Clearly, there was no better time for an outage. Other power outages would have been far more difficult to respond to. While many parts of ATT's service area were impacted by this power outage, not all of it was. If the cascading effect had continued to more generators, the outage could have been 100% in South Florida or even all of Florida.
Information to evaluate response:
Questions I hope someone will ask ATT (former BellSouth):
In my opinion, we got lucky. If this outage had been more like the FPL outages of the 1970s and 80s, or Hurricane Wilma, or the recent outages in the Northeast, we would have had significant telecom outages. Even another hour of power outage could have resulted in tens of thousands of people being without any telephone or datacom service, including cellular service or even calling 911 from a landline. Hopefully ATT and their regulators (FCC and the Florida Public Service Commission) will use this event as a learning experience and come up with some realistic projections based on the data gathered. Clearly we have been reminded that it is very easy to have a huge part of South Florida without power, even during a perfect day. Only because a limited number of generators were effected was FPL able to get the area back on in less than the magic 4 hour period. Any longer and this would have been far more devastating to telecommunications. I feel that after seeing the projections regulators will seriously reconsider the 4 hour backup requirement and increase it to a realistic time period to get all RTs in a large geographic area on generator power before the batteries are exhausted.