Re: Never push the Big Red Button (New York City subway failure)

[Chris Kane <ccie14430 () gmail com>]

True EPO story; maintenance crew carrying new drywall into the data center
backed into the EPO that didn't have a cover on it. One of the most
eerie sounds in networking...a completely silent data center.

-chris

On Fri, Sep 10, 2021 at 2:48 PM Christopher Morrow <morrowc.lists () gmail com>
wrote:

> On Fri, Sep 10, 2021 at 1:49 PM Matthew Huff <mhuff () ox com> wrote:
>
> > Reminds me of something that happened about 25 years ago when an
> > elementary school visited our data center of the insurance company where I
> > worked. One of our operators strategically positioned himself between the
> > kids and the mainframe, leaned back and hit it's EPO button.
> Or when your building engineering team cuts themselves a new key for the
> 'main breaker' for the facility... and tests it at 2pm on a tuesday.
> Or when that same team cuts a second key (gotta have 2 keys!) and tests
> that key on the same 'main breaker' ... at 2pm on the following tuesday.
>
> <quadruple face palm>
>
> not fakenews, a real story from a large building full of gov't employees
> and computers and all manner of 'critical infrastructure' for the agency
> occupying said building.
>
> Matthew Huff | Director of Technical Operations | OTA Management LLC
> > Office: 914-460-4039
> > mhuff () ox com | www.ox.com
> >
> > ...........................................................................................................................................
> >
> > -----Original Message-----
> > From: NANOG <nanog-bounces+mhuff=ox.com () nanog org> On Behalf Of Sean
> > Donelan
> > Sent: Friday, September 10, 2021 12:38 PM
> > To: nanog () nanog org
> > Subject: Never push the Big Red Button (New York City subway failure)
> >
> > NEW YORK CITY TRANSIT RAIL CONTROL CENTER POWER
> > OUTAGE ISSUE ON AUGUST 29, 2021
> > Key Findings
> > September 8, 2021
> >
> >
> >
> > https://www.governor.ny.gov/sites/default/files/2021-09/WSP_Key_Findings_Summary-for_release.pdf
> >
> > Key Findings
> > [...]
> >
> > 3. Based on the electrical equipment log readings and the manufacturer’s
> > official assessment, it was determined that the most likely cause of RCC
> > shutdown was the “Emergency Power Off” button being manually activated.
> >
> > Secondary Findings
> >
> > 1. The “Emergency Power Off” button did not have a protective cover at
> > the
> > time of the shutdown or the following WSP investigation.
> >
> > [...]
> > Mitigation Steps
> >
> > 1. Set up the electrical equipment Control and Communication systems
> > properly to stay active so that personnel can monitor RCC electrical
> > system operations.
> >
> > [...]

-- 
Chris Kane