OSB:20130928-01

OPERATIONAL STATUS BULLETIN: 20130928-01

Issued: Kradorex Xeron (talk) 03:14, 28 September 2013 (EDT)

In Regards To: Core Router Failure

Facility: Unicomplex One (Hamilton, ON, Canada)

Affected: *.digibase.ca (all systems, all services), especially cplexus.unimatrix01.digibase.ca

Ticket #: CT-0000074

Expected Duration: Unknown

Status: Event started at 22:40 on 25 September 2013, ended at 07:30 on 25 September 2013, failovers were completed in between. Outage of equipment was estimated 8.5 hours. Service outage was estimated 1.5 hours.

Situation Description

Starting 22:40 on 25 September 2013, our central plexus router experienced a hardware failure, this failure was impacting to the non-volatile storage of the system where the operating system and configuration are stored.

Impact

This incident caused our network to transiently become unavailable to the public.

Updates

23:30, 25 September 2013

Manual failover to a network switch capable of rudimentary routing was completed. Services temporarily operational.

04:30, 25 September 2013

Plexus core router was put in place again.

05:00, 25 September 2013

Restoration was completed approximately 2013 09 26 05:00.

07:30, 25 September 2013

Services operational. Traffic verified flowing.

48 hour monitoring commences to end 2013 09 28 07:30