News: Reason for Outage at USA, FL Datacenter - February 20, 2014

Published: 22/02/2014

Dear Customers/Partners,

This response is in regards to the outage that affected the MangoXchange, USA, Jacksonville facility on February 20, 2014.

MangoXchange experienced a power outage at 3:39PM EST on February 20, 2014 due to an upstream main breaker failing within our colocation facility (GoRACK). This was the main breaker to a power distribution panel that fed roughly 20% of the datacenter itself. The onsite facility engineers quickly identified the issue and worked to replace the breaker and fully restored power service within 30 minutes. After power was restored, there were a large number of individual customer servers that had to be powered on and individually checked for possible issues and hardware failures.

This outage affected roughly 600 of our total server footprint within the facility. This was not a total outage of all of our services.

During the outage, overall network connectivity was affected for approximately 8 minutes due to a considerable amount of upstream capacity dropping and routes having to reconverge to other paths.

We had full staff onsite at the time of the outage and everyone worked as quickly as possible to power machines on and return services to normal. Our internal systems were affected by this outage and caused our website and client portal to be unavailable for a period of time. We began restoring customers prior to restoring access to our internal systems.

The bulk majority of services were restored within 45 minutes of the outage occurring, however, there were some individual isolated issues where machines required extended FSCKs or hardware replacements. We worked as quickly as possible through the ticket queue to correct these individual problems with the affected customers.

Going forward we will be scheduling network maintenance windows to perform various upgrades to redundancies to ensure all switching and routing equipment is dual fed from redundant A & B power feeds. Additionally, we will be establishing a proper Disaster Recovery procedure for internal systems to allow for failover in the event of an outage to allow for quicker client communication.

We greatly apologize for not providing more information and details during the outage yesterday afternoon. We will be implementing new outage reporting procedures to ensure information is relayed as quickly as possible to those affected.

Regards,
Don
MangoXchange Support Department