1-866-349-0689

Real-Time System Status

Operational Transparency: Detailed Event Log

Detailed Event Log

04/02

Impact

Severity

Non-impacting Core Maintenance
Status : Closed

Posted at: 04/02/2014 05:59 am
The maintenance is now complete.

Posted at: 04/02/2014 05:43 am
The maintenance is continuing.

Posted at: 04/02/2014 05:30 am
The maintenance is continuing.

Posted at: 04/02/2014 05:04 am
We are starting the maintenance now.

Started 05:00 AM Apr 02, 2014
Ended 06:00 AM Apr 02, 2014

03/29

Impact

Severity

Network impacting event - Chi-2
Status : Closed

Posted at: 03/29/2014 04:39 pm
At this time, all network operations do appear to be functioning properly at CHI-2. We can say, now, that this was not caused by a capacity issue or hardware failure. Our network operations technicians are still looking for the root cause, and to determine the true scope of the issue.

Thank you for your continued patience.

Posted at: 03/29/2014 04:11 pm
Our network operations technicians are currently looking into an issue regarding network functionality at our CHI-2 facility. The investigation of the cause is ongoing, and we will be updating this post as soon as we have more information.

Started 03:56 PM Mar 29, 2014
Ended 04:14 PM Mar 29, 2014

03/18

Impact

Severity

Billing system upgrade
Status : Closed

Posted at: 03/18/2014 12:02 pm
As of 6am CDT the ability to purchase physical servers has been restored.

Posted at: 03/18/2014 02:43 am
The billing system upgrade is now complete and everything is working without issue, except for the ability to purchase new physical servers which should be working again by 6AM CDT.

Posted at: 03/17/2014 11:59 pm
This maintenance is necessary to perform upgrades on our billing system. During this maintenance we will be upgrading to a newer version of our billing and client relationship management system.

This maintenance will interrupt interacting with the the following systems:
Leap
Leap3
Dropzone API Calls
Billing

If you have any questions or concerns please contact our Support Operations Center at noc@singlehop.com or call us at 1-866-817-2811x2

Started 12:00 AM Mar 18, 2014
Ended 12:44 AM Mar 18, 2014

03/10

Impact

Severity

Phonesystem Outage
Status : Closed

Posted at: 03/10/2014 02:58 pm
Greetings,

The phonesystem is now back online.

Posted at: 03/10/2014 02:30 pm
Greetings,

Currently, we are experiencing an issue with our Phonesystem that is affecting the ability of our sales and support staff to take incoming phone calls. Our network operations team is currently investigating the issue. If you have a support request, please use the ticket system as it is still online. Thanks.

Started 02:30 PM Mar 10, 2014
Ended 03:00 PM Mar 10, 2014

02/18

Impact

Severity

Non-impacting CR3 Aggregation Maintenance
Status : Closed

Posted at: 02/18/2014 11:18 am
All work has been completed at this time.

Posted at: 02/18/2014 10:04 am
Hello, The maintenance is continuing. No impact is expected.

Posted at: 02/18/2014 08:44 am
Hello, The maintenance is continuing. No impact is expected.

Posted at: 02/18/2014 08:00 am
Hello, The maintenance is continuing. No impact is expected.

Posted at: 02/18/2014 07:01 am
Hello,

The maintenance is continuing. There was a slight outage for the first switch but we have found the issue and it will be resolved for the additional devices.

Posted at: 02/18/2014 06:09 am
Hello,

We are starting this maintenance now, please standby.

Started 06:08 AM Feb 18, 2014
Ended 09:00 AM Feb 18, 2014

02/18

Impact

Severity

CHI-2 Network outage
Status : Closed

Posted at: 02/18/2014 06:54 pm
The official RFO for this event has been uploaded here:

http://adrepo.singlehop.com/rfo/02182014.pdf

Please contact our support team with any questions.

Posted at: 02/18/2014 04:23 am


Started 12:47 AM Feb 18, 2014
Ended 01:25 AM Feb 18, 2014

02/14

Impact

Severity

Non-impacting Core Maintenance
Status : Closed

Posted at: 02/14/2014 06:03 am
The maintenance is now complete.

Posted at: 02/14/2014 05:31 am
The maintenance is continuing, please standby.

Posted at: 02/14/2014 05:04 am
We are starting the maintenance at our CHI-2 data center now.

Started 05:00 AM Feb 14, 2014
Ended 06:00 AM Feb 14, 2014

02/13

Impact

Severity

Inbound Denial of Service Attack affecting CHI-1 Datacenter
Status : Closed

Posted at: 02/13/2014 04:32 pm
Greetings,

The SingleHop Network Operations Center has detected a large Denial of Service (DoS) attack in the CHI-1 Datacenter that may have impacted network services. Network Engineering staff began working on the issue immediately, and have isolated and removed the affected target. Network Engineering staff continue to monitor the situation closely for any further issues.

Posted at: 02/13/2014 04:29 pm


Started 03:50 PM Feb 13, 2014
Ended 04:01 PM Feb 13, 2014

02/09

Impact

Severity

Transport Link - Chi-1/Chi-2
Status : Closed

Posted at: 02/09/2014 05:03 pm
Our link provider has corrected the issue and services are returning to normal.

Posted at: 02/09/2014 04:59 pm
At 3:46PM the transport link between Chi-1 and Chi-2 went down. All systems remain publicly accessible. However, this issue is affecting LEAP logins and private network connections between the two data centers.

Started 03:46 PM Feb 09, 2014
Ended 04:45 PM Feb 09, 2014

02/07

Impact

Severity

DNS Resolvers - Chi-2
Status : Closed

Posted at: 02/07/2014 01:14 am
This maintenance has been completed. All DNS resolvers are verified online and responding properly.

Thank you.

Posted at: 02/07/2014 01:09 am
This is an emergency maintenance performed by one of our transport providers.
During this maintenance window, access to the DNS resolvers in Chi-2 DC may be impacted or unavailable at times.

Please standby for further updates.

Thank you.

Started 01:02 AM Feb 07, 2014
Ended 01:11 AM Feb 07, 2014

02/06

Impact

Severity

Upstream connectivity loss in PHX-1
Status : Closed

Posted at: 02/06/2014 03:02 pm
This issue was resolved, normal connectivity has been restored.

Posted at: 02/06/2014 03:02 pm
Our PHX-1 data center lost upstream connectivity for a few minutes. This affected the PHX-1 location as well as DNS resolvers for some servers in other facilities.

Started 02:49 PM Feb 06, 2014
Ended 02:52 PM Feb 06, 2014

02/06

Impact

Severity

Public network issues in dl.111
Status : Closed

Posted at: 02/06/2014 10:50 am
The switch replacement is complete. If you are still experiencing issues, please submit a new ticket to our Technical Support team so that we can check this for you. Thank you for your patience while we resolved this issue.

Posted at: 02/06/2014 10:37 am
The switch replacement is underway and the ETA for full restoration of service is about 20 minutes.

Posted at: 02/06/2014 10:13 am
We are currently replacing the public network switch in this cab. This will cause 100% downtime while we perform the hardware swap. We apologize for the inconvenience; unfortunately, we are unable to resolve this issue without replacing the hardware. We will continue to keep this status up-to-date with the latest information.

Posted at: 02/06/2014 09:35 am
We are currently experiencing an issue with the public network on some servers in the dl.111 cab at the CHI-2 facility. We are investigating and expect this to be resolved shortly. No other locations are currently affected.

Started 09:05 AM Feb 06, 2014
Ended 10:50 AM Feb 06, 2014

01/20

Impact

Severity

Outage at Chi-2
Status : Closed

Posted at: 01/20/2014 09:48 pm
All services are normal at this time.

Posted at: 01/20/2014 09:36 pm
Many devices and services have come back online. We are still working to ensure all services are up.

Posted at: 01/20/2014 09:24 pm
Starting around 9:00PM Central we started receiving alerts from the network at Chi-2. Many customers with servers at Chi-2 are affected. Our network engineers are currently troubleshooting nd working to restore service.

Started 09:00 PM Jan 20, 2014
Ended 09:45 PM Jan 20, 2014

01/15

Impact

Severity

AMS Core Maintenance
Status : Closed

Posted at: 01/15/2014 11:08 pm
The maintenance is now complete. Thank you for your patience.

Posted at: 01/15/2014 10:05 pm
We are starting the AMS core maintenance now. No outage is expected.

Started 10:05 AM Jan 15, 2014
Ended 10:08 AM Jan 16, 2014

12/23

Impact

Severity

Scheduled emergency maintenance - aswg-c06.d03.s02.phx01
Status : Closed

Posted at: 12/23/2013 10:35 pm
Greetings,

The maintenance is completed and full service has been restored.

Thank you for your continued patience in this process.

Posted at: 12/23/2013 10:28 pm
Greetings,

A scheduled emergency maintenance for aswg-c06.d03.s02.phx01 has begun. As soon as more information is available, we will update this status page.

Thank you for your patience in this matter.

Started 10:26 PM Dec 23, 2013
Ended 10:32 PM Dec 23, 2013

12/18

Impact

Severity

priv-asw-er186-4.ord03 private network outage
Status : Closed

Posted at: 12/18/2013 03:56 pm
The problem has been corrected and private network communication has been restored. We apologize for any inconvenience this may have caused and appreciate your patience!

Posted at: 12/18/2013 03:42 pm
Greetings,

Our network operations team is currently investigating issues with the switch priv-asw-er186-4.ord03.singlehop.net. This switch provides access to private network resources.

At this time, we are still working to restore service. As soon as more information is available, we will let you know.

Please note that this outage should *not* affect public network communications for servers connected to this switch.

Please stand by for further updates. Thank you for your patience!

Started 03:04 PM Dec 18, 2013
Ended 03:42 PM Dec 18, 2013

12/03

Impact

Severity

cab switch asw-fy132.ord03 outage
Status : Closed

Posted at: 12/03/2013 06:42 pm
Greetings,

The switch has been restored to a normal operational status. We will continue to investigate the root cause for the issue with an RFO to follow.

We apologize for the inconvenience.

Posted at: 12/03/2013 06:37 pm
Cab switch asw-fy132.ord03 is experiencing trouble at this time. Please standby while we are still working to troubleshoot the issue and locate the root cause.

Posted at: 12/03/2013 06:27 pm
Cab switch asw-fy132.ord03 is experiencing trouble at this time. Please standby while we are still working to troubleshoot the issue and locate the root cause.

Posted at: 12/03/2013 06:20 pm
We are still working to restore service to this switch. Please standby for additional updates.

Posted at: 12/03/2013 06:03 pm
Greetings,

Cab switch asw-fy132.ord03 is experiencing trouble at this time. Please standby while we troubleshoot the issue and locate the root cause.

Started 05:55 PM Dec 03, 2013
Ended 06:40 PM Dec 03, 2013

11/19

Impact

Severity

Security Upgrade Maintenance
Status : Closed

Posted at: 11/19/2013 11:01 pm
This maintenance is now complete.

Posted at: 11/19/2013 10:22 pm
We are starting this maintenance now. No impact is expected.

Started 10:20 AM Nov 19, 2013
Ended 11:00 AM Nov 20, 2013

11/01

Impact

Severity

Cabinet switch aswg-eg186.ord03 outage
Status : Closed

Posted at: 11/01/2013 01:15 am
The switch is now online and functioning normally. This was a power related issue. The issue will be investigated and addressed internally.
Please open a Technical Support ticket if you need assistance.
This status report is now closed.
Thank you.

Posted at: 11/01/2013 01:10 am
Our monitoring systems indicate this switch is down.
Please stand by for further updates.

Started 01:05 AM Nov 01, 2013
Ended 01:09 AM Nov 01, 2013

10/29

Impact

Severity

Cab switch aswg-ej183-2.ord03
Status : Closed

Posted at: 10/30/2013 12:51 am
This emergency maintenance has been completed. The switch is functioning normally and connectivity verified.
This status report is now closed.

Posted at: 10/30/2013 12:40 am
We are now proceeding to reset the switch chassis. This may take up to 10 minutes.
Please stand by for further updates.

Posted at: 10/30/2013 12:08 am
We are still troubleshooting this issue. Traffic is being forwarded normally, however, we are still seeing an issue that may require further action.
Please stand by for further updates.

Posted at: 10/29/2013 11:51 pm
We have identified an issue with this cab switch that is causing intermittent latency. We have opened an emergency maintenance window for this cab switch. Customers may notice up to 7 minutes of traffic disruption.
Please stand by for further updates.

Started 11:35 PM Oct 29, 2013
Ended 12:48 AM Oct 30, 2013

10/11

Impact

Severity

Phone system to be temporarily offline for maintenance
Status : Closed

Posted at: 10/12/2013 12:09 am
Phone system maintenance has been completed.

Posted at: 10/11/2013 11:31 pm
Our around-the-clock phone support will be offline for about 30 minutes as we perform upgrades to our internal office network. During this time our ticket system will continue to be online, so for the fastest support responses we encourage clients to submit support tickets.

Thank you.

Started 11:45 PM Oct 11, 2013
Ended 12:10 AM Oct 12, 2013

10/10

Impact

Severity

PHX-1 to CHI-2 Transport Link Outage
Status : Closed

Posted at: 10/10/2013 05:51 pm
Greetings,

Hardware replacement has been completed and the transport link is back online at this time.

An official RFO will be made available as soon as we have received it.

Thank you for your continued patience in this matter.

Posted at: 10/10/2013 05:09 pm
Greetings,

We've received reports that hardware replacement is underway. Estimated completion time for this process is 6:00PM CDT.

Posted at: 10/10/2013 04:27 pm
Greetings,

A field tech has arrived on site at the location of the outage and is investigating the issue.

Please stand by for further updates.

Posted at: 10/10/2013 02:41 pm
Greetings,

A technician is en route to the upstream provider site to further investigate this issue. The ETA for the technician's arrival is approximately 30 minutes.

As soon as we have further information we will update this status.

Thank you for your continued patience in this matter.

Posted at: 10/10/2013 02:16 pm
The upstream provider is still working to restore service to this circuit. At this time we do not have an ETA but as soon as we have more information we will update this page. Please stand by for further updates. Thank you for your continued patience!

Posted at: 10/10/2013 01:54 pm
Greetings,

The transport link between our CHI-2 and PHX-1 data centers has gone down. We are working with our upstream providers to restore service as quickly as possible.

This issue is actively being worked on. The impact should only be seen on private network traffic between the two locations.

We will update this thread again as soon as more information becomes available. Thank you.

Started 01:40 PM Oct 10, 2013
Ended 05:40 PM Oct 10, 2013

10/01

Impact

Severity

Phone system outage
Status : Closed

Posted at: 10/01/2013 02:43 pm
This has been resolved by our ISP.

Posted at: 10/01/2013 01:41 pm
An ISP issue is causing issues with our phone support system. Some clients may be experiencing issues contacting SingleHop via phone. All client services are online, including the LEAP ticket system.

Started 01:13 PM Oct 01, 2013
Ended 02:30 PM Oct 01, 2013

09/30

Impact

Severity

CHI-2 access switch aswg-ej183-2.ord03 outage
Status : Closed

Posted at: 09/30/2013 09:47 pm
Greetings,

aswg-ej183-2.ord03 is back online and operating properly at this time.

Posted at: 09/30/2013 09:41 pm
Greetings,

We are presently experiencing an outage one of our public gigabit access switches at our CHI-2 facility aswg-ej183-2.ord03. Please standby for further updates while we investigate the issue.

Started 09:30 PM Sep 30, 2013
Ended 09:40 PM Sep 30, 2013

09/19

Impact

Severity

CHI-2 access switch aswg-ej183-2.ord03 outage
Status : Closed

Posted at: 09/19/2013 06:39 pm
Greetings,

We experienced a short outage with aswg-ej183-2.ord03. The switch is back online at this time. If you are still encountering problems, please contact our support department for additional assistance. We will provide an RFO for this event shortly.

Thank You.

Started 06:22 PM Sep 19, 2013
Ended 06:28 PM Sep 19, 2013

09/18

Impact

Severity

CHI-1 Core Code Upgrade
Status : Closed

Posted at: 09/18/2013 02:34 am
All services have been fully restored. If you experience any additional issues please open a ticket with our NOC.

Posted at: 09/18/2013 02:18 am
We are continuing to bring the router back to full redundancy. Customers should see no other impacts while we finish this work.

Posted at: 09/18/2013 02:08 am
Router has been successfully rebooted and services are coming back online now.

Posted at: 09/18/2013 01:56 am
Hello,

We will not be upgrading the core, please standby.

Posted at: 09/18/2013 01:07 am
We are starting this maintenance now. We will be doing some prep work before the upgrade. There will be another update before the upgrade starts.

Started 01:05 AM Sep 18, 2013
Ended 02:34 AM Sep 18, 2013

09/11

Impact

Severity

Chi-2 CR-3 Aggregation - intermittent packet loss
Status : Closed

Posted at: 09/11/2013 11:59 am
This issue has been resolved. We verified connectivity.
This status report is now closed.

Posted at: 09/11/2013 11:44 am
We are still working on this issue. Further updates will provided shortly.

Posted at: 09/11/2013 11:12 am
We are still working on this issue. We will provide more information as it becomes available.

Posted at: 09/11/2013 11:00 am
We are noticing intermittent packet loss to some customers in this segment of the network. Not all customers in the segment are impacted.
We are troubleshooting this issue right now.
Please stand by for further updates.

Started 10:35 AM Sep 11, 2013
Ended 11:55 AM Sep 11, 2013

09/11

Impact

Severity

Provider intermittent connectivity in Chi-2
Status : Closed

Posted at: 09/11/2013 09:10 am
The issue has been resolved as of 8:40AM CDT. Connectivity has been verified.
This status report is now closed.

Posted at: 09/11/2013 08:24 am
We are continuing to work on this issue.
Thank you for your patience in this matter.

Posted at: 09/11/2013 07:52 am
Troubleshooting continues, we will provide more information as it becomes available.

Posted at: 09/11/2013 07:31 am
We are still working on this issue.
We will keep this status report updated with more information as it becomes available.

Posted at: 09/11/2013 07:06 am
We are still troubleshooting intermittent connectivity issue.
Further updates will be posted here. Please standby.

Posted at: 09/11/2013 05:50 am
We are observing intermittent connectivity issues that may impact some customers in Chi-2
We are working with our provider to resolve these issues.
Please standby for further updates.

Started 05:30 AM Sep 11, 2013
Ended 08:40 AM Sep 11, 2013

09/10

Impact

Severity

CHI-2 fu.139 cab switch outage
Status : Closed

Posted at: 09/10/2013 07:49 pm
Greetings,

The access switch located in fu.139 experienced an unrecoverable error. We completed a hard reboot of the device, which is back online and working properly at this time. We will closely monitor this switch and open a new status message in the event that this should occur again. Clients should have seen their network connectivity restored at this time. If you are still experiencing network related issues, please contact our support department.

Thank You,

Posted at: 09/10/2013 07:35 pm
Greetings,

We are still working the issue at this time. Please standby for additional updates.

Thank You,

Posted at: 09/10/2013 07:23 pm
Greetings,

We are still troubleshooting this issue attempting to find the root cause. Please standby for additional updates.

Thank You,

Posted at: 09/10/2013 07:16 pm
Greetings,

We are experiencing trouble with one of our cab access switches at CHI-2. Please standby while we gather more information and identify the issue.

Thank You

Started 07:10 PM Sep 10, 2013
Ended 07:45 PM Sep 10, 2013

09/04

Impact

Severity

Cab switch outage - aswg-dl112.ord03
Status : Closed

Posted at: 09/04/2013 03:19 am
The switch is now functioning normally. Please open a Technical Support ticket if you are still experiencing any issues with any server in this cab switch.

Thank you.

Posted at: 09/04/2013 03:10 am
The switch hardware has been replaced. We are now verifying connectivity.

Posted at: 09/04/2013 02:45 am
We are working on replacing the switch hardware at this time.

Posted at: 09/04/2013 02:30 am
We have an active alert on possible switch outage.
We are investigating this issue right now.
Please standby for further updates.

Started 02:08 AM Sep 04, 2013
Ended 03:04 AM Sep 04, 2013

08/11

Impact

Severity

Networking Issues with CHI-2 Core Routers
Status : Closed

Posted at: 08/11/2013 07:00 am
The issue with our core routers and associated cab switches has been resolved. At this time we are investigating how to mitigate this issue from occurring in the future. We will provide an RFO shortly regarding this event.

Posted at: 08/11/2013 06:23 am
We are still working on bringing up 2 access switches at CHI-2. Most clients are not affected by this issue and have seen network connectivity fully restored. Please standby for additional updates.

Posted at: 08/11/2013 05:37 am
We have identified the root cause of the issue and are working to rectify it at this time. Clients should begin to see network connectivity restored shortly. Please standby for additional updates.

Posted at: 08/11/2013 04:14 am
We have identified the issue and are still working on stabilizing the network at this time. Please standby for additional updates.

Posted at: 08/11/2013 03:57 am
We have identified the issue with our core routers in CHI-2 and continue to resolve it at this time. Please standby for additional updates

Posted at: 08/11/2013 03:39 am
We have identified the issue with our core routers in CHI-2 and are working to resolve it at this time. Please standby for additional updates

Posted at: 08/11/2013 03:24 am
We are continuing to stabilize the situation and bring the network back to full functionality. Please standby for additional updates.

Posted at: 08/11/2013 03:12 am
We are still trying to stabilize the situation and bring the network back to full functionality. Please standby for additional updates.

Posted at: 08/11/2013 03:07 am
We are still working the issue with our CHI-2 routers and will continue to provide periodic updates regarding this issue.

Posted at: 08/11/2013 02:54 am
We are still working the issue on our CHI-2 core routes and will continue to provide periodic updates.

Posted at: 08/11/2013 02:43 am
We are experiencing an issue with our CHI-2 core routers. At this time we continue to investigate the issue and are working to resolve it as soon as possible.

Posted at: 08/11/2013 02:05 am
We are currently experiencing some networking issues that appear to be effecting servers connectivity. This is being investigated currently.

Started 01:45 AM Aug 11, 2013
Ended 06:45 AM Aug 11, 2013

08/06

Impact

Severity

Cab switch outage - aswg-el180.ord03
Status : Closed

Posted at: 08/07/2013 12:12 am
The issue has been resolved and the switch is now functioning properly.
This status report is now closed.
Thank you.

Posted at: 08/07/2013 12:00 am
We've been alerted to connectivity issues with this cab switch. We are gathering initial information at this stage.
Please standby for further updates.

Started 11:51 PM Aug 06, 2013
Ended 12:09 AM Aug 07, 2013

07/28

Impact

Severity

priv-asw-fx185 private switch issue
Status : Closed

Posted at: 07/28/2013 10:02 pm
We have fixed the issue with the private access switch in FX.185. Private network connectivity has been restored. This was caused by a faulty uplink cross connect cable between the cab and our core. We have moved the device over to a different cable which immediately restored service. We will be repairing the faulty cable, which will not impact any services.

Posted at: 07/28/2013 09:42 pm
We are presently working toward a solution to restore connectivity to the private access switch in FX.185. Please standby for additional updates.

Posted at: 07/28/2013 09:32 pm
We are still experiencing a loss of connectivity with the private switch located in FX.185. We have identified the issue and are presently working on a solution at this time. Please standby for additional updates.

Posted at: 07/28/2013 09:19 pm
We are still working the issue with this particular switch and will provide an update shortly.

Posted at: 07/28/2013 09:05 pm
We are presently experiencing an issue with our private access switch in FX.185. Please standby for additional updates while we continue to work the issue and investigate the root cause.

Posted at: 07/28/2013 08:57 pm
We are presently experiencing an issue with one of our private access switches in FX.185. We are investigating the issue and will provide regular updates once we have found the root cause.

Started 08:40 PM Jul 28, 2013
Ended 09:59 PM Jul 28, 2013

07/26

Impact

Severity

Public cab switch maintenance - asw-eh183.ord03
Status : Closed

Posted at: 07/26/2013 12:31 am
The maintenance has been completed. If you are experiencing any issues, then please open a Technical Support ticket.
This status report is now closed.
Thank you.

Posted at: 07/26/2013 12:16 am
This is a scheduled maintenance. The switch will be brought down during this maintenance window.
Please standby for further updates.

Started 12:16 AM Jul 26, 2013
Ended 12:28 AM Jul 26, 2013

07/25

Impact

Severity

Public cab switch maintenance - asw-ex183.ord03
Status : Closed

Posted at: 07/25/2013 10:36 pm
The maintenance has been completed. If you are experiencing any issues, then please open a Technical Support ticket.
This status report is now closed.
Thank you.

Posted at: 07/25/2013 10:22 pm
This is a scheduled maintenance. The switch will be brought down during this maintenance window.
Please standby for further updates.

Started 10:23 PM Jul 25, 2013
Ended 10:34 PM Jul 25, 2013

07/25

Impact

Severity

Public cab switch maintenance - asw-eh176.ord03
Status : Closed

Posted at: 07/25/2013 12:32 am
The maintenance has been completed. If you are experiencing any issues, then please open a Technical Support ticket.
This status report is now closed.
Thank you.

Posted at: 07/25/2013 12:11 am
This is a scheduled maintenance. The switch will be brought down during this maintenance window.
Please standby for further updates.

Started 12:12 AM Jul 25, 2013
Ended 12:25 AM Jul 25, 2013

07/24

Impact

Severity

Public cab switch maintenance - asw-ei176.ord03
Status : Closed

Posted at: 07/24/2013 10:33 pm
The maintenance has been completed. If you are experiencing any issues, then please open a Technical Support ticket.
This status report is now closed.
Thank you.

Posted at: 07/24/2013 10:12 pm
This is a scheduled maintenance. The switch will be brought down during this maintenance window.
Please standby for further updates.

Started 10:11 PM Jul 24, 2013
Ended 10:23 PM Jul 24, 2013

07/08

Impact

Severity

Cab switch outage - aswg-gf150.ord03
Status : Closed

Posted at: 07/08/2013 12:32 am
This issue has been resolved. The cab switch is functioning normally.

This status report is now closed.

Thank you.

Posted at: 07/08/2013 12:16 am
We have been alerted to cab switch aswg-gf150.ord03 possible outage. We are collecting initial information and investigating this issue.
Please standby for further updates.

Started 12:05 PM Jul 08, 2013
Ended 12:27 PM Jul 08, 2013

06/28

Impact

Severity

VMware Public Cloud at CHI-2 issue
Status : Closed

Posted at: 06/28/2013 12:42 pm
All VMs were confirmed online at 12:34PM. If you have any further issues, please contact our Support team.

Posted at: 06/28/2013 10:34 am
The network connection on VMware compute node 4 experienced a flap on the connection to the SAN device. Because the network device had a heartbeat, the failover did not occur, although the issue caused intermittent connectivity between the VMs and the SAN.

A systematic rebooting of all impacted virtual machines was immediately conducted and is ongoing. Currently, all nodes have access to the SAN device again, although some VMs are experiencing read-only file systems that require a reboot to resolve.

Started 04:15 AM Jun 28, 2013
Ended

06/26

Impact

Severity

Cab switch outage - asw-c02.r02.ord02
Status : Closed

Posted at: 06/26/2013 07:01 am
The switch is now online. We will be collecting more information and working with our vendor in order to identify and fix the root cause of this crash.

This status report is now closed.

Thank you.

Posted at: 06/26/2013 06:53 am
We have been alerted to cab switch asw-c02.r02.ord02 possible outage. We are investigating this issue and collecting initial information.
Please stand by for further updates.

Started 06:45 AM Jun 26, 2013
Ended 06:49 AM Jun 26, 2013

06/22

Impact

Severity

Provider Outage in AMS
Status : Closed

Posted at: 06/22/2013 07:17 pm
The provider has stabilized. We will be looking into this issue further with our provider.

Posted at: 06/22/2013 07:00 pm
We had a provider flap in AMS. We are currently working with the vendor to find the root cause.

Started 06:51 AM Jun 22, 2013
Ended 06:54 AM Jun 22, 2013

06/20

Impact

Severity

Scheduled Maintenance - asw-c16g.r02.ord02.singlehop.net
Status : Closed

Posted at: 06/20/2013 10:50 pm
The maintenance is completed and this window is now being closed.

If you are still experiencing connectivity issues, please feel free to contact our support team to have the issue further investigated.

Thank you for your patience throughout this process.

Posted at: 06/20/2013 10:37 pm
Greetings,

The maintenance is completed and the switch and all connected servers should be back online. We are performing final cleanup and checks of connectivity at this time.

Please stand by for further updates.

Posted at: 06/20/2013 10:01 pm
The scheduled maintenance window for asw-c16g.r02.ord02.singlehop.net is now open and maintenance will begin shortly.

Please stand by for further updates. Thank you.

Started 10:00 AM Jun 20, 2013
Ended 10:37 AM Jun 20, 2013

06/20

Impact

Severity

CHI-2 Routing Issues
Status : Closed

Posted at: 06/20/2013 10:14 pm
The network is currently stable and we have not seen any further issues. We will be closing this status entry at this time. A detailed incident report will be provided on request within 72 hours.

Posted at: 06/20/2013 09:55 pm
We are still monitoring the situation to verify stability.

Posted at: 06/20/2013 09:37 pm
The last update was made in error. Everything is still functioning normally and we are still monitoring the situation.

Posted at: 06/20/2013 09:35 pm
We are still working on providing a temporary solution to this issue. Please standby for additional updates.

Posted at: 06/20/2013 09:20 pm
We applied a solution to resolve the initial outage but that solution triggered a bug in some of the older versions of code running on our network. This caused some additional outages for our customers. A work around has been found and the network has been stabilized. We will continue to monitor this situation to verify full stability.

Posted at: 06/20/2013 09:17 pm
We are still working on providing a temporary solution to this issue. Please standby for additional updates.

Posted at: 06/20/2013 09:05 pm
The network has been stabilized once again. We are continuing to work the issue at this time. Please standby for additional updates.

Posted at: 06/20/2013 08:53 pm
We are continuing to work the issue, and provide a temporary solution to this event at this time. Please standby for additional updates.

Posted at: 06/20/2013 08:44 pm
Customers may be experiencing a routing impact at this time. Please standby for additional updates as we continue to investigate.

Posted at: 06/20/2013 08:34 pm
We are still working on providing a temporary solution to this issue. Please standby for additional updates.

Posted at: 06/20/2013 08:23 pm
We are continuing to work the issue, and provide a temporary solution to this event at this time. Please standby for additional updates.

Posted at: 06/20/2013 08:15 pm
The network has been stabilized once again. We are continuing to work the issue at this time. Please standby for additional updates.

Posted at: 06/20/2013 08:04 pm
Customers may still be experiencing issues with routing, we are working on stabilizing the issue and will continue to investigate the root cause which triggered this event. Please standby for additional updates.

Posted at: 06/20/2013 07:57 pm
Customers may still be experiencing issues with routing again, we are looking into what is causing this event and will provide updates shortly.

Posted at: 06/20/2013 07:46 pm
Customers may be experiencing issues with routing again, we are looking into what is causing this event and will provide updates shortly.

Posted at: 06/20/2013 07:27 pm
Once again, we have found the root cause of what triggered the event and are continuing to work on a temporary solution. Please standby for additional updates.

Posted at: 06/20/2013 07:20 pm
We have determined the root cause of the event and are now working toward a temporary solution. Please standby for additional updates.

Posted at: 06/20/2013 06:58 pm
We are still looking into the root cause of this event. Please standby for additional updates.

Posted at: 06/20/2013 06:31 pm
The network has returned to a stable condition at this time. We are still investigating what triggered this event. Please standby for additional updates.

Posted at: 06/20/2013 06:13 pm
We are still investigating the issues with our core at CHI-2 in regards to routing impact. Please standby for additional updates.

Posted at: 06/20/2013 06:00 pm
Customers may be experiencing a routing impact at this time. Please standby for additional updates as we continue to investigate.

Posted at: 06/20/2013 05:55 pm
We are presently experiencing an issue with a core router at our CHI-2 datacenter. Please standby while we investigate further.

Started 05:52 AM Jun 20, 2013
Ended 09:00 AM Jun 20, 2013

06/14

Impact

Severity

Public Switch Outage at CHI-1
Status : Closed

Posted at: 06/14/2013 09:43 pm
Our network administrators traced the outage to a massive DDoS that took the switch completely offline and was also slightly affecting other switches in the data center. If there are any issues, please open a support ticket so we can investigate further.

Posted at: 06/14/2013 09:24 pm
The public network switch asw-c21.r02.ord02 is not responding and is not reachable after a reboot. Data center and Network staff have been notified and will begin checking on this very shortly. Thanks for your patience.

Started 09:10 AM Jun 14, 2013
Ended 09:40 AM Jun 14, 2013

06/13

Impact

Severity

Public cab switch maintenance - asw-fw139.ord03
Status : Closed

Posted at: 06/13/2013 10:57 pm
Switch maintenance has been completed. Switch passed all post-maintenance tests, and connectivity has been verified.

This status report is now closed.

Thank you.

Posted at: 06/13/2013 10:10 pm
We are now starting this scheduled maintenance. The switch will be brought down for maintenance.

Started 10:00 AM Jun 13, 2013
Ended 10:50 AM Jun 13, 2013

05/30

Impact

Severity

Provider flap in PHX
Status : Closed

Posted at: 05/30/2013 11:00 am
We had provider flap at our PHX DC which could have created some flapping of services for our customers. The issue has been resolved and we are currently investigating the root cause of the outage.

Started 10:41 AM May 30, 2013
Ended 10:43 AM May 30, 2013

05/25

Impact

Severity

Upstream provider issue at CHI-2
Status : Closed

Posted at: 05/25/2013 04:57 pm
All services have now been restored.

Posted at: 05/25/2013 04:45 pm
Services are now being restored, please standby. No outage is expected from this.

Posted at: 05/25/2013 04:22 pm
We are still waiting for our provider to restore services.

Posted at: 05/25/2013 03:54 pm
The routing issue with Nlayer has been resolved, but the provider is still working on restoring service.

Posted at: 05/25/2013 03:43 pm
There seems to be some stale routes on the Nlayer network. We are working with them to get these removed ASAP.

Posted at: 05/25/2013 03:29 pm
We are seeing some potential lingering issues with NLayer and our Network Operations team is investigating.

Posted at: 05/25/2013 03:18 pm
We experienced an outage with one of our upstream providers, Nlayer, that may have caused a service flap at our CHI-2 facility. This issue was resolved within less than 2 minutes.

Started 02:53 AM May 25, 2013
Ended 02:55 AM May 25, 2013

05/20

Impact

Severity

Public Switch Down
Status : Closed

Posted at: 05/20/2013 07:06 am
This was due to an attack on a client connected to this switch. The attack has been mitigated, and the switch is now back online.

Posted at: 05/20/2013 06:59 am
The switch in the server cabinent gj174 is currently unresponsive. We are aware of the issue, and our network operators are investigating this now.

Started 06:45 AM May 20, 2013
Ended

03/25

Impact

Severity

Routing issue on aswg-gf150
Status : Closed

Posted at: 03/25/2013 09:16 am
This issue was resolved at 9:01AM CST and has not reoccurred. The routing issue was resolved by our Network Operations team.

Posted at: 03/25/2013 09:00 am
We are currently experiencing a routing issue on this switch. Our Network Operations team is working to resolve this issue now. We will keep this space updated.

Started 08:39 AM Mar 25, 2013
Ended 09:01 AM Mar 25, 2013

03/21

Impact

Severity

Connectivity Issue - CHI-1 BGP reset
Status : Closed

Posted at: 03/21/2013 11:11 am
The issue is fully resolved at this time. Details on what caused the outage will follow once our final investigation is complete.

Posted at: 03/21/2013 10:58 am
Access has been restored to our ticketing system. We are still focusing our efforts to restore access to the remaining portions of our network that are still experiencing issues.

Please stand by for further updates. Thank you for your patience in this matter.

Posted at: 03/21/2013 10:32 am
Most services are restored however we are still seeing issues with a small portion of our network. We are working to resolve this as quickly as possible.

Please stand by for further updates.

Posted at: 03/21/2013 10:21 am
We are still working to resolve this issue and will have further updates as soon as they become available.

Please stand by. Thank you.

Posted at: 03/21/2013 10:10 am
Network Operations has detected a BGP session reset on dr6506b.ord02.singlehop.net. We are actively investigating the issue.

Started 10:05 AM Mar 21, 2013
Ended 11:00 AM Mar 21, 2013

03/18

Impact

Severity

Power issue at CHI-1
Status : Closed

Posted at: 03/18/2013 08:08 pm
At this time, we have verified that all connectivity has been restored. Thanks and have a good day.

Posted at: 03/18/2013 07:42 pm
The power has already been restored by our upstream provider, but we are working to ensure our infrastructure and affected servers are back online. Thank you for your patience.

Posted at: 03/18/2013 07:41 pm
We are experiencing a power issue at our CHI-1 facility that may be affecting some client servers as well as some of our backend infrastructure. We will update this post once we have further details.

Started 07:30 AM Mar 18, 2013
Ended 07:40 AM Mar 18, 2013

03/18

Impact

Severity

Power issue at CHI-1
Status : Closed

Posted at: 03/18/2013 07:27 pm
At this time power has been restored and our internal networks, infrastructure, and client's with affected servers are now online. If you are having issues still, please let us know.

Posted at: 03/18/2013 06:22 pm
Power is coming back on. The LEAP3 control panel system is back online. Service may be slow at this time as the systems start to come back online. We are still verifying the full resolution of this issue.

Posted at: 03/18/2013 06:11 pm
The electricians working on this issue have given us an ETA of about 15 minutes. Full service restoration may take slightly longer as servers and network equipment boot back up.

Posted at: 03/18/2013 05:36 pm
An electrician is en route to investigate the issue further.

Posted at: 03/18/2013 05:12 pm
A power outage at our CHI-1 facility is affecting 5 cabinets (207.1, 207.2, 207.3, 207.4, and 206.1). We are working with our upstream provider who has a technician on-site working to resolve this issue.

Posted at: 03/18/2013 04:42 pm
Due to a bug in our update system, this status post improperly posted the start time as 4:28am, and that the issue has ended. The issue started at 4:28PM CST, and we are still working on resolving this. Thank you for your patience.

Posted at: 03/18/2013 04:38 pm
We are experiencing a power issue at our CHI-1 facility that may be affecting some client servers as well as some of our backend infrastructure. We will update this post once we have further details.

Started 04:27 AM Mar 18, 2013
Ended 06:14 AM Mar 18, 2013

03/13

Impact

Severity

SnapShield Firewall Outage
Status : Closed

Posted at: 03/13/2013 02:33 pm
There was an attack that targeted a customer behind a SnapShield firewall. The customer was removed and service has been restored.

Started 02:20 PM Mar 13, 2013
Ended 02:30 PM Mar 13, 2013

03/11

Impact

Severity

Possible packet loss
Status : Closed

Posted at: 03/11/2013 12:31 pm
We stopped seeing packet loss around 12:00 CST. We will provide an RFO once our analysis of the cause of these issues is complete.

Posted at: 03/11/2013 11:31 am
We are investigating an issue that may be causing packet loss for some customers on our network. We do not have the full scope of this issue yet, but it appears to be affecting some clients in the CHI-2 datacenter. We will keep you posted as we determine the scope, cause, and resolution to this issue.

Started 10:57 AM Mar 11, 2013
Ended 11:28 AM Mar 12, 2013

03/11

Impact

Severity

Provider Outage
Status : Closed

Posted at: 03/11/2013 08:42 pm
Hello,

Everything has continued to be stable. Service should be fully restored. We are going to work with our vendor and provider to investigate the issue further and find a solution.

Posted at: 03/11/2013 08:12 pm
Hello,

We are currently experiencing outage with one of our transit providers. We are currently investigating the issue. As of right now service should be restored.

Started 07:55 AM Mar 11, 2013
Ended 08:05 AM Mar 11, 2013

03/01

Impact

Severity

Private network cab switch outage
Status : Closed

Posted at: 03/01/2013 07:50 am
As part of this status report, we had an alert for a Public Cloud node outage. This issue has also been resolved and connectivity verified.

Pleas open a Technical Support ticket if you need further assistance.

Thank you.

Posted at: 03/01/2013 07:43 am
Update: We still have an alert for a Public Cloud node being down.
This alert has not been cleared yet.
Please stand by for further updates.

Posted at: 03/01/2013 07:32 am
The issue has been isolated to a power interruption event, and it has been resolved.
While this outage did not impact the public network connectivity, some customers may have noticed interruption to some back-end functions, such as running backup operations.

This status report is now closed.
Please open a Technical Support ticket if you need further assistance.

Thank you.

Posted at: 03/01/2013 07:07 am
We have been alerted by our monitoring system to a private network switch outage.
This may be impacting some of our Public Cloud virtual servers functionalities.
Please stand by for further updates.

Started 07:00 AM Mar 01, 2013
Ended 07:43 AM Mar 01, 2013

02/18

Impact

Severity

LEAP3 / Dropzone API Temporary Outage
Status : Closed

Posted at: 02/18/2013 01:52 pm
This has been resolved

Posted at: 02/18/2013 01:51 pm
Greetings,

We are aware of an issue with logging into LEAP3 or using our Dropzone API. We are working quickly to resolve this issue.

Started 01:50 AM Feb 18, 2013
Ended 01:52 AM Feb 18, 2013

02/07

Impact

Severity

c05.d03.s02.phx01 Public switch upgrade
Status : Closed

Posted at: 02/07/2013 10:55 pm
This maintenance has been completed successfully.

Thank you.

Posted at: 02/07/2013 10:44 pm
The hardware has been upgraded and the switch is no back online.
We are now performing post-maintenance tasks and cleaning up.

Posted at: 02/07/2013 10:19 pm
We are now bringing the cab switch offline for maintenance.

Posted at: 02/07/2013 10:12 pm
This scheduled maintenance window is now open. We will be upgrading the public switch hardware for cab c05.d03.s02

Please stand by for further updates.

Started 10:10 AM Feb 07, 2013
Ended 10:55 AM Feb 07, 2013

01/31

Impact

Severity

Ge.150 Public switch upgrade
Status : Closed

Posted at: 01/31/2013 11:03 pm
This maintenance has been completed and connectivity has been verified.

Thank you.

Posted at: 01/31/2013 10:58 pm
The upgrade is complete, and the switch is back up.
We are now verifying connectivity and cleaning up.

Posted at: 01/31/2013 10:42 pm
We are performing scheduled maintenance to upgrade this cab switch. We will be bringing the switch down briefly.

Started 10:30 AM Jan 31, 2013
Ended 11:00 AM Jan 31, 2013

01/30

Impact

Severity

aswg-ej183-2.ord03.singlehop.net - Maintenance
Status : Closed

Posted at: 01/31/2013 12:04 am
The uplink issue has been corrected and this switch is online and working properly at this time.

Posted at: 01/30/2013 11:41 pm
We are still working on restoring connectivity to one of the switches uplinks. This should not cause any interruption for any devices on this switch. Please standby for additional updates.

Posted at: 01/30/2013 11:21 pm
We are still working on restoring connectivity to one of the switches uplinks. This should not cause any interruption for any devices on this switch. Please standby for additional updates.

Posted at: 01/30/2013 11:09 pm
We are still working on restoring connectivity to one of the switches uplinks. This should not cause any interruption for any devices on this switch. Please standby for additional updates.

Posted at: 01/30/2013 10:51 pm
We are still working to complete this maintenance. Please standby for additional updates.

Posted at: 01/30/2013 10:25 pm
We have completed the switch replacement and have restored network connectivity. Please standby while we verify everything is working properly.

Posted at: 01/30/2013 10:01 pm
We will be starting the switch replacement for aswg-ej183-2.ord03.singlehop.net now. Please standby as we replace the device and restore network connectivity as soon as possible.

Started 10:00 AM Jan 30, 2013
Ended 12:05 PM Jan 31, 2013

01/29

Impact

Severity

Gigabit switch issue: aswg-ej183-2
Status : Closed

Posted at: 01/29/2013 01:11 pm
This device is being scheduled for an emergency maintenance to replace the hardware. Should the hardware fail again before the scheduled maintenance, the device will be immediately replaced.

Posted at: 01/29/2013 09:32 am
We are experiencing some connectivity issues for some clients on the switch aswg-ej183-2 and are working to resolve these as quickly as possible. Thank you for your patience, further information will be provided as it is available.

Started 09:25 AM Jan 29, 2013
Ended 10:00 AM Jan 29, 2013

01/29

Impact

Severity

Public switch emergency maintenance - aswg-ej183-2.ord03
Status : Closed

Posted at: 01/29/2013 05:20 am
The switch code has been updated. The switch is back up, and connectivity has been verified.

Please open a support ticket if you still need assistance.

This status report is now closed.

Thank you.

Posted at: 01/29/2013 05:04 am
We are scheduling an emergency maintenance to update the code on this public switch.
We will bring the switch down in 5 minutes.
Please stand by for updates.

Started 05:05 AM Jan 29, 2013
Ended 05:15 AM Jan 29, 2013

01/28

Impact

Severity

Customer Gigabit Switch Outage: aswg-ej183-2 inaccessible
Status : Closed

Posted at: 01/28/2013 04:03 pm
We have been able to pinpoint the reason for the connectivity issues with this switch. In the event that we loose connectivity again, we will promptly update the code to restore service. We apologize for any inconvenience this may have caused and appreciate your patience regarding this matter.

Posted at: 01/28/2013 11:42 am
The switch is accessible again. We are still investigating the root cause of this issue and ensuring a permanent resolution is applied.

Posted at: 01/28/2013 11:26 am
The gigabit switch aswg-ej183-2 in the CHI-2 datacenter is currently inaccessible. Our Network Operations team is currently working with Cisco TAC to ensure that this issue is permanently resolved. Thank you for your patience, we will keep this posted with all available updates.

Started 11:23 AM Jan 28, 2013
Ended 11:55 AM Jan 28, 2013

01/27

Impact

Severity

Customer Gigabit Switch Outage - aswg-ej183-2.ord03.singlehop.net
Status : Closed

Posted at: 01/28/2013 12:13 am
We will continue to monitor this switch and be looking into the root cause of this issue.

Posted at: 01/28/2013 12:03 am
Network connectivity has been restored to servers on the affected switch.

Posted at: 01/27/2013 11:49 pm
At this time we are still attempting to restore network connectivity to servers connected to this device. Please standby for additional updates.

Posted at: 01/27/2013 11:38 pm
We continue to investigate and work toward restoring network connectivity on this switch. Please standby for additional updates.

Posted at: 01/27/2013 11:26 pm
We are still in the process restoring network connectivity regarding this device. Please standby for additional updates.

Posted at: 01/27/2013 11:11 pm
We are experiencing a connectivity issue with this customer gigabit switch. We are in the process of investigating and restoring service as soon as possible. Please standby for additional updates.

Started 11:00 AM Jan 27, 2013
Ended 12:00 PM Jan 28, 2013

01/24

Impact

Severity

Customer Cab Switch Outage
Status : Closed

Posted at: 01/24/2013 07:31 pm
The customer cab switch is complete. Network connectivity has been restored to this cab.

Posted at: 01/24/2013 07:21 pm
We are still working on restoring network connectivity to servers in this cab. Please standby for additional updates.

Posted at: 01/24/2013 07:07 pm
We are in the process of bringing the new customer cab switch online. Please standby for additional updates.

Posted at: 01/24/2013 06:50 pm
The customer cab switch has failed and we are in the process of replacing it at this time. Please standby for additional updates.

Posted at: 01/24/2013 06:25 pm
At approximately 18:15 CST (GMT -6) our network monitoring system indicated that asw-c21.r01.ord02.singlehop.net had gone offline. We are investigating this issue at this time and working to restore services as quickly as possible.

Started 06:15 AM Jan 24, 2013
Ended 07:32 AM Jan 24, 2013

01/19

Impact

Severity

Public Switch Outage - aswg-ep180.ord03
Status : Closed

The switch replacement is completed and the new device is online. Service has been fully restored. If you still have trouble, please open a trouble ticket.

Thank you for your patience.

--

Network Operations is currently configuring the replacement hotspare device with the most recent configuration backup of the failed device.

Please stand by for further updates.

--

The switch device for EP.180 has failed. We are currently in the process of preparing a replacement hotspare device. Further updates will be provided shortly. Thank you for your patience.

--

Network Operations is currently investigating an outage in EP.180 with the public network switch servicing this cab. We will update with more information as it becomes available.

Started 11:12 AM Jan 19, 2013
Ended 12:05 PM Jan 20, 2013

01/04

Impact

Severity

Fx139 Public Switch Outage
Status : Closed

The switch has been replaced and service has been restored.

-----

The switch has failed. We are preparing a new switch at this time for replacement.

-----

We are currently experiencing an outage with the public switch in cabinet FX139. We are working to resolve the situation now.

Started 01:03 AM Jan 04, 2013
Ended 02:05 AM Jan 04, 2013

12/11

Impact

Severity

Configuration anomaly
Status : Closed

A configuration anomaly caused all or part of the CHI-1 DC to temporarily lose connectivity for approximately 2 minutes. The configuration that was causing the issue was removed and service has been fully restored.

Started 07:45 PM Dec 11, 2012
Ended 07:47 PM Dec 11, 2012

12/06

Impact

Severity

Cab switch scheduled maintenance
Status : Closed

This maintenance has been completed. Service has been restored, and connectivity has been verified.
Please open a Technical Support ticket if you need further assistance.

Thank you.

----

Services have been restored. We are now verifying connectivity and cleaning up.

----

We are now reloading the device and gradually bringing up service.
Please stand by.

----

This maintenance window is now open.
Please stand by for further updates.

Started 10:08 AM Dec 06, 2012
Ended 10:35 AM Dec 06, 2012

12/06

Impact

Severity

Emergency Maintenance
Status : Closed

This maintenance is now complete.

----

The device has been successfully reloaded. We are now finishing up our clean up process. We should incur no other outages during this maintenance.

----

We will now be reloading the device.

-----

We are starting this maintenance now. Once we are ready to reload the device we will send another update.

Started 03:29 AM Dec 06, 2012
Ended 06:20 AM Dec 06, 2012

12/03

Impact

Severity

Non-impacting Core Maintenance
Status : Closed

The Maintenance is complete.

----

Completing final tasks for the maintenance.

----

For approximately 2 minutes starting at 10:31PM CST, access to the CHI-2 facility was unavailable. This was due to some BGP sessions on the device clearing after a new transit provider was brought online. This is not a normal situation, we are going to contact the vendor for further investigation. Normal service has already been restored.

----

Starting the maintenance now.

----

We are just about ready to start the maintenance.

----

Started 10:00 PM Dec 03, 2012
Ended 11:04 PM Dec 03, 2012

11/30

Impact

Severity

Cab switch outage
Status : Closed

We've been alerted by our monitoring system to an issue with this cab switch.
We are now investigating the issue and will update the status here shortly.
Please standby.

----

This issue has been resolved, the impact was minimal.
Thank you for your patience.

Started 08:23 AM Nov 30, 2012
Ended 08:27 AM Nov 30, 2012

11/29

Impact

Severity

DNS Resolution Temporary Outage
Status : Closed

Although our DNS resolver system has been restarted, we are having issues with internal networking and DNS resolution still. We are looking into this, and we will update you shortly. Thanks for your patience.

*- This issue is now resolved, thank you for your patience in this matter.

Started 07:16 AM Nov 29, 2012
Ended 08:55 AM Nov 29, 2012

11/18

Impact

Severity

CHI-2 Connectivity Issues
Status : Closed

The SingleHop network operations team is investigating connectivity issues at CHI-2. Please stand by for further updates.

--

We are still investigating the issue. Services are partially restored and we are working to achieve full restoration of services at the CHI-2 datacenter. As more details become available we will update this status post.

--

The network event has ended and we have reached a stable state. The SingleHop network operations team is still investigating the cause of this event. As soon as we have completed our investigation an RFO will be provided.

Thank you for your patience in this matter.

Started 01:10 PM Nov 18, 2012
Ended 01:43 PM Jan 18, 2012

11/13

Impact

Severity

Emergency reboot of ticket system backend required
Status : Closed

The ticketing system backend required an emergency reboot. This completed without major incident and the ticketing system is now back online and operational. We apologize for any inconvenience this may have caused. No client servers were affected and our phone system remained online during the outage.

Started 07:00 PM Nov 13, 2012
Ended 07:20 PM Nov 13, 2012

11/08

Impact

Severity

Connectivity Issues - dr6506a.ord03.singlehop.net
Status : Closed

The SingleHop network operations team is currently investigating network connectivity issues at our CHI-2 Datacenter.

Please stand by for further updates.

---
We are still experiencing issues with dr6506a.ord03.singlehop.net. We are currently moving traffic over to the second core at CHI-2.

Please stand by for further updates.

---
We've reached stability at this time but are still looking into the cause of this outage.

Further updates will be provided as more information becomes available.

Started 03:31 PM Nov 08, 2012
Ended 04:06 PM Nov 08, 2012

11/05

Impact

Severity

CHI-2 Public Network Upgrade
Status : Closed

The maintenance is now complete

-----

We are in the process of migrating cabinets.

-----

We are still in the process of migrating cabinets.

-----

Done with prep work. Starting to migrate cabinets now.

-----

Starting the maintenance now.

Started 10:00 PM Nov 05, 2012
Ended 11:01 AM Nov 06, 2012

11/03

Impact

Severity

Switch outage
Status : Closed

The switch in this cabinet is unresponsive. Technicians are currently investigating and working to bring this back online as soon as possible. We are working to replace the switch hardware now. As of 10:15PM CST, the switch software configuration is complete and all servers have been re-attached.

Started 08:55 PM Nov 03, 2012
Ended 10:15 PM Nov 03, 2012

10/15

Impact

Severity

shared-cloud10-sata.singlehop.com outage
Status : Closed

09:19 AM CDT The cloud node is now back online, and vms are in the process of booting. The raid controller had failed, and needed to be replaced. No data loss should occur due to this.
--
This shared cloud node is down due to some seemingly storage related issues. Or datacenter operations team is currently working to restore service to the cloud vms contained on this machine. We'll update as more information becomes available.

Started 09:00 AM Oct 15, 2012
Ended 09:15 AM Oct 15, 2012

09/28

Impact

Severity

Cab switch scheduled maintenance
Status : Closed

Connectivity has been verified. This scheduled maintenance is now complete.

---

The maintenance has been completed. We are now verifying connectivity for each server.

---

This is a scheduled maintenance to replace the cab switch.
The switch will be brought down. Please stand by for further updates.

Started 12:05 PM Sep 28, 2012
Ended 12:20 PM Sep 28, 2012

09/25

Impact

Severity

PHX-1 Aggregation Upgrade
Status : Closed

The Maintenance is now complete. No outage was detected. All maintenance tickets should close within two days.

----

We are starting the maintenance now.

Started 01:17 PM Sep 25, 2012
Ended 11:56 PM Sep 25, 2012

09/20

Impact

Severity

Packet Loss on Cab Switch
Status : Closed

This issue should now be resolved. A misconfiguration between two devices on the switch impacted the CPU and caused packet loss for other customers. The configuration issue has been resolved and the CPU is back down to normal levels.

-----

We are currently working on a gig cab switch issue. Customers may currently be seeing packet loss on this switch. Stay tuned for further updates.

Started 01:00 PM Sep 20, 2012
Ended 01:10 PM Sep 20, 2012

09/19

Impact

Severity

CHI-1 Private Core Upgrade
Status : Closed

The maintenance is now complete. All tickets will close within two business days.

-----

The upgrade is now complete. Cleaning up configuration now.

-----

First outage complete. Moving on to second outage.

-----

You should start seeing an outage now.

------

We are starting the private core upgrade in CHI-1 now.

Started 10:00 PM Sep 19, 2012
Ended 12:56 AM Sep 20, 2012

07/16

Impact

Severity

CHI-1 to CHi-2 Transport Link Down
Status : Closed

A Software bug in the provider's hardware caused instability in the protection protocol on the transport link between the CHI-1 and CHI-2 data centers. Stability was restored approximately 30 minutes later.

Started 01:24 PM Jul 16, 2012
Ended 01:52 PM Jul 16, 2012

06/07

Impact

Severity

CR3 Aggregation Device Failover
Status : Closed

While our colocation company was running new cross connections to one of our core cabinets, a power cable dislodged from an aggregation layer device. This incident caused a loss of power to the device, requiring us to perform an emergency failover to our backup aggregation device. It took twelve minutes for the backup aggregation device to become active and bring our systems back online.

Started 10:08 AM Jun 07, 2012
Ended 10:20 AM Jun 07, 2012

05/11

Impact

Severity

Outage from Software Upgrade
Status : Closed

During a normal scheduled maintenance there was unexpected impact that caused an outage throughout the CHI-2 data center. A time line of the events is as follows:

6:05 am - Customers experienced the outage that was detailed in the maintenance.
6:25 am - A device failed while being updated which caused a flap in network services. This negatively impacted the core and thus brought down all services at CHI-2.
6:47 am - Services were restored to 75% of the customers at CHI-2.
7:25 am - Services were restored to 100% of the customers at CHI-2.
8:20 am - During the cleanup of this outage, a secondary switch malfunctioned and created a layer 2 issue on the network that impacted the core once again.
9:04 am - All services were restored to 100% of the customers at CHI-2.

Started 06:12 AM May 11, 2012
Ended 09:35 AM May 11, 2012

09/11

Impact

Severity

Connectivity Issues CHI-2
Status : CLOSED

Connectivity issues at CHI-2- techs are aware and working to resolve it.

Posted at : Sep 11, 2012 at 06:31
Austin Wilson says:
We apologize for ending the maintenance late. The extra time was required to ensure the least possible impact for our customers. All work has now been completed. All maintenance tickets will automatically close within two days.

Posted at : Sep 11, 2012 at 05:51
Austin Wilson says:
The resources have been refreshed. Migrating traffic back now.

Posted at : Sep 11, 2012 at 05:07
Austin Wilson says:
Traffic migration complete, starting resource refresh now.

Posted at : Sep 11, 2012 at 04:46
Austin Wilson says:
80% done with migrating traffic to secondary core.

Posted at : Sep 11, 2012 at 04:05
Austin Wilson says:
Hello, We are starting our core maintenance now. Please standby for further updates.

08/28

Impact

Severity

Connectivity Issues CHI-2
Status : CLOSED

Connectivity issues at CHI-2- techs are aware and working to resolve it.

Posted at : Aug 28, 2012 at 06:43
Austin Wilson says:
All maintenance work has been completed. No outage was experienced. Your maintenance tickets will automatically close within the next two days.

Posted at : Aug 28, 2012 at 05:06
Austin Wilson says:
We are starting the maintenance now.

08/25

Impact

Severity

Cabinet issues
Status : CLOSED

Cabinet(s) #aswg-ej183-2.ord03 having connectivity issues our techs are working on it.

Posted at : Aug 25, 2012 at 22:28
Garett Murphy says:
Greetings, The maintenance is complete. If you need further technical assistance, please open a ticket. Thank you.

Posted at : Aug 25, 2012 at 22:01
Garett Murphy says:
Greetings, Maintenance for aswg-ej183-2.ord03 is now commencing. Please stand by for further updates. Thank you.

08/24

Impact

Severity

Connectivity issues in PHX-1
Status : CLOSED

Connectivity issues in PHX-1- techs are aware and working to resolve it.

Posted at : Aug 24, 2012 at 02:06
Mustafa Hussein says:
This maintenance has been completed. Connectivity has been verified for each system in this cab. Please open a separate Technical Support ticket if you are having problems with your server. This maintenance window is now closed, and this status report is now closed. Thank you.

Posted at : Aug 24, 2012 at 01:54
Mustafa Hussein says:
The scheduled maintenance for this cab is still underway. We apologize that the status report has not been posted properly. The maintenance tasks have been completed and the cab switch is online. We are now verifying connectivity for each server before closing this status report. Please stand by for further updates. Thank you.

08/16

Impact

Severity

Connectivity Issues CHI-1
Status : CLOSED

Connectivity issues at CHI-1- techs are aware and working to resolve it.

Posted at : Aug 17, 2012 at 01:04
Mustafa Hussein says:
This scheduled maintenance has been completed. Connectivity has been verified. If you need technical help, then please open a Technical Support ticket. This maintenance status report is now closed. Thank you.

Posted at : Aug 16, 2012 at 23:11
Mustafa Hussein says:
This is a scheduled maintenance for 207.16 The maintenance window is now open, during which, we will be performing hardware upgrade and other maintenance tasks. This status will be updated when the maintenance is finished. Thank you.

08/15

Impact

Severity

Cabinet issues
Status : CLOSED

Cabinet(s) #ED.108 having connectivity issues our techs are working on it.

Posted at : Aug 15, 2012 at 22:31
Garett Murphy says:
The maintenance is completed and we are now closing this issue. If you need technical assistance, please open a new ticket. Thank you.

Posted at : Aug 15, 2012 at 22:06
Garett Murphy says:
We are starting the Scheduled maintenance for ED.108 private network switch (priv-asw-ed108.ord03.singlehop.net). Please stand by for further updates.

08/04

Impact

Severity

Upstream Provider Issue
Status : CLOSED

Upstream providers is having technical issues. Route-reconvergence will occur in a few minutes.

Posted at : Aug 05, 2012 at 00:17
Garett Murphy says:
This issue is resolved. If you need further technical assistance, please open a new ticket. Thank you.

08/04

Impact

Severity

Upstream Provider Issue
Status : CLOSED

Upstream providers is having technical issues. Route-reconvergence will occur in a few minutes.

Posted at : Aug 05, 2012 at 00:16
Garett Murphy says:
This issue is resolved. If you need further technical assistance, please open a new ticket. Thank you.

Posted at : Aug 04, 2012 at 19:00
Garett Murphy says:
Greetings, We are aware of some possible connectivity issues for traffic going over our upstream provider NLayer at the CHI-2 Datacenter. This is related to an issue with the upstream provider's hardware and they are aware and working to resolve it. Please stand by for further updates regarding this issue. We apologize for the inconvenience.

08/02

Impact

Severity

Connectivity Issues CHI-2
Status : CLOSED

Connectivity issues at CHI-2- techs are aware and working to resolve it.

Posted at : Aug 02, 2012 at 22:19
Garett Murphy says:
Greetings, The maintenance is completed. Thank you for your patience. Should you require further technical assistance, please open a support ticket. Thank you.

Posted at : Aug 02, 2012 at 22:02
Garett Murphy says:
We are starting the scheduled maintenance for SnapShield device sfw01.ord03.singlehop.net. Please stand by for further updates. Thank you.

07/29

Impact

Severity

Connectivity Issues CHI-2
Status : CLOSED

Connectivity issues at CHI-2- techs are aware and working to resolve it.

Posted at : Jul 29, 2012 at 08:27
Mustafa Hussein says:
The issue has been resolved, and connectivity has been verified. The impact of this hardware malfunction was minimal as the fail-over system kicked in. We will be working with our hardware vendor to determine the root cause of this brief outage. This status update is now closed. Thank you for you patience.

Posted at : Jul 29, 2012 at 08:18
Mustafa Hussein says:
The hardware issue has been resolved, and we are working on verifying connectivity at this time. Please stand by for further updates. Thank you.

Posted at : Jul 29, 2012 at 08:09
Mustafa Hussein says:
We are now working on bringing the faulty hardware back on line. This hardware malfunction may have impacted traffic flow briefly, some customers may have noticed a brief disruption. Please stand by for further updates. Thank you.

Posted at : Jul 29, 2012 at 07:53
Mustafa Hussein says:
We have been alerted by our monitoring system to a malfunctioning hardware. We are working on resolving the issue and gathering more information. Please stand by for updates.

What do those colored bars mean?

Client Impact indicates how many customers are impacted by the issue. The higher the Client Impact score, the more customers that will experience this issue.

Severity indicates how impactful the issue is. Remember, the issue's impact is a factor of both Client Impact and Severity.

Some issues impact a small number of customers in a very bad way. Those issues would have a low Client Impact score but a high Severity score.

Issues that impact many customers but are minor, such as minor network latency, would be reversed. Those issues would have a high Client Impact score but a low Severity score. Issues that impact a large number of clients in a very impactful way would have two very high scores.

Tech Support: Toll-Free: 1-866-817-2811

Submit a ticket to LEAP

Submit Now
Click for automatic reload every 30 seconds