Posted at: 03/25/2013 09:16 am
This issue was resolved at 9:01AM CST and has not reoccurred. The routing issue was resolved by our Network Operations team.

Posted at: 03/25/2013 09:00 am
We are currently experiencing a routing issue on this switch. Our Network Operations team is working to resolve this issue now. We will keep this space updated.

Started 08:39 AM Mar 25, 2013
Ended 09:01 AM Mar 25, 2013

Posted at: 03/21/2013 11:11 am
The issue is fully resolved at this time. Details on what caused the outage will follow once our final investigation is complete.

Posted at: 03/21/2013 10:58 am
Access has been restored to our ticketing system. We are still focusing our efforts to restore access to the remaining portions of our network that are still experiencing issues.

Please stand by for further updates. Thank you for your patience in this matter.

Posted at: 03/21/2013 10:32 am
Most services are restored however we are still seeing issues with a small portion of our network. We are working to resolve this as quickly as possible.

Please stand by for further updates.

Posted at: 03/21/2013 10:21 am
We are still working to resolve this issue and will have further updates as soon as they become available.

Please stand by. Thank you.

Posted at: 03/21/2013 10:10 am
Network Operations has detected a BGP session reset on dr6506b.ord02.singlehop.net. We are actively investigating the issue.

Started 10:05 AM Mar 21, 2013
Ended 11:00 AM Mar 21, 2013

Posted at: 03/18/2013 08:08 pm
At this time, we have verified that all connectivity has been restored. Thanks and have a good day.

Posted at: 03/18/2013 07:42 pm
The power has already been restored by our upstream provider, but we are working to ensure our infrastructure and affected servers are back online. Thank you for your patience.

Posted at: 03/18/2013 07:41 pm
We are experiencing a power issue at our CHI-1 facility that may be affecting some client servers as well as some of our backend infrastructure. We will update this post once we have further details.

Started 07:30 AM Mar 18, 2013
Ended 07:40 AM Mar 18, 2013

Posted at: 03/18/2013 07:27 pm
At this time power has been restored and our internal networks, infrastructure, and client's with affected servers are now online. If you are having issues still, please let us know.

Posted at: 03/18/2013 06:22 pm
Power is coming back on. The LEAP3 control panel system is back online. Service may be slow at this time as the systems start to come back online. We are still verifying the full resolution of this issue.

Posted at: 03/18/2013 06:11 pm
The electricians working on this issue have given us an ETA of about 15 minutes. Full service restoration may take slightly longer as servers and network equipment boot back up.

Posted at: 03/18/2013 05:36 pm
An electrician is en route to investigate the issue further.

Posted at: 03/18/2013 05:12 pm
A power outage at our CHI-1 facility is affecting 5 cabinets (207.1, 207.2, 207.3, 207.4, and 206.1). We are working with our upstream provider who has a technician on-site working to resolve this issue.

Posted at: 03/18/2013 04:42 pm
Due to a bug in our update system, this status post improperly posted the start time as 4:28am, and that the issue has ended. The issue started at 4:28PM CST, and we are still working on resolving this. Thank you for your patience.

Posted at: 03/18/2013 04:38 pm
We are experiencing a power issue at our CHI-1 facility that may be affecting some client servers as well as some of our backend infrastructure. We will update this post once we have further details.

Started 04:27 AM Mar 18, 2013
Ended 06:14 AM Mar 18, 2013

Posted at: 03/13/2013 02:33 pm
There was an attack that targeted a customer behind a SnapShield firewall. The customer was removed and service has been restored.

Started 02:20 PM Mar 13, 2013
Ended 02:30 PM Mar 13, 2013

Posted at: 03/11/2013 12:31 pm
We stopped seeing packet loss around 12:00 CST. We will provide an RFO once our analysis of the cause of these issues is complete.

Posted at: 03/11/2013 11:31 am
We are investigating an issue that may be causing packet loss for some customers on our network. We do not have the full scope of this issue yet, but it appears to be affecting some clients in the CHI-2 datacenter. We will keep you posted as we determine the scope, cause, and resolution to this issue.

Started 10:57 AM Mar 11, 2013
Ended 11:28 AM Mar 12, 2013

Posted at: 03/11/2013 08:42 pm
Hello,

Everything has continued to be stable. Service should be fully restored. We are going to work with our vendor and provider to investigate the issue further and find a solution.

Posted at: 03/11/2013 08:12 pm
Hello,

We are currently experiencing outage with one of our transit providers. We are currently investigating the issue. As of right now service should be restored.

Started 07:55 AM Mar 11, 2013
Ended 08:05 AM Mar 11, 2013

Posted at: 03/01/2013 07:50 am
As part of this status report, we had an alert for a Public Cloud node outage. This issue has also been resolved and connectivity verified.

Pleas open a Technical Support ticket if you need further assistance.

Thank you.

Posted at: 03/01/2013 07:43 am
Update: We still have an alert for a Public Cloud node being down.
This alert has not been cleared yet.
Please stand by for further updates.

Posted at: 03/01/2013 07:32 am
The issue has been isolated to a power interruption event, and it has been resolved.
While this outage did not impact the public network connectivity, some customers may have noticed interruption to some back-end functions, such as running backup operations.

This status report is now closed.
Please open a Technical Support ticket if you need further assistance.

Thank you.

Posted at: 03/01/2013 07:07 am
We have been alerted by our monitoring system to a private network switch outage.
This may be impacting some of our Public Cloud virtual servers functionalities.
Please stand by for further updates.

Started 07:00 AM Mar 01, 2013
Ended 07:43 AM Mar 01, 2013

Posted at: 02/18/2013 01:52 pm
This has been resolved

Posted at: 02/18/2013 01:51 pm
Greetings,

We are aware of an issue with logging into LEAP3 or using our Dropzone API. We are working quickly to resolve this issue.

Started 01:50 AM Feb 18, 2013
Ended 01:52 AM Feb 18, 2013

Posted at: 02/07/2013 10:55 pm
This maintenance has been completed successfully.

Thank you.

Posted at: 02/07/2013 10:44 pm
The hardware has been upgraded and the switch is no back online.
We are now performing post-maintenance tasks and cleaning up.

Posted at: 02/07/2013 10:19 pm
We are now bringing the cab switch offline for maintenance.

Posted at: 02/07/2013 10:12 pm
This scheduled maintenance window is now open. We will be upgrading the public switch hardware for cab c05.d03.s02

Please stand by for further updates.

Started 10:10 AM Feb 07, 2013
Ended 10:55 AM Feb 07, 2013

Posted at: 01/31/2013 11:03 pm
This maintenance has been completed and connectivity has been verified.

Thank you.

Posted at: 01/31/2013 10:58 pm
The upgrade is complete, and the switch is back up.
We are now verifying connectivity and cleaning up.

Posted at: 01/31/2013 10:42 pm
We are performing scheduled maintenance to upgrade this cab switch. We will be bringing the switch down briefly.

Started 10:30 AM Jan 31, 2013
Ended 11:00 AM Jan 31, 2013

Posted at: 01/31/2013 12:04 am
The uplink issue has been corrected and this switch is online and working properly at this time.

Posted at: 01/30/2013 11:41 pm
We are still working on restoring connectivity to one of the switches uplinks. This should not cause any interruption for any devices on this switch. Please standby for additional updates.

Posted at: 01/30/2013 11:21 pm
We are still working on restoring connectivity to one of the switches uplinks. This should not cause any interruption for any devices on this switch. Please standby for additional updates.

Posted at: 01/30/2013 11:09 pm
We are still working on restoring connectivity to one of the switches uplinks. This should not cause any interruption for any devices on this switch. Please standby for additional updates.

Posted at: 01/30/2013 10:51 pm
We are still working to complete this maintenance. Please standby for additional updates.

Posted at: 01/30/2013 10:25 pm
We have completed the switch replacement and have restored network connectivity. Please standby while we verify everything is working properly.

Posted at: 01/30/2013 10:01 pm
We will be starting the switch replacement for aswg-ej183-2.ord03.singlehop.net now. Please standby as we replace the device and restore network connectivity as soon as possible.

Started 10:00 AM Jan 30, 2013
Ended 12:05 PM Jan 31, 2013

Posted at: 01/29/2013 01:11 pm
This device is being scheduled for an emergency maintenance to replace the hardware. Should the hardware fail again before the scheduled maintenance, the device will be immediately replaced.

Posted at: 01/29/2013 09:32 am
We are experiencing some connectivity issues for some clients on the switch aswg-ej183-2 and are working to resolve these as quickly as possible. Thank you for your patience, further information will be provided as it is available.

Started 09:25 AM Jan 29, 2013
Ended 10:00 AM Jan 29, 2013

Posted at: 01/29/2013 05:20 am
The switch code has been updated. The switch is back up, and connectivity has been verified.

Please open a support ticket if you still need assistance.

This status report is now closed.

Thank you.

Posted at: 01/29/2013 05:04 am
We are scheduling an emergency maintenance to update the code on this public switch.
We will bring the switch down in 5 minutes.
Please stand by for updates.

Started 05:05 AM Jan 29, 2013
Ended 05:15 AM Jan 29, 2013

Posted at: 01/28/2013 04:03 pm
We have been able to pinpoint the reason for the connectivity issues with this switch. In the event that we loose connectivity again, we will promptly update the code to restore service. We apologize for any inconvenience this may have caused and appreciate your patience regarding this matter.

Posted at: 01/28/2013 11:42 am
The switch is accessible again. We are still investigating the root cause of this issue and ensuring a permanent resolution is applied.

Posted at: 01/28/2013 11:26 am
The gigabit switch aswg-ej183-2 in the CHI-2 datacenter is currently inaccessible. Our Network Operations team is currently working with Cisco TAC to ensure that this issue is permanently resolved. Thank you for your patience, we will keep this posted with all available updates.

Started 11:23 AM Jan 28, 2013
Ended 11:55 AM Jan 28, 2013

Posted at: 01/28/2013 12:13 am
We will continue to monitor this switch and be looking into the root cause of this issue.

Posted at: 01/28/2013 12:03 am
Network connectivity has been restored to servers on the affected switch.

Posted at: 01/27/2013 11:49 pm
At this time we are still attempting to restore network connectivity to servers connected to this device. Please standby for additional updates.

Posted at: 01/27/2013 11:38 pm
We continue to investigate and work toward restoring network connectivity on this switch. Please standby for additional updates.

Posted at: 01/27/2013 11:26 pm
We are still in the process restoring network connectivity regarding this device. Please standby for additional updates.

Posted at: 01/27/2013 11:11 pm
We are experiencing a connectivity issue with this customer gigabit switch. We are in the process of investigating and restoring service as soon as possible. Please standby for additional updates.

Started 11:00 AM Jan 27, 2013
Ended 12:00 PM Jan 28, 2013

Posted at: 01/24/2013 07:31 pm
The customer cab switch is complete. Network connectivity has been restored to this cab.

Posted at: 01/24/2013 07:21 pm
We are still working on restoring network connectivity to servers in this cab. Please standby for additional updates.

Posted at: 01/24/2013 07:07 pm
We are in the process of bringing the new customer cab switch online. Please standby for additional updates.

Posted at: 01/24/2013 06:50 pm
The customer cab switch has failed and we are in the process of replacing it at this time. Please standby for additional updates.

Posted at: 01/24/2013 06:25 pm
At approximately 18:15 CST (GMT -6) our network monitoring system indicated that asw-c21.r01.ord02.singlehop.net had gone offline. We are investigating this issue at this time and working to restore services as quickly as possible.

Started 06:15 AM Jan 24, 2013
Ended 07:32 AM Jan 24, 2013

The switch replacement is completed and the new device is online. Service has been fully restored. If you still have trouble, please open a trouble ticket.

Thank you for your patience.

--

Network Operations is currently configuring the replacement hotspare device with the most recent configuration backup of the failed device.

Please stand by for further updates.

--

The switch device for EP.180 has failed. We are currently in the process of preparing a replacement hotspare device. Further updates will be provided shortly. Thank you for your patience.

--

Network Operations is currently investigating an outage in EP.180 with the public network switch servicing this cab. We will update with more information as it becomes available.

Started 11:12 AM Jan 19, 2013
Ended 12:05 PM Jan 20, 2013

The switch has been replaced and service has been restored.

-----

The switch has failed. We are preparing a new switch at this time for replacement.

-----

We are currently experiencing an outage with the public switch in cabinet FX139. We are working to resolve the situation now.

Started 01:03 AM Jan 04, 2013
Ended 02:05 AM Jan 04, 2013

A configuration anomaly caused all or part of the CHI-1 DC to temporarily lose connectivity for approximately 2 minutes. The configuration that was causing the issue was removed and service has been fully restored.

Started 07:45 PM Dec 11, 2012
Ended 07:47 PM Dec 11, 2012

This maintenance has been completed. Service has been restored, and connectivity has been verified.
Please open a Technical Support ticket if you need further assistance.

Thank you.

----

Services have been restored. We are now verifying connectivity and cleaning up.

----

We are now reloading the device and gradually bringing up service.
Please stand by.

----

This maintenance window is now open.
Please stand by for further updates.

Started 10:08 AM Dec 06, 2012
Ended 10:35 AM Dec 06, 2012

This maintenance is now complete.

----

The device has been successfully reloaded. We are now finishing up our clean up process. We should incur no other outages during this maintenance.

----

We will now be reloading the device.

-----

We are starting this maintenance now. Once we are ready to reload the device we will send another update.

Started 03:29 AM Dec 06, 2012
Ended 06:20 AM Dec 06, 2012

The Maintenance is complete.

----

Completing final tasks for the maintenance.

----

For approximately 2 minutes starting at 10:31PM CST, access to the CHI-2 facility was unavailable. This was due to some BGP sessions on the device clearing after a new transit provider was brought online. This is not a normal situation, we are going to contact the vendor for further investigation. Normal service has already been restored.

----

Starting the maintenance now.

----

We are just about ready to start the maintenance.

----

Started 10:00 PM Dec 03, 2012
Ended 11:04 PM Dec 03, 2012

We've been alerted by our monitoring system to an issue with this cab switch.
We are now investigating the issue and will update the status here shortly.
Please standby.

----

This issue has been resolved, the impact was minimal.
Thank you for your patience.

Started 08:23 AM Nov 30, 2012
Ended 08:27 AM Nov 30, 2012

Although our DNS resolver system has been restarted, we are having issues with internal networking and DNS resolution still. We are looking into this, and we will update you shortly. Thanks for your patience.

*- This issue is now resolved, thank you for your patience in this matter.

Started 07:16 AM Nov 29, 2012
Ended 08:55 AM Nov 29, 2012

The SingleHop network operations team is investigating connectivity issues at CHI-2. Please stand by for further updates.

--

We are still investigating the issue. Services are partially restored and we are working to achieve full restoration of services at the CHI-2 datacenter. As more details become available we will update this status post.

--

The network event has ended and we have reached a stable state. The SingleHop network operations team is still investigating the cause of this event. As soon as we have completed our investigation an RFO will be provided.

Thank you for your patience in this matter.

Started 01:10 PM Nov 18, 2012
Ended 01:43 PM Jan 18, 2012

The ticketing system backend required an emergency reboot. This completed without major incident and the ticketing system is now back online and operational. We apologize for any inconvenience this may have caused. No client servers were affected and our phone system remained online during the outage.

Started 07:00 PM Nov 13, 2012
Ended 07:20 PM Nov 13, 2012

The SingleHop network operations team is currently investigating network connectivity issues at our CHI-2 Datacenter.

Please stand by for further updates.

---
We are still experiencing issues with dr6506a.ord03.singlehop.net. We are currently moving traffic over to the second core at CHI-2.

Please stand by for further updates.

---
We've reached stability at this time but are still looking into the cause of this outage.

Further updates will be provided as more information becomes available.

Started 03:31 PM Nov 08, 2012
Ended 04:06 PM Nov 08, 2012

The maintenance is now complete

-----

We are in the process of migrating cabinets.

-----

We are still in the process of migrating cabinets.

-----

Done with prep work. Starting to migrate cabinets now.

-----

Starting the maintenance now.

Started 10:00 PM Nov 05, 2012
Ended 11:01 AM Nov 06, 2012

The switch in this cabinet is unresponsive. Technicians are currently investigating and working to bring this back online as soon as possible. We are working to replace the switch hardware now. As of 10:15PM CST, the switch software configuration is complete and all servers have been re-attached.

Started 08:55 PM Nov 03, 2012
Ended 10:15 PM Nov 03, 2012

09:19 AM CDT The cloud node is now back online, and vms are in the process of booting. The raid controller had failed, and needed to be replaced. No data loss should occur due to this.
--
This shared cloud node is down due to some seemingly storage related issues. Or datacenter operations team is currently working to restore service to the cloud vms contained on this machine. We'll update as more information becomes available.

Started 09:00 AM Oct 15, 2012
Ended 09:15 AM Oct 15, 2012

Connectivity has been verified. This scheduled maintenance is now complete.

---

The maintenance has been completed. We are now verifying connectivity for each server.

---

This is a scheduled maintenance to replace the cab switch.
The switch will be brought down. Please stand by for further updates.

Started 12:05 PM Sep 28, 2012
Ended 12:20 PM Sep 28, 2012

The Maintenance is now complete. No outage was detected. All maintenance tickets should close within two days.

----

We are starting the maintenance now.

Started 01:17 PM Sep 25, 2012
Ended 11:56 PM Sep 25, 2012

This issue should now be resolved. A misconfiguration between two devices on the switch impacted the CPU and caused packet loss for other customers. The configuration issue has been resolved and the CPU is back down to normal levels.

-----

We are currently working on a gig cab switch issue. Customers may currently be seeing packet loss on this switch. Stay tuned for further updates.

Started 01:00 PM Sep 20, 2012
Ended 01:10 PM Sep 20, 2012

The maintenance is now complete. All tickets will close within two business days.

-----

The upgrade is now complete. Cleaning up configuration now.

-----

First outage complete. Moving on to second outage.

-----

You should start seeing an outage now.

------

We are starting the private core upgrade in CHI-1 now.

Started 10:00 PM Sep 19, 2012
Ended 12:56 AM Sep 20, 2012

A Software bug in the provider's hardware caused instability in the protection protocol on the transport link between the CHI-1 and CHI-2 data centers. Stability was restored approximately 30 minutes later.

Started 01:24 PM Jul 16, 2012
Ended 01:52 PM Jul 16, 2012

While our colocation company was running new cross connections to one of our core cabinets, a power cable dislodged from an aggregation layer device. This incident caused a loss of power to the device, requiring us to perform an emergency failover to our backup aggregation device. It took twelve minutes for the backup aggregation device to become active and bring our systems back online.

Started 10:08 AM Jun 07, 2012
Ended 10:20 AM Jun 07, 2012

During a normal scheduled maintenance there was unexpected impact that caused an outage throughout the CHI-2 data center. A time line of the events is as follows:

6:05 am - Customers experienced the outage that was detailed in the maintenance.
6:25 am - A device failed while being updated which caused a flap in network services. This negatively impacted the core and thus brought down all services at CHI-2.
6:47 am - Services were restored to 75% of the customers at CHI-2.
7:25 am - Services were restored to 100% of the customers at CHI-2.
8:20 am - During the cleanup of this outage, a secondary switch malfunctioned and created a layer 2 issue on the network that impacted the core once again.
9:04 am - All services were restored to 100% of the customers at CHI-2.

Started 06:12 AM May 11, 2012
Ended 09:35 AM May 11, 2012

Connectivity issues at CHI-2- techs are aware and working to resolve it.

Posted at : Sep 11, 2012 at 06:31
Austin Wilson says:
We apologize for ending the maintenance late. The extra time was required to ensure the least possible impact for our customers. All work has now been completed. All maintenance tickets will automatically close within two days.

Posted at : Sep 11, 2012 at 05:51
Austin Wilson says:
The resources have been refreshed. Migrating traffic back now.

Posted at : Sep 11, 2012 at 05:07
Austin Wilson says:
Traffic migration complete, starting resource refresh now.

Posted at : Sep 11, 2012 at 04:46
Austin Wilson says:
80% done with migrating traffic to secondary core.

Posted at : Sep 11, 2012 at 04:05
Austin Wilson says:
Hello, We are starting our core maintenance now. Please standby for further updates.

Connectivity issues at CHI-2- techs are aware and working to resolve it.

Posted at : Aug 28, 2012 at 06:43
Austin Wilson says:
All maintenance work has been completed. No outage was experienced. Your maintenance tickets will automatically close within the next two days.

Posted at : Aug 28, 2012 at 05:06
Austin Wilson says:
We are starting the maintenance now.

Cabinet(s) #aswg-ej183-2.ord03 having connectivity issues our techs are working on it.

Posted at : Aug 25, 2012 at 22:28
Garett Murphy says:
Greetings, The maintenance is complete. If you need further technical assistance, please open a ticket. Thank you.

Posted at : Aug 25, 2012 at 22:01
Garett Murphy says:
Greetings, Maintenance for aswg-ej183-2.ord03 is now commencing. Please stand by for further updates. Thank you.

Connectivity issues in PHX-1- techs are aware and working to resolve it.

Posted at : Aug 24, 2012 at 02:06
Mustafa Hussein says:
This maintenance has been completed. Connectivity has been verified for each system in this cab. Please open a separate Technical Support ticket if you are having problems with your server. This maintenance window is now closed, and this status report is now closed. Thank you.

Posted at : Aug 24, 2012 at 01:54
Mustafa Hussein says:
The scheduled maintenance for this cab is still underway. We apologize that the status report has not been posted properly. The maintenance tasks have been completed and the cab switch is online. We are now verifying connectivity for each server before closing this status report. Please stand by for further updates. Thank you.

Connectivity issues at CHI-1- techs are aware and working to resolve it.

Posted at : Aug 17, 2012 at 01:04
Mustafa Hussein says:
This scheduled maintenance has been completed. Connectivity has been verified. If you need technical help, then please open a Technical Support ticket. This maintenance status report is now closed. Thank you.

Posted at : Aug 16, 2012 at 23:11
Mustafa Hussein says:
This is a scheduled maintenance for 207.16 The maintenance window is now open, during which, we will be performing hardware upgrade and other maintenance tasks. This status will be updated when the maintenance is finished. Thank you.

Cabinet(s) #ED.108 having connectivity issues our techs are working on it.

Posted at : Aug 15, 2012 at 22:31
Garett Murphy says:
The maintenance is completed and we are now closing this issue. If you need technical assistance, please open a new ticket. Thank you.

Posted at : Aug 15, 2012 at 22:06
Garett Murphy says:
We are starting the Scheduled maintenance for ED.108 private network switch (priv-asw-ed108.ord03.singlehop.net). Please stand by for further updates.

Upstream providers is having technical issues. Route-reconvergence will occur in a few minutes.

Posted at : Aug 05, 2012 at 00:17
Garett Murphy says:
This issue is resolved. If you need further technical assistance, please open a new ticket. Thank you.

Upstream providers is having technical issues. Route-reconvergence will occur in a few minutes.

Posted at : Aug 05, 2012 at 00:16
Garett Murphy says:
This issue is resolved. If you need further technical assistance, please open a new ticket. Thank you.

Posted at : Aug 04, 2012 at 19:00
Garett Murphy says:
Greetings, We are aware of some possible connectivity issues for traffic going over our upstream provider NLayer at the CHI-2 Datacenter. This is related to an issue with the upstream provider's hardware and they are aware and working to resolve it. Please stand by for further updates regarding this issue. We apologize for the inconvenience.

Connectivity issues at CHI-2- techs are aware and working to resolve it.

Posted at : Aug 02, 2012 at 22:19
Garett Murphy says:
Greetings, The maintenance is completed. Thank you for your patience. Should you require further technical assistance, please open a support ticket. Thank you.

Posted at : Aug 02, 2012 at 22:02
Garett Murphy says:
We are starting the scheduled maintenance for SnapShield device sfw01.ord03.singlehop.net. Please stand by for further updates. Thank you.

Connectivity issues at CHI-2- techs are aware and working to resolve it.

Posted at : Jul 29, 2012 at 08:27
Mustafa Hussein says:
The issue has been resolved, and connectivity has been verified. The impact of this hardware malfunction was minimal as the fail-over system kicked in. We will be working with our hardware vendor to determine the root cause of this brief outage. This status update is now closed. Thank you for you patience.

Posted at : Jul 29, 2012 at 08:18
Mustafa Hussein says:
The hardware issue has been resolved, and we are working on verifying connectivity at this time. Please stand by for further updates. Thank you.

Posted at : Jul 29, 2012 at 08:09
Mustafa Hussein says:
We are now working on bringing the faulty hardware back on line. This hardware malfunction may have impacted traffic flow briefly, some customers may have noticed a brief disruption. Please stand by for further updates. Thank you.

Posted at : Jul 29, 2012 at 07:53
Mustafa Hussein says:
We have been alerted by our monitoring system to a malfunctioning hardware. We are working on resolving the issue and gathering more information. Please stand by for updates.