Platform of Choice for Private Cloud & Dedicated Servers
Hello. Thanks for visiting.

Have questions about SingleHop?

I'm here to help.

Call us toll free at 866.349.0689

Real-time system status

Start Live Chat

Detailed Event Log

  • 12/19
    Impact
    Severity
    Chi-3 intermittent connectivity
    Status : Closed

    Posted at: 12/19/2014 06:47 am
    The connectivity issues have been resolved and no further disruption was recorded in the past 45 minutes. We are working with our vendors on root cause analysis.
    Please open a support ticket if you are seeing any connectivity issues.

    Posted at: 12/19/2014 01:36 am
    We are investigating connectivity issues that may have impacted a number of servers in Chi-3 data center.
    We will provide more information as it becomes available.
    Please standby

    Started 01:12 AM Dec 19, 2014

    Ended 05:53 AM Dec 19, 2014

  • 12/19
    Impact
    Severity
    Non-Impacting Core Maintenance, Phx-1
    Status : Closed

    Posted at: 12/19/2014 07:04 am
    This maintenance has been completed. There was no impact resulting from this maintenance.
    Thank you.

    Posted at: 12/19/2014 12:49 am
    We are about to begin our scheduled maintenance on our core networking systems at the Phoenix 1 Data Center.
    There is no impact expected to connectivity while these tasks are being performed. We will update this post as any further information becomes available.
    Thanks.

    Started 01:00 AM Dec 19, 2014

    Ended 05:00 AM Dec 19, 2014

  • 12/18
    Impact
    Severity
    Cab c130.101.2.10 Networking Down
    Status : Closed

    Posted at: 12/18/2014 03:07 pm
    This issue was caused by a malfunctioning PDU. The equipment has been replaced to prevent further interruptions.

    Posted at: 12/18/2014 03:02 pm
    Service has been restored. We are investigating the cause of the issue.

    Posted at: 12/18/2014 02:56 pm
    We've received alerts indicating all networking in this cab is down. Our technicians are working to restore service.

    Started 02:45 PM Dec 18, 2014

    Ended 03:00 PM Dec 18, 2014

  • 12/11
    Impact
    Severity
    Public cloud - degraded performance
    Status : Closed

    Posted at: 12/11/2014 08:53 am
    Connectivity has been restored to the majority of the virtual servers. However there is still work being done to verify connectivity for a small number of servers.
    Please standby

    Posted at: 12/11/2014 07:34 am
    We have already identified the root cause and we are working on rectifying it. There is currently no estimated time for repair available, however we anticipate the issue to be resolved shortly.
    Please standby for further updates.

    Posted at: 12/11/2014 06:57 am
    We are still investigating this issue. Please standby for further updates.

    Posted at: 12/11/2014 06:26 am
    We are experiencing an issue with a segment of our cloud systems. This issue is impacting a limited number of virtual servers at the moment.
    We will provide more information as it becomes available.
    Please standby

    Started 06:09 AM Dec 11, 2014

    Ended 08:57 AM Dec 11, 2014

  • 12/07
    Impact
    Severity
    Non-Impacting Core Maintenance
    Status : Closed

    Posted at: 12/08/2014 01:57 am
    This maintenance event has been completed. No unexpected problems occurred during the window.

    Posted at: 12/07/2014 11:11 pm
    We have begun our scheduled maintenance on our core networking systems at the Phoenix 1 Data Center. There is no impact expected to connectivity while these tasks are being performed. We will update this post as any further information becomes available. Thanks.

    Started 11:00 PM Dec 07, 2014

    Ended 02:00 AM Dec 08, 2014

  • 12/07
    Impact
    Severity
    Failing Private Network Switch in cr130.101.5.8
    Status : Closed

    Posted at: 12/08/2014 12:40 am
    The replacement is complete and all devices are confirmed functional.

    Posted at: 12/08/2014 12:14 am
    A replacement switch has been configured. We are starting the replacement process.

    Posted at: 12/07/2014 08:52 pm
    The private network switch in cab cr130.101.5.8 has crashed and rebooted. We are preparing a replacement device. In the meantime, private network connectivity in this cab is restored. This has no effect on public networking.

    Started 08:00 PM Dec 07, 2014

    Ended 12:40 AM Dec 08, 2014

  • 11/24
    Impact
    Severity
    Private Network Outage in Ams Cab A05.R01.B13
    Status : Closed

    Posted at: 11/24/2014 11:14 pm
    The replacement is configured and online.

    Posted at: 11/24/2014 10:57 pm
    The replacement device is in place and is being configured.

    Posted at: 11/24/2014 10:21 pm
    Technicians are continuing to work on the switch replacement.

    Posted at: 11/24/2014 09:00 pm
    The top-of-rack private network switch in Amsterdam cab A05.R01.B13 has shutdown due to a hardware failure. Our technicians are working to replace the switch.

    Started 08:20 PM Nov 24, 2014

    Ended 11:10 PM Nov 24, 2014

  • 11/14
    Impact
    Severity
    Chi-2 / Phx Transport Link Flap
    Status : Closed

    Posted at: 11/14/2014 06:44 am
    The flapping has ceased and the link has been restored. All connectivity between the Chicago-2 and Phoenix data centers has been restored.

    We are working with our carrier to determine the root cause of the issue in order to avoid further disruptions.

    Posted at: 11/14/2014 06:41 am
    Our transport link between the Chicago-2 and Phoenix data centers was disrupted but our carrier link flapping. This primarily affects private networking between the two data centers, as well as SSL VPN connectivity to servers in Phoenix.

    Started 04:45 AM Nov 14, 2014

    Ended 06:05 AM Nov 14, 2014

  • 11/09
    Impact
    Severity
    CHI-3 Private Cabinet Switch in 101.5.18.0
    Status : Closed

    Posted at: 11/09/2014 10:21 am
    The cabinet switch in question has been replaced and service to private connectivity has been restored to normal for servers in this cabinet. Thank you!

    Posted at: 11/09/2014 09:31 am
    We are currently experiencing packet loss with private connectivity for servers in cabinet 101.5.18.0. We are actively troubleshooting the issue and are preparing to replace this switch. Clients in this cabinet will experience several periods of disruption while we make this replacement. We will update this posting with further information as it becomes available. We apologize for the inconvenience.

    Started 07:00 AM Nov 09, 2014

    Ended 10:00 AM Nov 09, 2014

  • 11/07
    Impact
    Severity
    Public Cloud Service Interruption
    Status : Closed

    Posted at: 11/07/2014 10:41 am
    We have experienced a brief period of packet loss on the SingleHop public cloud infrastructure. The issue was immediately noticed and resolved within 5 minutes. We are investigating the root cause of this issue, and sincerely apologize for any inconvenience this has caused.

    Started 10:28 AM Nov 07, 2014

    Ended 10:34 AM Nov 07, 2014

  • 10/15
    Impact
    Severity
    CHI2 Public Network Degraded
    Status : Closed

    Posted at: 10/15/2014 04:53 am
    The hardware has been revived, thus restoring the network back to a normal state. We will continue to closely monitor this hardware over the next 24 hours to ensure stability.

    Posted at: 10/15/2014 04:23 am
    We are currently experiencing a degraded state with the CHI2 Public network infrastructure. A line card within one of our core switches has crashed, causing some switch uplinks to shutdown. Because of the redundancy currently in place with this architecture, clients should not be experiencing any impact from this event. We are actively pursuing a case with our hardware vendor to restore these connections. Thank you for your patience.

    Started 03:12 AM Oct 15, 2014

    Ended 04:50 AM Oct 15, 2014

  • 10/08
    Impact
    Severity
    Public Cloud Connectivity Disruption
    Status : Closed

    Posted at: 10/08/2014 05:45 pm
    There were two brief disruptions during this time.

    Posted at: 10/08/2014 05:32 pm
    Public Cloud connectivity was briefly disrupted.

    Started 05:25 PM Oct 08, 2014

    Ended 05:35 PM Oct 08, 2014

  • 10/07
    Impact
    Severity
    Public Cloud Service Interruption
    Status : Closed

    Posted at: 10/07/2014 07:05 pm
    We experienced a network related event with our Public Cloud in the CHI2 Data Center. Clients using this service may have experienced a small period of intermittent loss of connectivity. The problem was identified and corrected within 5 minutes.

    If you continue to experience problems, please contact our support team to troubleshoot further.

    We apologize for the inconvenience this may have caused.

    Started 05:17 PM Oct 07, 2014

    Ended 05:23 PM Oct 07, 2014

  • 10/02
    Impact
    Severity
    CHI3 Public Network Maintenance
    Status : Closed

    Posted at: 10/02/2014 06:13 am
    This maintenance window has ended. We apologize for any inconvenience caused by this.

    Posted at: 10/02/2014 05:02 am
    Full connectivity has been restored and we are verifying right now.
    Please standby for further updates.

    Posted at: 10/02/2014 04:44 am
    This issue is still being worked on at this time. Some customers may have noticed extended service disruption.
    Please standby for further updates.

    Posted at: 10/02/2014 04:18 am
    We identified an issue that is causing an extended connectivity disruption for some customers. We are working on resolving this issue.
    Please standby

    Posted at: 10/02/2014 04:12 am
    This maintenance is still underway. We will post another update here shortly.

    Posted at: 10/02/2014 02:59 am
    We will begin out scheduled public network maintenance for our CHI-3 Data Center. As a reminder, the activities being performed will cause a brief disruption in service for up to 60 seconds. We will update this post as more information becomes available. Thank you!

    Started 03:00 AM Oct 02, 2014

    Ended 05:00 AM Oct 02, 2014

  • 10/02
    Impact
    Severity
    Non-impacting provider maintenance - Phx-1
    Status : Closed

    Posted at: 10/02/2014 03:17 am
    This maintenance window is now closed.

    Posted at: 10/02/2014 02:03 am
    This is a non-impacting scheduled maintenance.
    Our vendor will be performing a configuration change on two of our transit circuits at our PHX-1 DC.

    Started 02:00 AM Oct 02, 2014

    Ended 03:00 AM Oct 02, 2014

  • 09/29
    Impact
    Severity
    SingleHop - Non-Impacting Public Network Emergency Maintenance
    Status : Closed

    Posted at: 09/29/2014 03:47 pm
    This maintenance is now complete.

    Posted at: 09/29/2014 02:04 pm
    We are starting the non impacting maintenance now.

    Started 02:00 PM Sep 29, 2014

    Ended 03:30 PM Sep 29, 2014

  • 09/28
    Impact
    Severity
    CHI3 Non-Impacting Public Network Maintenance
    Status : Closed

    Posted at: 09/29/2014 01:23 am
    All activities have been completed, so the maintenance window will be ended early. We appreciate your patience during this period.

    Posted at: 09/28/2014 11:56 pm
    We have started our scheduled maintenance window on the CHI-3 Public Network Core. No service impacts are expected while the work is performed. We will update this post with further information as it becomes available. Thanks.

    Started 11:00 PM Sep 28, 2014

    Ended 01:24 AM Sep 29, 2014

  • 09/22
    Impact
    Severity
    CHI3 Non-Impacting Public Network Maintenance
    Status : Closed

    Posted at: 09/23/2014 01:44 am
    This maintenance window has now ended. We appreciate your patience.

    Posted at: 09/22/2014 10:03 pm
    Our scheduled public network maintenance has begun. We do not anticipate any service interruptions while this work is being performed. We will update this post as more information becomes available. Thanks.

    Started 10:00 PM Sep 22, 2014

    Ended 01:45 AM Sep 23, 2014

  • 09/18
    Impact
    Severity
    SingleHop - Non Impacting Public Network Maintenance
    Status : Closed

    Posted at: 09/18/2014 07:26 am
    This maintenance has been completed. There was no impact.

    Posted at: 09/18/2014 05:33 am
    The maintenance window remains open at this time, pending confirmation from our upstream providers.

    Posted at: 09/18/2014 02:51 am
    We are starting this scheduled maintenance now. No impact is expected during this maintenance window.

    Started 02:50 AM Sep 18, 2014

    Ended 07:25 AM Sep 18, 2014

  • 09/17
    Impact
    Severity
    Scheduled Private Network Maintenance
    Status : Closed

    Posted at: 09/17/2014 10:37 pm
    This maintenance window has ended early. We thank you for your patience.

    Posted at: 09/17/2014 10:01 pm
    We have started our scheduled maintenance window on our Private Network core infrastructure. No impacts are expected during this activity. We will update this post as more information becomes available. Thanks.

    Started 10:00 PM Sep 17, 2014

    Ended 08:30 PM Sep 17, 2014

  • 09/17
    Impact
    Severity
    SingleHop Spam Filtering outage
    Status : Closed

    Posted at: 09/17/2014 11:44 am
    Issue has been resolved

    Posted at: 09/17/2014 11:32 am
    Clients with the spam filtering service will be expecting a delay in mail delivery currently as we are working on fixing an issue with the service. Mail is still being retrieved fine to the spam filtering queue and will be delivered to inboxes once the issue is resolved.

    Started 11:00 AM Sep 17, 2014

    Ended 11:44 AM Sep 17, 2014

  • 09/17
    Impact
    Severity
    SingleHop - Non-Impacting Public Network Emergency Maintenance
    Status : Closed

    Posted at: 09/17/2014 05:38 am
    This maintenance is now complete.

    Posted at: 09/17/2014 03:13 am
    We are starting the non-impacting maintenance now.

    Started 03:00 AM Sep 17, 2014

    Ended 05:30 AM Sep 17, 2014

  • 09/16
    Impact
    Severity
    SingleHop - Non-Impacting Public Network Emergency Maintenance
    Status : Closed

    Posted at: 09/16/2014 04:43 am
    This maintenance is now complete.

    Posted at: 09/16/2014 03:07 am
    Hello,

    We are starting the non impacting public maintenance now.

    Started 03:05 AM Sep 16, 2014

    Ended 04:30 AM Sep 16, 2014

  • 09/12
    Impact
    Severity
    Public Switch Outage - asw1.c09.r10.s101.chi03.singlehop.net
    Status : Closed

    Posted at: 09/12/2014 05:58 pm
    The switch is back online.

    If you need further technical assistance, please open a trouble ticket.

    Thank you!

    Posted at: 09/12/2014 05:48 pm
    Greetings,

    The SingleHop Network Operations team is currently in the process of repairing a public network outage in CHI-3 in cab 9.10 (asw1.c09.r10.s101.chi03.singlehop.net). Please stand by for further updates.

    Started 05:20 PM Sep 12, 2014

    Ended 05:50 PM Sep 12, 2014

  • 09/12
    Impact
    Severity
    Private Network Scheduled Maintenance
    Status : Closed

    Posted at: 09/12/2014 06:11 am
    This maintenance has been completed. Thank you for your patience during this period.

    Posted at: 09/12/2014 03:07 am
    We are starting the scheduled maintenance for our Private Network Core upgrades for the CHI-3 Data Center. There will be a brief outage on the private network while this work is being conducted. We will update this post with any further information as it becomes available. Thank you.

    Started 03:00 AM Sep 12, 2014

    Ended 06:00 AM Sep 12, 2014

  • 09/11
    Impact
    Severity
    APC failure in CHI-3 cr130.101.3.4
    Status : Closed

    Posted at: 09/11/2014 12:16 pm
    Greetings,

    A failure of an APC Power Distribution unit was detected in Cab CR130.101.3.4 at CHI-3. The APC has been replaced and power has been restored to servers/devices that were powered by this APC.

    If you need further assistance, please let us know. Thank you.

    Started 11:55 AM Sep 11, 2014

    Ended 11:58 AM Sep 11, 2014

  • 09/11
    Impact
    Severity
    Private Network Scheduled Maintenance
    Status : Closed

    Posted at: 09/11/2014 01:09 pm
    This maintenance has now been completed. We appreciate your patience during this period.

    Posted at: 09/11/2014 06:44 am
    We are continuing to perform this work. The activities are being performed on a redundant system, so no impact to network services are expected. We will continue to update this post. Thank you.

    Posted at: 09/11/2014 05:47 am
    At this time the scheduled maintenance is still ongoing, however there are no further impacts expected to the network. We will continue to update this post as more information becomes available.

    Posted at: 09/11/2014 03:08 am
    We are starting the scheduled maintenance for our Private Network Core upgrades for the CHI-3 Data Center. There will be a brief outage on the private network while this work is being conducted. We will update this post with any further information as it becomes available. Thank you.

    Started 03:00 AM Sep 11, 2014

    Ended 01:00 PM Sep 11, 2014

  • 09/05
    Impact
    Severity
    SingleHop - Non-Impacting Public Vendor Maintenance
    Status : Closed

    Posted at: 09/05/2014 03:03 am
    This maintenance has been completed.

    Posted at: 09/05/2014 02:29 am
    We are now conducting verification and monitoring tasks before concluding this maintenance.

    Posted at: 09/05/2014 02:01 am
    This status update is being opened to coincide with the Non-Impacting Public Vendor Maintenance
    that starts on 9/5/2014 at 2:00 AM CDT. The window will remain open until 3:00 AM or until the maintenance is completed.

    Started 02:00 AM Sep 05, 2014

    Ended 03:00 AM Sep 05, 2014

  • 09/04
    Impact
    Severity
    Connectivity issues to CHI-3 datacenter.
    Status : Closed

    Posted at: 09/04/2014 06:29 pm
    The issue has been resolved

    Posted at: 09/04/2014 06:26 pm
    There was a brief but large attack against a customer. The attack was handled by our Network Operators and affected a few customers for about 15 minutes, lasting from 6.00pm CDT to about 6.15pm CDT

    Started 06:00 PM Sep 04, 2014

    Ended 06:15 PM Sep 04, 2014

  • 09/02
    Impact
    Severity
    SingleHop - Non-Impacting Public Network Maintenance
    Status : Closed

    Posted at: 09/03/2014 12:41 am
    The maintenance is now complete.

    Posted at: 09/02/2014 09:23 pm
    We are starting the non-impacting maintenance now.

    Started 09:20 PM Sep 02, 2014

    Ended 12:30 AM Sep 03, 2014

  • 09/02
    Impact
    Severity
    Provider Outage in Chicago Market
    Status : Closed

    Posted at: 09/02/2014 10:00 am
    One of our providers just experienced a hardware failure in the Chicago market. This caused traffic to fail over to our redundant paths. This transition could have a caused periodic packet loss as routing tables were updated. We are currently working with the provider to create a detailed report of the outage. Once an RFO has been generated it will be provided on request.

    Started 09:20 AM Sep 02, 2014

    Ended 09:32 AM Sep 02, 2014

  • 09/01
    Impact
    Severity
    CHI-3 Private Network Event
    Status : Closed

    Posted at: 09/01/2014 01:12 pm
    This has been resolved. If you have further issues as a result of this event or need more information, please let us know by opening a new ticket.

    Thank you for your patience in this matter.

    Posted at: 09/01/2014 01:00 pm
    Greetings,

    The SingleHop Network Operations team is currently investigating a network event in progress on the CHI-3 Private Network. We are actively troubleshooting this issue and working to reach a resolution as soon as possible.

    Please stand by for further updates. Thank you.

    Started 12:40 PM Sep 01, 2014

    Ended 01:06 PM Sep 01, 2014

  • 08/31
    Impact
    Severity
    Several reports of lack of connectivity to CHI-2 datacenter.
    Status : Closed

    Posted at: 08/31/2014 04:48 pm
    There was a brief but large attack against a customer. The attack traffic ended on its own before we were able to react.

    Started 04:30 PM Aug 31, 2014

    Ended

  • 08/30
    Impact
    Severity
    HSRP system latency and possible packet loss - Chi-2
    Status : Closed

    Posted at: 08/30/2014 06:16 am
    This issue has been resolved. The impact was limited to a specific customer whom we are in touch with. No other customers were impacted by this issue.
    We are now closing this status report.
    Thank you.

    Posted at: 08/30/2014 05:04 am
    Our team is still investigating this issue. Please standby for further updates.
    Thank you

    Posted at: 08/30/2014 04:45 am
    We are investigating a possible packet loss issue to some servers on this system.
    Please standby for further updates.

    Started 04:35 AM Aug 30, 2014

    Ended 06:13 AM Aug 30, 2014

  • 08/28
    Impact
    Severity
    SingleHop - Non-Impacting Public Network Maintenance
    Status : Closed

    Posted at: 08/29/2014 02:06 am
    This maintenance is now complete.

    Posted at: 08/28/2014 09:38 pm
    Starting the non-impacting core maintenance now.

    Started 11:30 AM Aug 28, 2014

    Ended 02:00 AM Aug 29, 2014

  • 08/28
    Impact
    Severity
    aswg-eg113.ord03 - switch down
    Status : Closed

    Posted at: 08/28/2014 12:40 pm
    Greetings,

    The switch replacement is complete and servers connectivity is now restored.

    If you need further technical assistance as a result of this event, please open a support ticket.

    Thank you for your patience in this process.

    Posted at: 08/28/2014 12:15 pm
    The replacement device is configured and is being installed.

    As soon as more information becomes available we will let you know.

    Posted at: 08/28/2014 12:10 pm
    We are still working on replacing the switch. Please stand by for further updates.

    Posted at: 08/28/2014 11:41 am
    We are still working on replacing the switch. Please stand by for further updates.

    Posted at: 08/28/2014 11:23 am
    SingleHop Network Operations has been alerted of a hardware failure on switch aswg-eg113.ord03. We are investigating and preparing a replacement device at this time.

    Please stand by for further updates. Thank you.

    Started 11:10 AM Aug 28, 2014

    Ended 12:30 PM Aug 28, 2014

  • 08/28
    Impact
    Severity
    Private network maintenance - cabinet eg.107
    Status : Closed

    Posted at: 08/28/2014 01:41 am
    This maintenance has been completed and connectivity verified.
    This status report is now closed.
    Thank you.

    Posted at: 08/28/2014 01:11 am
    This is a scheduled maintenance work on cabinet eg.107
    We are anticipating a brief disruption to private network traffic in this cab. The impact is limited only to the private network; no impact to the public network traffic.

    Started 01:10 AM Aug 28, 2014

    Ended 01:40 AM Aug 28, 2014

  • 08/27
    Impact
    Severity
    Non-impacting Core Maintenance
    Status : Closed

    Posted at: 08/27/2014 11:07 pm
    This maintenance is now complete.

    Posted at: 08/27/2014 09:32 pm
    We are starting the non-impacting maintenance now.

    Started 09:30 PM Aug 27, 2014

    Ended 12:07 AM Aug 28, 2014

  • 08/26
    Impact
    Severity
    SSLVPN Maintenance
    Status : Closed

    Posted at: 08/26/2014 05:36 pm
    This maintenance is now complete.

    Posted at: 08/26/2014 05:21 pm
    The maintenance is continuing.

    Posted at: 08/26/2014 05:02 pm
    The maintenance is continuing. We will provide additional details as they are available.

    Posted at: 08/26/2014 01:02 pm
    We are starting the sslvpn maintenance now. It will be offline for upto 4 hours.

    Started 01:00 PM Aug 26, 2014

    Ended 05:35 PM Aug 26, 2014

  • 08/23
    Impact
    Severity
    Cab switch outage c22.r06.s101.chi03
    Status : Closed

    Posted at: 08/23/2014 01:52 am
    Connectivity has been verified on this switch.
    We are now closing this status report.

    Posted at: 08/23/2014 12:58 am
    The issue has been resolved. We are now verifying connectivity before closing this status report.
    Please standby

    Posted at: 08/23/2014 12:36 am
    We have and active alarm on network switch in cabinet c22.r06.s101.chi03
    We are currently investigating this issue.
    Please standby.

    Started 12:08 AM Aug 23, 2014

    Ended 01:02 AM Aug 23, 2014

  • 08/16
    Impact
    Severity
    Scheduled Phone System Maintenance
    Status : Closed

    Posted at: 08/17/2014 03:09 am
    This maintenance has been completed.

    Posted at: 08/17/2014 12:23 am
    We have begun maintenance on SingleHop's telephone systems. During this time, calls being made to our toll-free or local numbers may experience periods of unavailability. We encourage our clients to reach our support staff through our LEAP3 portal located at https://leap3.singlehop.com

    Thank you for your patience.

    Started 12:00 AM Aug 16, 2014

    Ended 03:00 AM Aug 17, 2014

  • 08/15
    Impact
    Severity
    Leap Ticket Replies
    Status : Closed

    Posted at: 08/15/2014 06:22 pm
    This issue should now be corrected.

    Posted at: 08/15/2014 04:15 pm
    We have received reports that some clients are receiving blank replies from us in tickets. We are aware of the situation and are working on resolving the issue.

    Started 04:00 PM Aug 15, 2014

    Ended

  • 08/14
    Impact
    Severity
    Leap 3/Dropzone Maintenance
    Status : Closed

    Posted at: 08/14/2014 09:28 pm
    This maintenance will be performed to conduct upgrades to our Leap 3 and Dropzone infrastructure for improved performance and service delivery. During this maintenance you may experience being logged out from the Leap 3 system. This is expected and after logging in Leap 3 should function normally again.

    With any system change there is a possibility of an unforeseen situation to arise, though every effort is being made to ensure this will be a non impacting maintenance.

    Started 09:00 PM Aug 14, 2014

    Ended 09:30 PM Aug 14, 2014

  • 08/13
    Impact
    Severity
    SingleHop - Non-Impacting Public Network Maintenance
    Status : Closed

    Posted at: 08/13/2014 10:40 pm
    The maintenance is now complete.

    Posted at: 08/13/2014 09:21 pm
    Starting the non impacting maintenance now.

    Started 09:20 PM Aug 13, 2014

    Ended 10:40 PM Aug 13, 2014

  • 08/09
    Impact
    Severity
    Scheduled non-impacting maintenance - cab ez.180
    Status : Closed

    Posted at: 08/09/2014 01:23 am
    The maintenance work has been completed. There was no impact as a result of this maintenance.
    Thank you.

    Posted at: 08/09/2014 12:24 am
    We are starting this maintenance now. This maintenance is scheduled in order to improve the cab switch capacity.

    Started 12:20 AM Aug 09, 2014

    Ended 01:22 AM Aug 09, 2014

  • 08/01
    Impact
    Severity
    Non-impacting Core Maintenance
    Status : Closed

    Posted at: 08/01/2014 09:13 pm
    This maintenance is now complete.

    Posted at: 08/01/2014 08:01 pm
    We are starting the non impacting maintenance now.

    Started 08:00 PM Aug 01, 2014

    Ended 09:13 PM Aug 01, 2014

  • 08/01
    Impact
    Severity
    Non-impacting Core Maintenance
    Status : Closed

    Posted at: 08/01/2014 02:23 am
    All the maintenance tasks have been completed. There was no impact to any service as a result of this maintenance.
    We are now closing this status report.
    Thank you.

    Posted at: 07/31/2014 11:24 pm
    We are starting the maintenance now.

    Started 11:24 AM Aug 01, 2014

    Ended 02:14 AM Aug 01, 2014

  • 07/31
    Impact
    Severity
    PHX-1 Emergency Network Maintenance
    Status : Closed

    Posted at: 07/31/2014 10:29 pm
    The maintenance is completed and we are now closing this window.

    If you have further technical assistance requirements, please open a ticket.

    Thank you.

    Posted at: 07/31/2014 10:11 pm
    Greetings,

    We are now beginning tonight's maintenance.

    Please stand by for further updates.

    Started 10:10 PM Jul 31, 2014

    Ended 10:29 PM Jul 31, 2014

  • 07/29
    Impact
    Severity
    Chi-2 Core Maintenance
    Status : Closed

    Posted at: 07/30/2014 02:38 am
    The maintenance tasks have been completed. Newly added connections have been verified.
    This status report is now closed.
    Thank you.

    Posted at: 07/30/2014 12:45 am
    The physical connections have been made and the links are being verified.

    Posted at: 07/29/2014 11:48 pm
    During this maintenance window we will be adding new connections to the Chi-2 core routers. No impact is expected.

    Started 11:48 PM Jul 29, 2014

    Ended 02:25 AM Jul 30, 2014

  • 07/29
    Impact
    Severity
    Non-Impacting Public Network Maintenance
    Status : Closed

    Posted at: 07/29/2014 04:40 am
    The maintenance window has ended. We appreciate your patience during this process.

    Posted at: 07/29/2014 12:04 am
    We are now conducting our scheduled emergency maintenance activity for our CHI-2 data center. No impact is expected during this window. We will update this post with any information as it becomes available. Thanks.

    Started 12:00 AM Jul 29, 2014

    Ended 04:31 AM Jul 29, 2014

  • 07/25
    Impact
    Severity
    Cab switch aswg-ed113-2.ord03
    Status : Closed

    Posted at: 07/25/2014 03:33 am
    Full connectivity has been restored and verified. Please open a Technical Support ticket if you are experiencing any issues.
    This status report is now closed.
    Thank you.

    Posted at: 07/25/2014 03:16 am
    We have an active alarm about degraded performance on this cab switch. We are troubleshooting now, please standby.

    Started 03:13 AM Jul 25, 2014

    Ended 03:22 AM Jul 25, 2014

  • 07/21
    Impact
    Severity
    GTT connectivity to Verizon Florida market customers
    Status : Closed

    Posted at: 07/22/2014 09:59 am
    Our upstream provider has not yet received official word from Verizon confirming
    that the issue has been resolved, however they state that they are now seeing
    service restoration with pinging Verizon IPs in Florida from the NLayer network.

    We are requesting an official RFO for this issue from NLayer however since this issue was not on our network it may be faster for customers on the Verizon network to request the RFO from Verizon's support department directly.

    If you need further technical assistance or wish to receive an RFO for this event, please open a ticket.

    Thank you for your continued patience in this matter.

    Posted at: 07/21/2014 08:24 pm
    These intermittent issues continue to affect connections to Verizon in the Miami area. Verizon has not yet provided an update regarding the issue nor an ETA for a fix. Thank you for your patience in this matter.

    Posted at: 07/21/2014 03:07 pm
    Verizon NOC is currently investigating the issue on their side of the network. We have requested further updates and an ETA regarding this issue and will provide more information as it becomes available. Please stand by.

    Posted at: 07/21/2014 01:37 pm
    SingleHop Network Operations is currently investigating an issue with connectivity between the NLayer/GTT network in Chicago (CHI-2 Datacenter) and the Verizon network in and around Miami Florida. This seems to be intermittent issues affecting only some networks or IPs on the SingleHop network. This issue is being actively reviewed by both the SingleHop network operations team and the GTT Network Engineering team.

    We will update this status as more information becomes available.

    Started 11:00 AM Jul 21, 2014

    Ended 08:54 AM Jul 22, 2014

  • 07/11
    Impact
    Severity
    BGP Routing Table maintnenace
    Status : Closed

    Posted at: 07/11/2014 08:20 pm
    The window is now closed. Thank you for your patience during this process.

    Posted at: 07/11/2014 08:13 pm
    The maintenance has been completed and we are performing final checks.

    We will be closing the window shortly.

    Posted at: 07/11/2014 07:59 pm
    We will be beginning our routing table maintenance for AMS-1 in a few minutes. Below are the details from the original announcement.

    Event Date(s): Friday, July 11th, 2014 from 8:00PM until 9:00PM CDT (UTC-5)
    Event Location: AMS-1
    Impact: No impact to public/private network connectivity is expected during this maintenance.

    Event: We are performing routing table maintenance during this window, the duration of which is 1 hour.

    With any network change there is a possibility of an unforeseen situation to arise, though every effort is being made to ensure this will be a non-impactingmaintenance.

    Started 08:00 PM Jul 11, 2014

    Ended 08:20 PM Jul 11, 2014

  • 07/11
    Impact
    Severity
    PHX-1 Non-Impacting Public Network Maintenance
    Status : Closed

    Posted at: 07/10/2014 11:57 pm
    We will begin our scheduled maintenance window for our PHX-1 facility. No impact is expected to services while the tasks are being performed. We will update this post with any additional information as it becomes available. Thanks!

    Started 12:00 AM Jul 11, 2014

    Ended 02:00 AM Jul 11, 2014

  • 07/10
    Impact
    Severity
    Private Network Emergency Maintenance
    Status : Closed

    Posted at: 07/10/2014 11:05 pm
    Our maintenance has been completed and the window has now ended. We appreciate your patience during this upgrade.

    Posted at: 07/10/2014 10:02 pm
    We have begun our activities for the scheduled maintenance window our our private aggregation in CHI-2. An outage of up to 15 minutes for private network access will occur as part of this maintenance. Impact to public network connectivity is not expected. We will update this post with more information as it becomes available. Thank you.

    Started 10:00 PM Jul 10, 2014

    Ended 11:00 PM Jul 10, 2014

  • 07/10
    Impact
    Severity
    Non-impacting Private Core Maintenance
    Status : Closed

    Posted at: 07/10/2014 09:09 pm
    We are starting the maintenance now.

    Started 09:00 PM Jul 10, 2014

    Ended 12:00 AM Jul 11, 2014

  • 07/09
    Impact
    Severity
    Brief Resolver Outage
    Status : Closed

    Posted at: 07/09/2014 04:03 pm
    Greetings,

    There was a brief outage of our DNS Resolver cluster, which would have impacted any device using the following resolvers:

    216.104.43.102
    216.104.43.86

    This was quickly resolved by our infrastructure and network staff. You should be able to verify that DNS resolution is now working. If you continue to have issues, please open a support ticket. Thanks.

    Started 03:24 PM Jul 09, 2014

    Ended 03:42 PM Jul 09, 2014

  • 07/03
    Impact
    Severity
    Private Network Emergency Maintenance
    Status : Closed

    Posted at: 07/03/2014 11:27 pm
    We have completed our maintenance activity and we are closing this window. We appreciate your patience during this period.

    Posted at: 07/03/2014 10:00 pm
    We have begun our scheduled maintenance window for our private aggregation switch within the CHI-2 Data Center. During this maintenance, an outage of up to 15 minutes is expected on the private network. Public connectivity will not be impacted during this work. We will continue to post updates as they become available. Thanks.

    Started 10:00 PM Jul 03, 2014

    Ended 11:26 PM Jul 03, 2014

  • 07/03
    Impact
    Severity
    CHI-2 Private Core Maintenance
    Status : Closed

    Posted at: 07/03/2014 11:48 pm
    This maintenance is now complete.

    Posted at: 07/03/2014 09:04 pm
    The non-impacting private core maintenance is starting now.

    Started 09:00 AM Jul 03, 2014

    Ended 11:45 AM Jul 04, 2014

  • 07/02
    Impact
    Severity
    Nlayer Route Flap in CHI2
    Status : Closed

    Posted at: 07/02/2014 11:54 am
    We noticed packet loss for some traffic traveling over the Nlayer network in CHI. This was caused by the reboot of a peering router on the Nlayer network. We are seeking additional information from the vendor and it will be provided on request.

    Started 04:11 AM Jul 02, 2014

    Ended 04:37 AM Jul 02, 2014

  • 06/30
    Impact
    Severity
    SingleHop - Non-Impacting Public Network Maintenance
    Status : Closed

    Posted at: 06/30/2014 06:03 am
    The maintenance window has ended. No unexpected impacts to service occurred during this event. Thank you for your patience.

    Posted at: 06/30/2014 02:15 am
    We have begun our scheduled maintenance window for our PHX1 data center. No impact is expected during these activities. We will update this post with further information as it becomes available. Thanks.

    Started 02:00 AM Jun 30, 2014

    Ended 06:00 AM Jun 30, 2014

  • 06/26
    Impact
    Severity
    Non-Impacting Public Network Maintenance
    Status : Closed

    Posted at: 06/26/2014 04:21 am
    The maintenance has ended. No unexpected events occurred. Thank you for your patience.

    Posted at: 06/26/2014 02:10 am
    We have begun our two hour maintenance event for the PHX-1 Data Center. There is no impact expected to connectivity during this period. We will continue to update this post with any further details. Thanks.

    Started 02:00 AM Jun 26, 2014

    Ended 04:00 AM Jun 26, 2014

  • 06/25
    Impact
    Severity
    Non-Impacting Public Network Maintenance
    Status : Closed

    Posted at: 06/26/2014 12:12 am
    Scheduled maintenance tasks have been completed. There was no impact or any disruption during this maintenance window.
    This status report is now closed.
    Thank you.

    Posted at: 06/26/2014 12:03 am
    Scheduled maintenance tasks are underway. Please standby for further updates.

    Posted at: 06/25/2014 11:26 pm
    Scheduled maintenance is still underway. Please standby for further updates.

    Posted at: 06/25/2014 10:52 pm
    Scheduled maintenance tasks continue at this time. Standby for further updates.

    Posted at: 06/25/2014 10:41 pm
    We are still performing maintenance tasks at this time. We will update this ticket when we have more information available. Please stand by. Thank you.

    Posted at: 06/25/2014 09:59 pm
    This status update is being opened to coincide with the Non-Impacting public network maintenance that starts on 6/25/2014 at 10:00pm CDT. The window will remain open until 1AM or until the maintenance is completed.

    Started 10:00 PM Jun 25, 2014

    Ended 12:10 AM Jun 26, 2014

  • 06/20
    Impact
    Severity
    SnapShield Firewall 1 - Chi-2
    Status : Closed

    Posted at: 06/20/2014 11:59 pm
    Traffic through the Snapshield Firewall has been verified.
    Please open a Technical Support ticket if you are still experiencing any latency or connectivity issues.
    This status report is now closed.
    Thank you.

    Posted at: 06/20/2014 11:47 pm
    The Snapshield Firewall has been stabilized. We are still verifying connectivity at this time before closing this status report.
    Please standby.

    Posted at: 06/20/2014 11:25 pm
    We have an active alarm that the Snapshield Firewall devices may be experiencing degraded performance and some latency.
    We are still investigating.
    Please standby

    Started 11:02 PM Jun 20, 2014

    Ended 11:44 PM Jun 20, 2014

  • 06/20
    Impact
    Severity
    CHI-2 and PHX-1 Non-Impacting Public Network Maintenance Beginning
    Status : Closed

    Posted at: 06/20/2014 08:36 pm
    The maintenance has been completed. As such, we are closing the maintenance window and this event page.

    Posted at: 06/20/2014 08:04 pm
    We are beginning our non-impacting public network maintenance.

    Please stand by for further updates.

    Started 08:00 PM Jun 20, 2014

    Ended 08:30 PM Jun 20, 2014

  • 06/20
    Impact
    Severity
    Emergency Network Impacting Maintenance
    Status : Closed

    Posted at: 06/20/2014 04:45 am
    This maintenance window is ending early, as all planned activities have been completed. Thank you for your continued patience.

    Posted at: 06/20/2014 02:29 am
    We are conducting our scheduled emergency maintenance for our public cloud platform in CHI-2. There is a possibility of a network outage of up to 5 minutes to this service. This posting will be updated with further information as it becomes available. Thank you.

    Started 02:00 AM Jun 20, 2014

    Ended 04:45 AM Jun 20, 2014

  • 06/19
    Impact
    Severity
    Private Network Emergency Maintenance
    Status : Closed

    Posted at: 06/19/2014 11:15 pm
    The maintenance window has ended early, as all work has been completed. We appreciate your patience during this activity.

    Posted at: 06/19/2014 10:02 pm
    We have begun our scheduled emergency maintenance on the private network in CHI-2. A 10 minute outage to private connectivity is expected during this period. We will continue to update this posting with any further information as it becomes available. Thanks!

    Started 10:00 PM Jun 19, 2014

    Ended 11:20 PM Jun 19, 2014

  • 06/19
    Impact
    Severity
    SSLVPN Upgrade Maintenance
    Status : Closed

    Posted at: 06/19/2014 08:52 pm
    Greetings,

    The SSLVPN upgrade is completed. We are now closing this maintenance window.

    Posted at: 06/19/2014 08:03 pm
    We are beginning our scheduled maintenance window for the SingleHop SSLVPN Upgrade.

    Please stand by for further updates.

    Started 08:01 PM Jun 19, 2014

    Ended 08:53 PM Jun 19, 2014

  • 06/16
    Impact
    Severity
    SingleHop - Non-Impacting Public Network Maintenance
    Status : Closed

    Posted at: 06/16/2014 06:42 pm
    This maintenance is now complete.

    Posted at: 06/16/2014 06:00 pm
    The maintenance continues.

    Posted at: 06/16/2014 05:21 pm
    We are starting the maintenance now. No impact is expected.

    Started 05:00 PM Jun 16, 2014

    Ended 06:30 PM Jun 16, 2014

  • 06/12
    Impact
    Severity
    Planned Maintenance on Private Network
    Status : Closed

    Posted at: 06/13/2014 12:02 am
    The maintenance has ended, and services on the private network at Chicago-2 have returned to normal operation. We appreciate your patience during this period.

    Posted at: 06/12/2014 10:07 pm
    There is a planned maintenance from 10PM-12AM CDT for part of the private network at Chicago-2. There will be brief loss of connectivity during the maintenance window.

    Started 10:00 PM Jun 12, 2014

    Ended 11:59 PM Jun 12, 2014

  • 06/11
    Impact
    Severity
    Provider Packet Loss
    Status : Closed

    Posted at: 06/11/2014 12:23 pm
    All services should be restored at this time. We are continuing to work our vendor to provide an RFO.

    Posted at: 06/11/2014 12:00 pm
    Greetings,

    Our Network Administrators have noticed significant packet loss from one of our upstream providers, AboveNet. This may cause issues for servers that are currently utilizing circuits with this provider. Our network administrators are working with them to resolve the issue.

    Started 11:50 AM Jun 11, 2014

    Ended 11:57 AM Jun 11, 2014

  • 06/10
    Impact
    Severity
    Non-impacting AMS Core Maintenance
    Status : Closed

    Posted at: 06/10/2014 09:18 pm
    The maintenance window has now ended. All work is now complete.

    Posted at: 06/10/2014 08:02 pm
    The maintenance is continuing.

    Posted at: 06/10/2014 07:37 pm
    The maintenance is still continuing.

    Posted at: 06/10/2014 06:40 pm
    The maintenance is still continuing.

    Posted at: 06/10/2014 05:29 pm
    We are starting our network maintenance in AMS now. This maintenance is expected to be non-impacting.

    Started 05:28 AM Jun 10, 2014

    Ended 09:00 AM Jun 10, 2014

  • 06/04
    Impact
    Severity
    Network Provider Issue
    Status : Closed

    Posted at: 06/04/2014 10:28 pm
    A brief period of high latency was noticed on a upstream provider's network. All services are restored at this time and we are working with the provider to identify the root cause.

    Started 08:50 PM Jun 04, 2014

    Ended 08:55 PM Jun 04, 2014

  • 05/15
    Impact
    Severity
    Provider Transport Maintenance
    Status : Closed

    Posted at: 05/15/2014 11:51 pm
    The maintenance is now complete.

    Posted at: 05/15/2014 09:57 pm
    We will be starting this maintenance promptly at 10 pm CDT.

    Started 10:00 AM May 15, 2014

    Ended 11:52 AM May 16, 2014

  • 05/14
    Impact
    Severity
    Private Aggregation Maintenance
    Status : Closed

    Posted at: 05/15/2014 12:21 am
    This maintenance is now complete.

    Posted at: 05/14/2014 10:46 pm
    We are taking down the switch at this time.

    Posted at: 05/14/2014 10:04 pm
    We are starting the maintenance now. Once the preconfiguration work is done, there will be another update before we take down services.

    Started 10:00 AM May 14, 2014

    Ended 12:00 PM May 15, 2014

  • 05/14
    Impact
    Severity
    Cab switch scheduled maintenance - EW.176 Chi-2
    Status : Closed

    Posted at: 05/14/2014 02:54 am
    Connectivity has been verified and all maintenance tasks has been completed.
    This status report is now closed.

    Posted at: 05/14/2014 02:45 am
    The hardware has been replaced and the switch is now online.
    We are verifying connectivity at this time.

    Posted at: 05/14/2014 02:27 am
    We are now starting the hardware replacement.
    Please standby for further updates.

    Posted at: 05/14/2014 02:13 am
    This is a scheduled maintenance to replace faulty cabinet switch hardware.
    During this maintenance the switch hardware will be replaced.
    We will provide an update before the switch is taken offline.
    Please standby

    Started 02:00 AM May 14, 2014

    Ended 02:55 AM May 14, 2014

  • 04/02
    Impact
    Severity
    Non-impacting Core Maintenance
    Status : Closed

    Posted at: 04/02/2014 05:59 am
    The maintenance is now complete.

    Posted at: 04/02/2014 05:43 am
    The maintenance is continuing.

    Posted at: 04/02/2014 05:30 am
    The maintenance is continuing.

    Posted at: 04/02/2014 05:04 am
    We are starting the maintenance now.

    Started 05:00 AM Apr 02, 2014

    Ended 06:00 AM Apr 02, 2014

  • 03/29
    Impact
    Severity
    Network impacting event - Chi-2
    Status : Closed

    Posted at: 03/29/2014 04:39 pm
    At this time, all network operations do appear to be functioning properly at CHI-2. We can say, now, that this was not caused by a capacity issue or hardware failure. Our network operations technicians are still looking for the root cause, and to determine the true scope of the issue.

    Thank you for your continued patience.

    Posted at: 03/29/2014 04:11 pm
    Our network operations technicians are currently looking into an issue regarding network functionality at our CHI-2 facility. The investigation of the cause is ongoing, and we will be updating this post as soon as we have more information.

    Started 03:56 PM Mar 29, 2014

    Ended 04:14 PM Mar 29, 2014

  • 03/18
    Impact
    Severity
    Billing system upgrade
    Status : Closed

    Posted at: 03/18/2014 12:02 pm
    As of 6am CDT the ability to purchase physical servers has been restored.

    Posted at: 03/18/2014 02:43 am
    The billing system upgrade is now complete and everything is working without issue, except for the ability to purchase new physical servers which should be working again by 6AM CDT.

    Posted at: 03/17/2014 11:59 pm
    This maintenance is necessary to perform upgrades on our billing system. During this maintenance we will be upgrading to a newer version of our billing and client relationship management system.

    This maintenance will interrupt interacting with the the following systems:
    Leap
    Leap3
    Dropzone API Calls
    Billing

    If you have any questions or concerns please contact our Support Operations Center at noc@singlehop.com or call us at 1-866-817-2811x2

    Started 12:00 AM Mar 18, 2014

    Ended 12:44 AM Mar 18, 2014

  • 03/10
    Impact
    Severity
    Phonesystem Outage
    Status : Closed

    Posted at: 03/10/2014 02:58 pm
    Greetings,

    The phonesystem is now back online.

    Posted at: 03/10/2014 02:30 pm
    Greetings,

    Currently, we are experiencing an issue with our Phonesystem that is affecting the ability of our sales and support staff to take incoming phone calls. Our network operations team is currently investigating the issue. If you have a support request, please use the ticket system as it is still online. Thanks.

    Started 02:30 PM Mar 10, 2014

    Ended 03:00 PM Mar 10, 2014

  • 02/18
    Impact
    Severity
    Non-impacting CR3 Aggregation Maintenance
    Status : Closed

    Posted at: 02/18/2014 11:18 am
    All work has been completed at this time.

    Posted at: 02/18/2014 10:04 am
    Hello, The maintenance is continuing. No impact is expected.

    Posted at: 02/18/2014 08:44 am
    Hello, The maintenance is continuing. No impact is expected.

    Posted at: 02/18/2014 08:00 am
    Hello, The maintenance is continuing. No impact is expected.

    Posted at: 02/18/2014 07:01 am
    Hello,

    The maintenance is continuing. There was a slight outage for the first switch but we have found the issue and it will be resolved for the additional devices.

    Posted at: 02/18/2014 06:09 am
    Hello,

    We are starting this maintenance now, please standby.

    Started 06:08 AM Feb 18, 2014

    Ended 09:00 AM Feb 18, 2014

  • 02/18
    Impact
    Severity
    CHI-2 Network outage
    Status : Closed

    Posted at: 02/18/2014 06:54 pm
    The official RFO for this event has been uploaded here:

    http://adrepo.singlehop.com/rfo/02182014.pdf

    Please contact our support team with any questions.

    Posted at: 02/18/2014 04:23 am


    Started 12:47 AM Feb 18, 2014

    Ended 01:25 AM Feb 18, 2014

  • 02/14
    Impact
    Severity
    Non-impacting Core Maintenance
    Status : Closed

    Posted at: 02/14/2014 06:03 am
    The maintenance is now complete.

    Posted at: 02/14/2014 05:31 am
    The maintenance is continuing, please standby.

    Posted at: 02/14/2014 05:04 am
    We are starting the maintenance at our CHI-2 data center now.

    Started 05:00 AM Feb 14, 2014

    Ended 06:00 AM Feb 14, 2014

  • 02/13
    Impact
    Severity
    Inbound Denial of Service Attack affecting CHI-1 Datacenter
    Status : Closed

    Posted at: 02/13/2014 04:32 pm
    Greetings,

    The SingleHop Network Operations Center has detected a large Denial of Service (DoS) attack in the CHI-1 Datacenter that may have impacted network services. Network Engineering staff began working on the issue immediately, and have isolated and removed the affected target. Network Engineering staff continue to monitor the situation closely for any further issues.

    Posted at: 02/13/2014 04:29 pm


    Started 03:50 PM Feb 13, 2014

    Ended 04:01 PM Feb 13, 2014

  • 02/09
    Impact
    Severity
    Transport Link - Chi-1/Chi-2
    Status : Closed

    Posted at: 02/09/2014 05:03 pm
    Our link provider has corrected the issue and services are returning to normal.

    Posted at: 02/09/2014 04:59 pm
    At 3:46PM the transport link between Chi-1 and Chi-2 went down. All systems remain publicly accessible. However, this issue is affecting LEAP logins and private network connections between the two data centers.

    Started 03:46 PM Feb 09, 2014

    Ended 04:45 PM Feb 09, 2014

  • 02/07
    Impact
    Severity
    DNS Resolvers - Chi-2
    Status : Closed

    Posted at: 02/07/2014 01:14 am
    This maintenance has been completed. All DNS resolvers are verified online and responding properly.

    Thank you.

    Posted at: 02/07/2014 01:09 am
    This is an emergency maintenance performed by one of our transport providers.
    During this maintenance window, access to the DNS resolvers in Chi-2 DC may be impacted or unavailable at times.

    Please standby for further updates.

    Thank you.

    Started 01:02 AM Feb 07, 2014

    Ended 01:11 AM Feb 07, 2014

  • 02/06
    Impact
    Severity
    Upstream connectivity loss in PHX-1
    Status : Closed

    Posted at: 02/06/2014 03:02 pm
    This issue was resolved, normal connectivity has been restored.

    Posted at: 02/06/2014 03:02 pm
    Our PHX-1 data center lost upstream connectivity for a few minutes. This affected the PHX-1 location as well as DNS resolvers for some servers in other facilities.

    Started 02:49 PM Feb 06, 2014

    Ended 02:52 PM Feb 06, 2014

  • 02/06
    Impact
    Severity
    Public network issues in dl.111
    Status : Closed

    Posted at: 02/06/2014 10:50 am
    The switch replacement is complete. If you are still experiencing issues, please submit a new ticket to our Technical Support team so that we can check this for you. Thank you for your patience while we resolved this issue.

    Posted at: 02/06/2014 10:37 am
    The switch replacement is underway and the ETA for full restoration of service is about 20 minutes.

    Posted at: 02/06/2014 10:13 am
    We are currently replacing the public network switch in this cab. This will cause 100% downtime while we perform the hardware swap. We apologize for the inconvenience; unfortunately, we are unable to resolve this issue without replacing the hardware. We will continue to keep this status up-to-date with the latest information.

    Posted at: 02/06/2014 09:35 am
    We are currently experiencing an issue with the public network on some servers in the dl.111 cab at the CHI-2 facility. We are investigating and expect this to be resolved shortly. No other locations are currently affected.

    Started 09:05 AM Feb 06, 2014

    Ended 10:50 AM Feb 06, 2014

  • 01/20
    Impact
    Severity
    Outage at Chi-2
    Status : Closed

    Posted at: 01/20/2014 09:48 pm
    All services are normal at this time.

    Posted at: 01/20/2014 09:36 pm
    Many devices and services have come back online. We are still working to ensure all services are up.

    Posted at: 01/20/2014 09:24 pm
    Starting around 9:00PM Central we started receiving alerts from the network at Chi-2. Many customers with servers at Chi-2 are affected. Our network engineers are currently troubleshooting nd working to restore service.

    Started 09:00 PM Jan 20, 2014

    Ended 09:45 PM Jan 20, 2014

  • 01/15
    Impact
    Severity
    AMS Core Maintenance
    Status : Closed

    Posted at: 01/15/2014 11:08 pm
    The maintenance is now complete. Thank you for your patience.

    Posted at: 01/15/2014 10:05 pm
    We are starting the AMS core maintenance now. No outage is expected.

    Started 10:05 AM Jan 15, 2014

    Ended 10:08 AM Jan 16, 2014

  • 12/23
    Impact
    Severity
    Scheduled emergency maintenance - aswg-c06.d03.s02.phx01
    Status : Closed

    Posted at: 12/23/2013 10:35 pm
    Greetings,

    The maintenance is completed and full service has been restored.

    Thank you for your continued patience in this process.

    Posted at: 12/23/2013 10:28 pm
    Greetings,

    A scheduled emergency maintenance for aswg-c06.d03.s02.phx01 has begun. As soon as more information is available, we will update this status page.

    Thank you for your patience in this matter.

    Started 10:26 PM Dec 23, 2013

    Ended 10:32 PM Dec 23, 2013

  • 12/18
    Impact
    Severity
    priv-asw-er186-4.ord03 private network outage
    Status : Closed

    Posted at: 12/18/2013 03:56 pm
    The problem has been corrected and private network communication has been restored. We apologize for any inconvenience this may have caused and appreciate your patience!

    Posted at: 12/18/2013 03:42 pm
    Greetings,

    Our network operations team is currently investigating issues with the switch priv-asw-er186-4.ord03.singlehop.net. This switch provides access to private network resources.

    At this time, we are still working to restore service. As soon as more information is available, we will let you know.

    Please note that this outage should *not* affect public network communications for servers connected to this switch.

    Please stand by for further updates. Thank you for your patience!

    Started 03:04 PM Dec 18, 2013

    Ended 03:42 PM Dec 18, 2013

  • 12/03
    Impact
    Severity
    cab switch asw-fy132.ord03 outage
    Status : Closed

    Posted at: 12/03/2013 06:42 pm
    Greetings,

    The switch has been restored to a normal operational status. We will continue to investigate the root cause for the issue with an RFO to follow.

    We apologize for the inconvenience.

    Posted at: 12/03/2013 06:37 pm
    Cab switch asw-fy132.ord03 is experiencing trouble at this time. Please standby while we are still working to troubleshoot the issue and locate the root cause.

    Posted at: 12/03/2013 06:27 pm
    Cab switch asw-fy132.ord03 is experiencing trouble at this time. Please standby while we are still working to troubleshoot the issue and locate the root cause.

    Posted at: 12/03/2013 06:20 pm
    We are still working to restore service to this switch. Please standby for additional updates.

    Posted at: 12/03/2013 06:03 pm
    Greetings,

    Cab switch asw-fy132.ord03 is experiencing trouble at this time. Please standby while we troubleshoot the issue and locate the root cause.

    Started 05:55 PM Dec 03, 2013

    Ended 06:40 PM Dec 03, 2013

  • 11/19
    Impact
    Severity
    Security Upgrade Maintenance
    Status : Closed

    Posted at: 11/19/2013 11:01 pm
    This maintenance is now complete.

    Posted at: 11/19/2013 10:22 pm
    We are starting this maintenance now. No impact is expected.

    Started 10:20 AM Nov 19, 2013

    Ended 11:00 AM Nov 20, 2013

  • 11/01
    Impact
    Severity
    Cabinet switch aswg-eg186.ord03 outage
    Status : Closed

    Posted at: 11/01/2013 01:15 am
    The switch is now online and functioning normally. This was a power related issue. The issue will be investigated and addressed internally.
    Please open a Technical Support ticket if you need assistance.
    This status report is now closed.
    Thank you.

    Posted at: 11/01/2013 01:10 am
    Our monitoring systems indicate this switch is down.
    Please stand by for further updates.

    Started 01:05 AM Nov 01, 2013

    Ended 01:09 AM Nov 01, 2013

  • 10/29
    Impact
    Severity
    Cab switch aswg-ej183-2.ord03
    Status : Closed

    Posted at: 10/30/2013 12:51 am
    This emergency maintenance has been completed. The switch is functioning normally and connectivity verified.
    This status report is now closed.

    Posted at: 10/30/2013 12:40 am
    We are now proceeding to reset the switch chassis. This may take up to 10 minutes.
    Please stand by for further updates.

    Posted at: 10/30/2013 12:08 am
    We are still troubleshooting this issue. Traffic is being forwarded normally, however, we are still seeing an issue that may require further action.
    Please stand by for further updates.

    Posted at: 10/29/2013 11:51 pm
    We have identified an issue with this cab switch that is causing intermittent latency. We have opened an emergency maintenance window for this cab switch. Customers may notice up to 7 minutes of traffic disruption.
    Please stand by for further updates.

    Started 11:35 PM Oct 29, 2013

    Ended 12:48 AM Oct 30, 2013

  • 10/11
    Impact
    Severity
    Phone system to be temporarily offline for maintenance
    Status : Closed

    Posted at: 10/12/2013 12:09 am
    Phone system maintenance has been completed.

    Posted at: 10/11/2013 11:31 pm
    Our around-the-clock phone support will be offline for about 30 minutes as we perform upgrades to our internal office network. During this time our ticket system will continue to be online, so for the fastest support responses we encourage clients to submit support tickets.

    Thank you.

    Started 11:45 PM Oct 11, 2013

    Ended 12:10 AM Oct 12, 2013

  • 10/10
    Impact
    Severity
    PHX-1 to CHI-2 Transport Link Outage
    Status : Closed

    Posted at: 10/10/2013 05:51 pm
    Greetings,

    Hardware replacement has been completed and the transport link is back online at this time.

    An official RFO will be made available as soon as we have received it.

    Thank you for your continued patience in this matter.

    Posted at: 10/10/2013 05:09 pm
    Greetings,

    We've received reports that hardware replacement is underway. Estimated completion time for this process is 6:00PM CDT.

    Posted at: 10/10/2013 04:27 pm
    Greetings,

    A field tech has arrived on site at the location of the outage and is investigating the issue.

    Please stand by for further updates.

    Posted at: 10/10/2013 02:41 pm
    Greetings,

    A technician is en route to the upstream provider site to further investigate this issue. The ETA for the technician's arrival is approximately 30 minutes.

    As soon as we have further information we will update this status.

    Thank you for your continued patience in this matter.

    Posted at: 10/10/2013 02:16 pm
    The upstream provider is still working to restore service to this circuit. At this time we do not have an ETA but as soon as we have more information we will update this page. Please stand by for further updates. Thank you for your continued patience!

    Posted at: 10/10/2013 01:54 pm
    Greetings,

    The transport link between our CHI-2 and PHX-1 data centers has gone down. We are working with our upstream providers to restore service as quickly as possible.

    This issue is actively being worked on. The impact should only be seen on private network traffic between the two locations.

    We will update this thread again as soon as more information becomes available. Thank you.

    Started 01:40 PM Oct 10, 2013

    Ended 05:40 PM Oct 10, 2013

  • 10/01
    Impact
    Severity
    Phone system outage
    Status : Closed

    Posted at: 10/01/2013 02:43 pm
    This has been resolved by our ISP.

    Posted at: 10/01/2013 01:41 pm
    An ISP issue is causing issues with our phone support system. Some clients may be experiencing issues contacting SingleHop via phone. All client services are online, including the LEAP ticket system.

    Started 01:13 PM Oct 01, 2013

    Ended 02:30 PM Oct 01, 2013

  • 09/30
    Impact
    Severity
    CHI-2 access switch aswg-ej183-2.ord03 outage
    Status : Closed

    Posted at: 09/30/2013 09:47 pm
    Greetings,

    aswg-ej183-2.ord03 is back online and operating properly at this time.

    Posted at: 09/30/2013 09:41 pm
    Greetings,

    We are presently experiencing an outage one of our public gigabit access switches at our CHI-2 facility aswg-ej183-2.ord03. Please standby for further updates while we investigate the issue.

    Started 09:30 PM Sep 30, 2013

    Ended 09:40 PM Sep 30, 2013

  • 09/19
    Impact
    Severity
    CHI-2 access switch aswg-ej183-2.ord03 outage
    Status : Closed

    Posted at: 09/19/2013 06:39 pm
    Greetings,

    We experienced a short outage with aswg-ej183-2.ord03. The switch is back online at this time. If you are still encountering problems, please contact our support department for additional assistance. We will provide an RFO for this event shortly.

    Thank You.

    Started 06:22 PM Sep 19, 2013

    Ended 06:28 PM Sep 19, 2013

  • 09/18
    Impact
    Severity
    CHI-1 Core Code Upgrade
    Status : Closed

    Posted at: 09/18/2013 02:34 am
    All services have been fully restored. If you experience any additional issues please open a ticket with our NOC.

    Posted at: 09/18/2013 02:18 am
    We are continuing to bring the router back to full redundancy. Customers should see no other impacts while we finish this work.

    Posted at: 09/18/2013 02:08 am
    Router has been successfully rebooted and services are coming back online now.

    Posted at: 09/18/2013 01:56 am
    Hello,

    We will not be upgrading the core, please standby.

    Posted at: 09/18/2013 01:07 am
    We are starting this maintenance now. We will be doing some prep work before the upgrade. There will be another update before the upgrade starts.

    Started 01:05 AM Sep 18, 2013

    Ended 02:34 AM Sep 18, 2013

  • 09/11
    Impact
    Severity
    Chi-2 CR-3 Aggregation - intermittent packet loss
    Status : Closed

    Posted at: 09/11/2013 11:59 am
    This issue has been resolved. We verified connectivity.
    This status report is now closed.

    Posted at: 09/11/2013 11:44 am
    We are still working on this issue. Further updates will provided shortly.

    Posted at: 09/11/2013 11:12 am
    We are still working on this issue. We will provide more information as it becomes available.

    Posted at: 09/11/2013 11:00 am
    We are noticing intermittent packet loss to some customers in this segment of the network. Not all customers in the segment are impacted.
    We are troubleshooting this issue right now.
    Please stand by for further updates.

    Started 10:35 AM Sep 11, 2013

    Ended 11:55 AM Sep 11, 2013

  • 09/11
    Impact
    Severity
    Provider intermittent connectivity in Chi-2
    Status : Closed

    Posted at: 09/11/2013 09:10 am
    The issue has been resolved as of 8:40AM CDT. Connectivity has been verified.
    This status report is now closed.

    Posted at: 09/11/2013 08:24 am
    We are continuing to work on this issue.
    Thank you for your patience in this matter.

    Posted at: 09/11/2013 07:52 am
    Troubleshooting continues, we will provide more information as it becomes available.

    Posted at: 09/11/2013 07:31 am
    We are still working on this issue.
    We will keep this status report updated with more information as it becomes available.

    Posted at: 09/11/2013 07:06 am
    We are still troubleshooting intermittent connectivity issue.
    Further updates will be posted here. Please standby.

    Posted at: 09/11/2013 05:50 am
    We are observing intermittent connectivity issues that may impact some customers in Chi-2
    We are working with our provider to resolve these issues.
    Please standby for further updates.

    Started 05:30 AM Sep 11, 2013

    Ended 08:40 AM Sep 11, 2013

  • 09/10
    Impact
    Severity
    CHI-2 fu.139 cab switch outage
    Status : Closed

    Posted at: 09/10/2013 07:49 pm
    Greetings,

    The access switch located in fu.139 experienced an unrecoverable error. We completed a hard reboot of the device, which is back online and working properly at this time. We will closely monitor this switch and open a new status message in the event that this should occur again. Clients should have seen their network connectivity restored at this time. If you are still experiencing network related issues, please contact our support department.

    Thank You,

    Posted at: 09/10/2013 07:35 pm
    Greetings,

    We are still working the issue at this time. Please standby for additional updates.

    Thank You,

    Posted at: 09/10/2013 07:23 pm
    Greetings,

    We are still troubleshooting this issue attempting to find the root cause. Please standby for additional updates.

    Thank You,

    Posted at: 09/10/2013 07:16 pm
    Greetings,

    We are experiencing trouble with one of our cab access switches at CHI-2. Please standby while we gather more information and identify the issue.

    Thank You

    Started 07:10 PM Sep 10, 2013

    Ended 07:45 PM Sep 10, 2013

  • 09/04
    Impact
    Severity
    Cab switch outage - aswg-dl112.ord03
    Status : Closed

    Posted at: 09/04/2013 03:19 am
    The switch is now functioning normally. Please open a Technical Support ticket if you are still experiencing any issues with any server in this cab switch.

    Thank you.

    Posted at: 09/04/2013 03:10 am
    The switch hardware has been replaced. We are now verifying connectivity.

    Posted at: 09/04/2013 02:45 am
    We are working on replacing the switch hardware at this time.

    Posted at: 09/04/2013 02:30 am
    We have an active alert on possible switch outage.
    We are investigating this issue right now.
    Please standby for further updates.

    Started 02:08 AM Sep 04, 2013

    Ended 03:04 AM Sep 04, 2013

  • 08/11
    Impact
    Severity
    Networking Issues with CHI-2 Core Routers
    Status : Closed

    Posted at: 08/11/2013 07:00 am
    The issue with our core routers and associated cab switches has been resolved. At this time we are investigating how to mitigate this issue from occurring in the future. We will provide an RFO shortly regarding this event.

    Posted at: 08/11/2013 06:23 am
    We are still working on bringing up 2 access switches at CHI-2. Most clients are not affected by this issue and have seen network connectivity fully restored. Please standby for additional updates.

    Posted at: 08/11/2013 05:37 am
    We have identified the root cause of the issue and are working to rectify it at this time. Clients should begin to see network connectivity restored shortly. Please standby for additional updates.

    Posted at: 08/11/2013 04:14 am
    We have identified the issue and are still working on stabilizing the network at this time. Please standby for additional updates.

    Posted at: 08/11/2013 03:57 am
    We have identified the issue with our core routers in CHI-2 and continue to resolve it at this time. Please standby for additional updates

    Posted at: 08/11/2013 03:39 am
    We have identified the issue with our core routers in CHI-2 and are working to resolve it at this time. Please standby for additional updates

    Posted at: 08/11/2013 03:24 am
    We are continuing to stabilize the situation and bring the network back to full functionality. Please standby for additional updates.

    Posted at: 08/11/2013 03:12 am
    We are still trying to stabilize the situation and bring the network back to full functionality. Please standby for additional updates.

    Posted at: 08/11/2013 03:07 am
    We are still working the issue with our CHI-2 routers and will continue to provide periodic updates regarding this issue.

    Posted at: 08/11/2013 02:54 am
    We are still working the issue on our CHI-2 core routes and will continue to provide periodic updates.

    Posted at: 08/11/2013 02:43 am
    We are experiencing an issue with our CHI-2 core routers. At this time we continue to investigate the issue and are working to resolve it as soon as possible.

    Posted at: 08/11/2013 02:05 am
    We are currently experiencing some networking issues that appear to be effecting servers connectivity. This is being investigated currently.

    Started 01:45 AM Aug 11, 2013

    Ended 06:45 AM Aug 11, 2013

  • 08/06
    Impact
    Severity
    Cab switch outage - aswg-el180.ord03
    Status : Closed

    Posted at: 08/07/2013 12:12 am
    The issue has been resolved and the switch is now functioning properly.
    This status report is now closed.
    Thank you.

    Posted at: 08/07/2013 12:00 am
    We've been alerted to connectivity issues with this cab switch. We are gathering initial information at this stage.
    Please standby for further updates.

    Started 11:51 PM Aug 06, 2013

    Ended 12:09 AM Aug 07, 2013

  • 07/28
    Impact
    Severity
    priv-asw-fx185 private switch issue
    Status : Closed

    Posted at: 07/28/2013 10:02 pm
    We have fixed the issue with the private access switch in FX.185. Private network connectivity has been restored. This was caused by a faulty uplink cross connect cable between the cab and our core. We have moved the device over to a different cable which immediately restored service. We will be repairing the faulty cable, which will not impact any services.

    Posted at: 07/28/2013 09:42 pm
    We are presently working toward a solution to restore connectivity to the private access switch in FX.185. Please standby for additional updates.

    Posted at: 07/28/2013 09:32 pm
    We are still experiencing a loss of connectivity with the private switch located in FX.185. We have identified the issue and are presently working on a solution at this time. Please standby for additional updates.

    Posted at: 07/28/2013 09:19 pm
    We are still working the issue with this particular switch and will provide an update shortly.

    Posted at: 07/28/2013 09:05 pm
    We are presently experiencing an issue with our private access switch in FX.185. Please standby for additional updates while we continue to work the issue and investigate the root cause.

    Posted at: 07/28/2013 08:57 pm
    We are presently experiencing an issue with one of our private access switches in FX.185. We are investigating the issue and will provide regular updates once we have found the root cause.

    Started 08:40 PM Jul 28, 2013

    Ended 09:59 PM Jul 28, 2013

  • 07/26
    Impact
    Severity
    Public cab switch maintenance - asw-eh183.ord03
    Status : Closed

    Posted at: 07/26/2013 12:31 am
    The maintenance has been completed. If you are experiencing any issues, then please open a Technical Support ticket.
    This status report is now closed.
    Thank you.

    Posted at: 07/26/2013 12:16 am
    This is a scheduled maintenance. The switch will be brought down during this maintenance window.
    Please standby for further updates.

    Started 12:16 AM Jul 26, 2013

    Ended 12:28 AM Jul 26, 2013

  • 07/25
    Impact
    Severity
    Public cab switch maintenance - asw-ex183.ord03
    Status : Closed

    Posted at: 07/25/2013 10:36 pm
    The maintenance has been completed. If you are experiencing any issues, then please open a Technical Support ticket.
    This status report is now closed.
    Thank you.

    Posted at: 07/25/2013 10:22 pm
    This is a scheduled maintenance. The switch will be brought down during this maintenance window.
    Please standby for further updates.

    Started 10:23 PM Jul 25, 2013

    Ended 10:34 PM Jul 25, 2013

  • 07/25
    Impact
    Severity
    Public cab switch maintenance - asw-eh176.ord03
    Status : Closed

    Posted at: 07/25/2013 12:32 am
    The maintenance has been completed. If you are experiencing any issues, then please open a Technical Support ticket.
    This status report is now closed.
    Thank you.

    Posted at: 07/25/2013 12:11 am
    This is a scheduled maintenance. The switch will be brought down during this maintenance window.
    Please standby for further updates.

    Started 12:12 AM Jul 25, 2013

    Ended 12:25 AM Jul 25, 2013

  • 07/24
    Impact
    Severity
    Public cab switch maintenance - asw-ei176.ord03
    Status : Closed

    Posted at: 07/24/2013 10:33 pm
    The maintenance has been completed. If you are experiencing any issues, then please open a Technical Support ticket.
    This status report is now closed.
    Thank you.

    Posted at: 07/24/2013 10:12 pm
    This is a scheduled maintenance. The switch will be brought down during this maintenance window.
    Please standby for further updates.

    Started 10:11 PM Jul 24, 2013

    Ended 10:23 PM Jul 24, 2013

  • 07/08
    Impact
    Severity
    Cab switch outage - aswg-gf150.ord03
    Status : Closed

    Posted at: 07/08/2013 12:32 am
    This issue has been resolved. The cab switch is functioning normally.

    This status report is now closed.

    Thank you.

    Posted at: 07/08/2013 12:16 am
    We have been alerted to cab switch aswg-gf150.ord03 possible outage. We are collecting initial information and investigating this issue.
    Please standby for further updates.

    Started 12:05 PM Jul 08, 2013

    Ended 12:27 PM Jul 08, 2013

  • 06/28
    Impact
    Severity
    VMware Public Cloud at CHI-2 issue
    Status : Closed

    Posted at: 06/28/2013 12:42 pm
    All VMs were confirmed online at 12:34PM. If you have any further issues, please contact our Support team.

    Posted at: 06/28/2013 10:34 am
    The network connection on VMware compute node 4 experienced a flap on the connection to the SAN device. Because the network device had a heartbeat, the failover did not occur, although the issue caused intermittent connectivity between the VMs and the SAN.

    A systematic rebooting of all impacted virtual machines was immediately conducted and is ongoing. Currently, all nodes have access to the SAN device again, although some VMs are experiencing read-only file systems that require a reboot to resolve.

    Started 04:15 AM Jun 28, 2013

    Ended

  • 06/26
    Impact
    Severity
    Cab switch outage - asw-c02.r02.ord02
    Status : Closed

    Posted at: 06/26/2013 07:01 am
    The switch is now online. We will be collecting more information and working with our vendor in order to identify and fix the root cause of this crash.

    This status report is now closed.

    Thank you.

    Posted at: 06/26/2013 06:53 am
    We have been alerted to cab switch asw-c02.r02.ord02 possible outage. We are investigating this issue and collecting initial information.
    Please stand by for further updates.

    Started 06:45 AM Jun 26, 2013

    Ended 06:49 AM Jun 26, 2013

  • 06/22
    Impact
    Severity
    Provider Outage in AMS
    Status : Closed

    Posted at: 06/22/2013 07:17 pm
    The provider has stabilized. We will be looking into this issue further with our provider.

    Posted at: 06/22/2013 07:00 pm
    We had a provider flap in AMS. We are currently working with the vendor to find the root cause.

    Started 06:51 AM Jun 22, 2013

    Ended 06:54 AM Jun 22, 2013

  • 06/20
    Impact
    Severity
    Scheduled Maintenance - asw-c16g.r02.ord02.singlehop.net
    Status : Closed

    Posted at: 06/20/2013 10:50 pm
    The maintenance is completed and this window is now being closed.

    If you are still experiencing connectivity issues, please feel free to contact our support team to have the issue further investigated.

    Thank you for your patience throughout this process.

    Posted at: 06/20/2013 10:37 pm
    Greetings,

    The maintenance is completed and the switch and all connected servers should be back online. We are performing final cleanup and checks of connectivity at this time.

    Please stand by for further updates.

    Posted at: 06/20/2013 10:01 pm
    The scheduled maintenance window for asw-c16g.r02.ord02.singlehop.net is now open and maintenance will begin shortly.

    Please stand by for further updates. Thank you.

    Started 10:00 AM Jun 20, 2013

    Ended 10:37 AM Jun 20, 2013

  • 06/20
    Impact
    Severity
    CHI-2 Routing Issues
    Status : Closed

    Posted at: 06/20/2013 10:14 pm
    The network is currently stable and we have not seen any further issues. We will be closing this status entry at this time. A detailed incident report will be provided on request within 72 hours.

    Posted at: 06/20/2013 09:55 pm
    We are still monitoring the situation to verify stability.

    Posted at: 06/20/2013 09:37 pm
    The last update was made in error. Everything is still functioning normally and we are still monitoring the situation.

    Posted at: 06/20/2013 09:35 pm
    We are still working on providing a temporary solution to this issue. Please standby for additional updates.

    Posted at: 06/20/2013 09:20 pm
    We applied a solution to resolve the initial outage but that solution triggered a bug in some of the older versions of code running on our network. This caused some additional outages for our customers. A work around has been found and the network has been stabilized. We will continue to monitor this situation to verify full stability.

    Posted at: 06/20/2013 09:17 pm
    We are still working on providing a temporary solution to this issue. Please standby for additional updates.

    Posted at: 06/20/2013 09:05 pm
    The network has been stabilized once again. We are continuing to work the issue at this time. Please standby for additional updates.

    Posted at: 06/20/2013 08:53 pm
    We are continuing to work the issue, and provide a temporary solution to this event at this time. Please standby for additional updates.

    Posted at: 06/20/2013 08:44 pm
    Customers may be experiencing a routing impact at this time. Please standby for additional updates as we continue to investigate.

    Posted at: 06/20/2013 08:34 pm
    We are still working on providing a temporary solution to this issue. Please standby for additional updates.

    Posted at: 06/20/2013 08:23 pm
    We are continuing to work the issue, and provide a temporary solution to this event at this time. Please standby for additional updates.

    Posted at: 06/20/2013 08:15 pm
    The network has been stabilized once again. We are continuing to work the issue at this time. Please standby for additional updates.

    Posted at: 06/20/2013 08:04 pm
    Customers may still be experiencing issues with routing, we are working on stabilizing the issue and will continue to investigate the root cause which triggered this event. Please standby for additional updates.

    Posted at: 06/20/2013 07:57 pm
    Customers may still be experiencing issues with routing again, we are looking into what is causing this event and will provide updates shortly.

    Posted at: 06/20/2013 07:46 pm
    Customers may be experiencing issues with routing again, we are looking into what is causing this event and will provide updates shortly.

    Posted at: 06/20/2013 07:27 pm
    Once again, we have found the root cause of what triggered the event and are continuing to work on a temporary solution. Please standby for additional updates.

    Posted at: 06/20/2013 07:20 pm
    We have determined the root cause of the event and are now working toward a temporary solution. Please standby for additional updates.

    Posted at: 06/20/2013 06:58 pm
    We are still looking into the root cause of this event. Please standby for additional updates.

    Posted at: 06/20/2013 06:31 pm
    The network has returned to a stable condition at this time. We are still investigating what triggered this event. Please standby for additional updates.

    Posted at: 06/20/2013 06:13 pm
    We are still investigating the issues with our core at CHI-2 in regards to routing impact. Please standby for additional updates.

    Posted at: 06/20/2013 06:00 pm
    Customers may be experiencing a routing impact at this time. Please standby for additional updates as we continue to investigate.

    Posted at: 06/20/2013 05:55 pm
    We are presently experiencing an issue with a core router at our CHI-2 datacenter. Please standby while we investigate further.

    Started 05:52 AM Jun 20, 2013

    Ended 09:00 AM Jun 20, 2013

  • 06/14
    Impact
    Severity
    Public Switch Outage at CHI-1
    Status : Closed

    Posted at: 06/14/2013 09:43 pm
    Our network administrators traced the outage to a massive DDoS that took the switch completely offline and was also slightly affecting other switches in the data center. If there are any issues, please open a support ticket so we can investigate further.

    Posted at: 06/14/2013 09:24 pm
    The public network switch asw-c21.r02.ord02 is not responding and is not reachable after a reboot. Data center and Network staff have been notified and will begin checking on this very shortly. Thanks for your patience.

    Started 09:10 AM Jun 14, 2013

    Ended 09:40 AM Jun 14, 2013

  • 06/13
    Impact
    Severity
    Public cab switch maintenance - asw-fw139.ord03
    Status : Closed

    Posted at: 06/13/2013 10:57 pm
    Switch maintenance has been completed. Switch passed all post-maintenance tests, and connectivity has been verified.

    This status report is now closed.

    Thank you.

    Posted at: 06/13/2013 10:10 pm
    We are now starting this scheduled maintenance. The switch will be brought down for maintenance.

    Started 10:00 AM Jun 13, 2013

    Ended 10:50 AM Jun 13, 2013

  • 05/30
    Impact
    Severity
    Provider flap in PHX
    Status : Closed

    Posted at: 05/30/2013 11:00 am
    We had provider flap at our PHX DC which could have created some flapping of services for our customers. The issue has been resolved and we are currently investigating the root cause of the outage.

    Started 10:41 AM May 30, 2013

    Ended 10:43 AM May 30, 2013

  • 05/25
    Impact
    Severity
    Upstream provider issue at CHI-2
    Status : Closed

    Posted at: 05/25/2013 04:57 pm
    All services have now been restored.

    Posted at: 05/25/2013 04:45 pm
    Services are now being restored, please standby. No outage is expected from this.

    Posted at: 05/25/2013 04:22 pm
    We are still waiting for our provider to restore services.

    Posted at: 05/25/2013 03:54 pm
    The routing issue with Nlayer has been resolved, but the provider is still working on restoring service.

    Posted at: 05/25/2013 03:43 pm
    There seems to be some stale routes on the Nlayer network. We are working with them to get these removed ASAP.

    Posted at: 05/25/2013 03:29 pm
    We are seeing some potential lingering issues with NLayer and our Network Operations team is investigating.

    Posted at: 05/25/2013 03:18 pm
    We experienced an outage with one of our upstream providers, Nlayer, that may have caused a service flap at our CHI-2 facility. This issue was resolved within less than 2 minutes.

    Started 02:53 AM May 25, 2013

    Ended 02:55 AM May 25, 2013

  • 05/20
    Impact
    Severity
    Public Switch Down
    Status : Closed

    Posted at: 05/20/2013 07:06 am
    This was due to an attack on a client connected to this switch. The attack has been mitigated, and the switch is now back online.

    Posted at: 05/20/2013 06:59 am
    The switch in the server cabinent gj174 is currently unresponsive. We are aware of the issue, and our network operators are investigating this now.

    Started 06:45 AM May 20, 2013

    Ended

  • 03/25
    Impact
    Severity
    Routing issue on aswg-gf150
    Status : Closed

    Posted at: 03/25/2013 09:16 am
    This issue was resolved at 9:01AM CST and has not reoccurred. The routing issue was resolved by our Network Operations team.

    Posted at: 03/25/2013 09:00 am
    We are currently experiencing a routing issue on this switch. Our Network Operations team is working to resolve this issue now. We will keep this space updated.

    Started 08:39 AM Mar 25, 2013

    Ended 09:01 AM Mar 25, 2013

  • 03/21
    Impact
    Severity
    Connectivity Issue - CHI-1 BGP reset
    Status : Closed

    Posted at: 03/21/2013 11:11 am
    The issue is fully resolved at this time. Details on what caused the outage will follow once our final investigation is complete.

    Posted at: 03/21/2013 10:58 am
    Access has been restored to our ticketing system. We are still focusing our efforts to restore access to the remaining portions of our network that are still experiencing issues.

    Please stand by for further updates. Thank you for your patience in this matter.

    Posted at: 03/21/2013 10:32 am
    Most services are restored however we are still seeing issues with a small portion of our network. We are working to resolve this as quickly as possible.

    Please stand by for further updates.

    Posted at: 03/21/2013 10:21 am
    We are still working to resolve this issue and will have further updates as soon as they become available.

    Please stand by. Thank you.

    Posted at: 03/21/2013 10:10 am
    Network Operations has detected a BGP session reset on dr6506b.ord02.singlehop.net. We are actively investigating the issue.

    Started 10:05 AM Mar 21, 2013

    Ended 11:00 AM Mar 21, 2013

  • 03/18
    Impact
    Severity
    Power issue at CHI-1
    Status : Closed

    Posted at: 03/18/2013 08:08 pm
    At this time, we have verified that all connectivity has been restored. Thanks and have a good day.

    Posted at: 03/18/2013 07:42 pm
    The power has already been restored by our upstream provider, but we are working to ensure our infrastructure and affected servers are back online. Thank you for your patience.

    Posted at: 03/18/2013 07:41 pm
    We are experiencing a power issue at our CHI-1 facility that may be affecting some client servers as well as some of our backend infrastructure. We will update this post once we have further details.

    Started 07:30 AM Mar 18, 2013

    Ended 07:40 AM Mar 18, 2013

  • 03/18
    Impact
    Severity
    Power issue at CHI-1
    Status : Closed

    Posted at: 03/18/2013 07:27 pm
    At this time power has been restored and our internal networks, infrastructure, and client's with affected servers are now online. If you are having issues still, please let us know.

    Posted at: 03/18/2013 06:22 pm
    Power is coming back on. The LEAP3 control panel system is back online. Service may be slow at this time as the systems start to come back online. We are still verifying the full resolution of this issue.

    Posted at: 03/18/2013 06:11 pm
    The electricians working on this issue have given us an ETA of about 15 minutes. Full service restoration may take slightly longer as servers and network equipment boot back up.

    Posted at: 03/18/2013 05:36 pm
    An electrician is en route to investigate the issue further.

    Posted at: 03/18/2013 05:12 pm
    A power outage at our CHI-1 facility is affecting 5 cabinets (207.1, 207.2, 207.3, 207.4, and 206.1). We are working with our upstream provider who has a technician on-site working to resolve this issue.

    Posted at: 03/18/2013 04:42 pm
    Due to a bug in our update system, this status post improperly posted the start time as 4:28am, and that the issue has ended. The issue started at 4:28PM CST, and we are still working on resolving this. Thank you for your patience.

    Posted at: 03/18/2013 04:38 pm
    We are experiencing a power issue at our CHI-1 facility that may be affecting some client servers as well as some of our backend infrastructure. We will update this post once we have further details.

    Started 04:27 AM Mar 18, 2013

    Ended 06:14 AM Mar 18, 2013

  • 03/13
    Impact
    Severity
    SnapShield Firewall Outage
    Status : Closed

    Posted at: 03/13/2013 02:33 pm
    There was an attack that targeted a customer behind a SnapShield firewall. The customer was removed and service has been restored.

    Started 02:20 PM Mar 13, 2013

    Ended 02:30 PM Mar 13, 2013

  • 03/11
    Impact
    Severity
    Possible packet loss
    Status : Closed

    Posted at: 03/11/2013 12:31 pm
    We stopped seeing packet loss around 12:00 CST. We will provide an RFO once our analysis of the cause of these issues is complete.

    Posted at: 03/11/2013 11:31 am
    We are investigating an issue that may be causing packet loss for some customers on our network. We do not have the full scope of this issue yet, but it appears to be affecting some clients in the CHI-2 datacenter. We will keep you posted as we determine the scope, cause, and resolution to this issue.

    Started 10:57 AM Mar 11, 2013

    Ended 11:28 AM Mar 12, 2013

  • 03/11
    Impact
    Severity
    Provider Outage
    Status : Closed

    Posted at: 03/11/2013 08:42 pm
    Hello,

    Everything has continued to be stable. Service should be fully restored. We are going to work with our vendor and provider to investigate the issue further and find a solution.

    Posted at: 03/11/2013 08:12 pm
    Hello,

    We are currently experiencing outage with one of our transit providers. We are currently investigating the issue. As of right now service should be restored.

    Started 07:55 AM Mar 11, 2013

    Ended 08:05 AM Mar 11, 2013

  • 03/01
    Impact
    Severity
    Private network cab switch outage
    Status : Closed

    Posted at: 03/01/2013 07:50 am
    As part of this status report, we had an alert for a Public Cloud node outage. This issue has also been resolved and connectivity verified.

    Pleas open a Technical Support ticket if you need further assistance.

    Thank you.

    Posted at: 03/01/2013 07:43 am
    Update: We still have an alert for a Public Cloud node being down.
    This alert has not been cleared yet.
    Please stand by for further updates.

    Posted at: 03/01/2013 07:32 am
    The issue has been isolated to a power interruption event, and it has been resolved.
    While this outage did not impact the public network connectivity, some customers may have noticed interruption to some back-end functions, such as running backup operations.

    This status report is now closed.
    Please open a Technical Support ticket if you need further assistance.

    Thank you.

    Posted at: 03/01/2013 07:07 am
    We have been alerted by our monitoring system to a private network switch outage.
    This may be impacting some of our Public Cloud virtual servers functionalities.
    Please stand by for further updates.

    Started 07:00 AM Mar 01, 2013

    Ended 07:43 AM Mar 01, 2013

  • 02/18
    Impact
    Severity
    LEAP3 / Dropzone API Temporary Outage
    Status : Closed

    Posted at: 02/18/2013 01:52 pm
    This has been resolved

    Posted at: 02/18/2013 01:51 pm
    Greetings,

    We are aware of an issue with logging into LEAP3 or using our Dropzone API. We are working quickly to resolve this issue.

    Started 01:50 AM Feb 18, 2013

    Ended 01:52 AM Feb 18, 2013

  • 02/07
    Impact
    Severity
    c05.d03.s02.phx01 Public switch upgrade
    Status : Closed

    Posted at: 02/07/2013 10:55 pm
    This maintenance has been completed successfully.

    Thank you.

    Posted at: 02/07/2013 10:44 pm
    The hardware has been upgraded and the switch is no back online.
    We are now performing post-maintenance tasks and cleaning up.

    Posted at: 02/07/2013 10:19 pm
    We are now bringing the cab switch offline for maintenance.

    Posted at: 02/07/2013 10:12 pm
    This scheduled maintenance window is now open. We will be upgrading the public switch hardware for cab c05.d03.s02

    Please stand by for further updates.

    Started 10:10 AM Feb 07, 2013

    Ended 10:55 AM Feb 07, 2013

  • 01/31
    Impact
    Severity
    Ge.150 Public switch upgrade
    Status : Closed

    Posted at: 01/31/2013 11:03 pm
    This maintenance has been completed and connectivity has been verified.

    Thank you.

    Posted at: 01/31/2013 10:58 pm
    The upgrade is complete, and the switch is back up.
    We are now verifying connectivity and cleaning up.

    Posted at: 01/31/2013 10:42 pm
    We are performing scheduled maintenance to upgrade this cab switch. We will be bringing the switch down briefly.

    Started 10:30 AM Jan 31, 2013

    Ended 11:00 AM Jan 31, 2013

  • 01/30
    Impact
    Severity
    aswg-ej183-2.ord03.singlehop.net - Maintenance
    Status : Closed

    Posted at: 01/31/2013 12:04 am
    The uplink issue has been corrected and this switch is online and working properly at this time.

    Posted at: 01/30/2013 11:41 pm
    We are still working on restoring connectivity to one of the switches uplinks. This should not cause any interruption for any devices on this switch. Please standby for additional updates.

    Posted at: 01/30/2013 11:21 pm
    We are still working on restoring connectivity to one of the switches uplinks. This should not cause any interruption for any devices on this switch. Please standby for additional updates.

    Posted at: 01/30/2013 11:09 pm
    We are still working on restoring connectivity to one of the switches uplinks. This should not cause any interruption for any devices on this switch. Please standby for additional updates.

    Posted at: 01/30/2013 10:51 pm
    We are still working to complete this maintenance. Please standby for additional updates.

    Posted at: 01/30/2013 10:25 pm
    We have completed the switch replacement and have restored network connectivity. Please standby while we verify everything is working properly.

    Posted at: 01/30/2013 10:01 pm
    We will be starting the switch replacement for aswg-ej183-2.ord03.singlehop.net now. Please standby as we replace the device and restore network connectivity as soon as possible.

    Started 10:00 AM Jan 30, 2013

    Ended 12:05 PM Jan 31, 2013

  • 01/29
    Impact
    Severity
    Gigabit switch issue: aswg-ej183-2
    Status : Closed

    Posted at: 01/29/2013 01:11 pm
    This device is being scheduled for an emergency maintenance to replace the hardware. Should the hardware fail again before the scheduled maintenance, the device will be immediately replaced.

    Posted at: 01/29/2013 09:32 am
    We are experiencing some connectivity issues for some clients on the switch aswg-ej183-2 and are working to resolve these as quickly as possible. Thank you for your patience, further information will be provided as it is available.

    Started 09:25 AM Jan 29, 2013

    Ended 10:00 AM Jan 29, 2013

  • 01/29
    Impact
    Severity
    Public switch emergency maintenance - aswg-ej183-2.ord03
    Status : Closed

    Posted at: 01/29/2013 05:20 am
    The switch code has been updated. The switch is back up, and connectivity has been verified.

    Please open a support ticket if you still need assistance.

    This status report is now closed.

    Thank you.

    Posted at: 01/29/2013 05:04 am
    We are scheduling an emergency maintenance to update the code on this public switch.
    We will bring the switch down in 5 minutes.
    Please stand by for updates.

    Started 05:05 AM Jan 29, 2013

    Ended 05:15 AM Jan 29, 2013

  • 01/28
    Impact
    Severity
    Customer Gigabit Switch Outage: aswg-ej183-2 inaccessible
    Status : Closed

    Posted at: 01/28/2013 04:03 pm
    We have been able to pinpoint the reason for the connectivity issues with this switch. In the event that we loose connectivity again, we will promptly update the code to restore service. We apologize for any inconvenience this may have caused and appreciate your patience regarding this matter.

    Posted at: 01/28/2013 11:42 am
    The switch is accessible again. We are still investigating the root cause of this issue and ensuring a permanent resolution is applied.

    Posted at: 01/28/2013 11:26 am
    The gigabit switch aswg-ej183-2 in the CHI-2 datacenter is currently inaccessible. Our Network Operations team is currently working with Cisco TAC to ensure that this issue is permanently resolved. Thank you for your patience, we will keep this posted with all available updates.

    Started 11:23 AM Jan 28, 2013

    Ended 11:55 AM Jan 28, 2013

  • 01/27
    Impact
    Severity
    Customer Gigabit Switch Outage - aswg-ej183-2.ord03.singlehop.net
    Status : Closed

    Posted at: 01/28/2013 12:13 am
    We will continue to monitor this switch and be looking into the root cause of this issue.

    Posted at: 01/28/2013 12:03 am
    Network connectivity has been restored to servers on the affected switch.

    Posted at: 01/27/2013 11:49 pm
    At this time we are still attempting to restore network connectivity to servers connected to this device. Please standby for additional updates.

    Posted at: 01/27/2013 11:38 pm
    We continue to investigate and work toward restoring network connectivity on this switch. Please standby for additional updates.

    Posted at: 01/27/2013 11:26 pm
    We are still in the process restoring network connectivity regarding this device. Please standby for additional updates.

    Posted at: 01/27/2013 11:11 pm
    We are experiencing a connectivity issue with this customer gigabit switch. We are in the process of investigating and restoring service as soon as possible. Please standby for additional updates.

    Started 11:00 AM Jan 27, 2013

    Ended 12:00 PM Jan 28, 2013

  • 01/24
    Impact
    Severity
    Customer Cab Switch Outage
    Status : Closed

    Posted at: 01/24/2013 07:31 pm
    The customer cab switch is complete. Network connectivity has been restored to this cab.

    Posted at: 01/24/2013 07:21 pm
    We are still working on restoring network connectivity to servers in this cab. Please standby for additional updates.

    Posted at: 01/24/2013 07:07 pm
    We are in the process of bringing the new customer cab switch online. Please standby for additional updates.

    Posted at: 01/24/2013 06:50 pm
    The customer cab switch has failed and we are in the process of replacing it at this time. Please standby for additional updates.

    Posted at: 01/24/2013 06:25 pm
    At approximately 18:15 CST (GMT -6) our network monitoring system indicated that asw-c21.r01.ord02.singlehop.net had gone offline. We are investigating this issue at this time and working to restore services as quickly as possible.

    Started 06:15 AM Jan 24, 2013

    Ended 07:32 AM Jan 24, 2013

  • 01/19
    Impact
    Severity
    Public Switch Outage - aswg-ep180.ord03
    Status : Closed

    The switch replacement is completed and the new device is online. Service has been fully restored. If you still have trouble, please open a trouble ticket.

    Thank you for your patience.

    --

    Network Operations is currently configuring the replacement hotspare device with the most recent configuration backup of the failed device.

    Please stand by for further updates.

    --

    The switch device for EP.180 has failed. We are currently in the process of preparing a replacement hotspare device. Further updates will be provided shortly. Thank you for your patience.

    --

    Network Operations is currently investigating an outage in EP.180 with the public network switch servicing this cab. We will update with more information as it becomes available.

    Started 11:12 AM Jan 19, 2013

    Ended 12:05 PM Jan 20, 2013

  • 01/04
    Impact
    Severity
    Fx139 Public Switch Outage
    Status : Closed

    The switch has been replaced and service has been restored.

    -----

    The switch has failed. We are preparing a new switch at this time for replacement.

    -----

    We are currently experiencing an outage with the public switch in cabinet FX139. We are working to resolve the situation now.

    Started 01:03 AM Jan 04, 2013

    Ended 02:05 AM Jan 04, 2013

  • 12/11
    Impact
    Severity
    Configuration anomaly
    Status : Closed

    A configuration anomaly caused all or part of the CHI-1 DC to temporarily lose connectivity for approximately 2 minutes. The configuration that was causing the issue was removed and service has been fully restored.

    Started 07:45 PM Dec 11, 2012

    Ended 07:47 PM Dec 11, 2012

  • 12/06
    Impact
    Severity
    Cab switch scheduled maintenance
    Status : Closed

    This maintenance has been completed. Service has been restored, and connectivity has been verified.
    Please open a Technical Support ticket if you need further assistance.

    Thank you.

    ----

    Services have been restored. We are now verifying connectivity and cleaning up.

    ----

    We are now reloading the device and gradually bringing up service.
    Please stand by.

    ----

    This maintenance window is now open.
    Please stand by for further updates.

    Started 10:08 AM Dec 06, 2012

    Ended 10:35 AM Dec 06, 2012

  • 12/06
    Impact
    Severity
    Emergency Maintenance
    Status : Closed

    This maintenance is now complete.

    ----

    The device has been successfully reloaded. We are now finishing up our clean up process. We should incur no other outages during this maintenance.

    ----

    We will now be reloading the device.

    -----

    We are starting this maintenance now. Once we are ready to reload the device we will send another update.

    Started 03:29 AM Dec 06, 2012

    Ended 06:20 AM Dec 06, 2012

  • 12/03
    Impact
    Severity
    Non-impacting Core Maintenance
    Status : Closed

    The Maintenance is complete.

    ----

    Completing final tasks for the maintenance.

    ----

    For approximately 2 minutes starting at 10:31PM CST, access to the CHI-2 facility was unavailable. This was due to some BGP sessions on the device clearing after a new transit provider was brought online. This is not a normal situation, we are going to contact the vendor for further investigation. Normal service has already been restored.

    ----

    Starting the maintenance now.

    ----

    We are just about ready to start the maintenance.

    ----

    Started 10:00 PM Dec 03, 2012

    Ended 11:04 PM Dec 03, 2012

  • 11/30
    Impact
    Severity
    Cab switch outage
    Status : Closed

    We've been alerted by our monitoring system to an issue with this cab switch.
    We are now investigating the issue and will update the status here shortly.
    Please standby.

    ----

    This issue has been resolved, the impact was minimal.
    Thank you for your patience.

    Started 08:23 AM Nov 30, 2012

    Ended 08:27 AM Nov 30, 2012

  • 11/29
    Impact
    Severity
    DNS Resolution Temporary Outage
    Status : Closed

    Although our DNS resolver system has been restarted, we are having issues with internal networking and DNS resolution still. We are looking into this, and we will update you shortly. Thanks for your patience.

    *- This issue is now resolved, thank you for your patience in this matter.

    Started 07:16 AM Nov 29, 2012

    Ended 08:55 AM Nov 29, 2012

  • 11/18
    Impact
    Severity
    CHI-2 Connectivity Issues
    Status : Closed

    The SingleHop network operations team is investigating connectivity issues at CHI-2. Please stand by for further updates.

    --

    We are still investigating the issue. Services are partially restored and we are working to achieve full restoration of services at the CHI-2 datacenter. As more details become available we will update this status post.

    --

    The network event has ended and we have reached a stable state. The SingleHop network operations team is still investigating the cause of this event. As soon as we have completed our investigation an RFO will be provided.

    Thank you for your patience in this matter.

    Started 01:10 PM Nov 18, 2012

    Ended 01:43 PM Jan 18, 2012

  • 11/13
    Impact
    Severity
    Emergency reboot of ticket system backend required
    Status : Closed

    The ticketing system backend required an emergency reboot. This completed without major incident and the ticketing system is now back online and operational. We apologize for any inconvenience this may have caused. No client servers were affected and our phone system remained online during the outage.

    Started 07:00 PM Nov 13, 2012

    Ended 07:20 PM Nov 13, 2012

  • 11/08
    Impact
    Severity
    Connectivity Issues - dr6506a.ord03.singlehop.net
    Status : Closed

    The SingleHop network operations team is currently investigating network connectivity issues at our CHI-2 Datacenter.

    Please stand by for further updates.

    ---
    We are still experiencing issues with dr6506a.ord03.singlehop.net. We are currently moving traffic over to the second core at CHI-2.

    Please stand by for further updates.

    ---
    We've reached stability at this time but are still looking into the cause of this outage.

    Further updates will be provided as more information becomes available.

    Started 03:31 PM Nov 08, 2012

    Ended 04:06 PM Nov 08, 2012

  • 11/05
    Impact
    Severity
    CHI-2 Public Network Upgrade
    Status : Closed

    The maintenance is now complete

    -----

    We are in the process of migrating cabinets.

    -----

    We are still in the process of migrating cabinets.

    -----

    Done with prep work. Starting to migrate cabinets now.

    -----

    Starting the maintenance now.

    Started 10:00 PM Nov 05, 2012

    Ended 11:01 AM Nov 06, 2012

  • 11/03
    Impact
    Severity
    Switch outage
    Status : Closed

    The switch in this cabinet is unresponsive. Technicians are currently investigating and working to bring this back online as soon as possible. We are working to replace the switch hardware now. As of 10:15PM CST, the switch software configuration is complete and all servers have been re-attached.

    Started 08:55 PM Nov 03, 2012

    Ended 10:15 PM Nov 03, 2012

  • 10/15
    Impact
    Severity
    shared-cloud10-sata.singlehop.com outage
    Status : Closed

    09:19 AM CDT The cloud node is now back online, and vms are in the process of booting. The raid controller had failed, and needed to be replaced. No data loss should occur due to this.
    --
    This shared cloud node is down due to some seemingly storage related issues. Or datacenter operations team is currently working to restore service to the cloud vms contained on this machine. We'll update as more information becomes available.

    Started 09:00 AM Oct 15, 2012

    Ended 09:15 AM Oct 15, 2012

  • 09/28
    Impact
    Severity
    Cab switch scheduled maintenance
    Status : Closed

    Connectivity has been verified. This scheduled maintenance is now complete.

    ---

    The maintenance has been completed. We are now verifying connectivity for each server.

    ---

    This is a scheduled maintenance to replace the cab switch.
    The switch will be brought down. Please stand by for further updates.

    Started 12:05 PM Sep 28, 2012

    Ended 12:20 PM Sep 28, 2012

  • 09/25
    Impact
    Severity
    PHX-1 Aggregation Upgrade
    Status : Closed

    The Maintenance is now complete. No outage was detected. All maintenance tickets should close within two days.

    ----

    We are starting the maintenance now.

    Started 01:17 PM Sep 25, 2012

    Ended 11:56 PM Sep 25, 2012

  • 09/20
    Impact
    Severity
    Packet Loss on Cab Switch
    Status : Closed

    This issue should now be resolved. A misconfiguration between two devices on the switch impacted the CPU and caused packet loss for other customers. The configuration issue has been resolved and the CPU is back down to normal levels.

    -----

    We are currently working on a gig cab switch issue. Customers may currently be seeing packet loss on this switch. Stay tuned for further updates.

    Started 01:00 PM Sep 20, 2012

    Ended 01:10 PM Sep 20, 2012

  • 09/19
    Impact
    Severity
    CHI-1 Private Core Upgrade
    Status : Closed

    The maintenance is now complete. All tickets will close within two business days.

    -----

    The upgrade is now complete. Cleaning up configuration now.

    -----

    First outage complete. Moving on to second outage.

    -----

    You should start seeing an outage now.

    ------

    We are starting the private core upgrade in CHI-1 now.

    Started 10:00 PM Sep 19, 2012

    Ended 12:56 AM Sep 20, 2012

  • 07/16
    Impact
    Severity
    CHI-1 to CHi-2 Transport Link Down
    Status : Closed

    A Software bug in the provider's hardware caused instability in the protection protocol on the transport link between the CHI-1 and CHI-2 data centers. Stability was restored approximately 30 minutes later.

    Started 01:24 PM Jul 16, 2012

    Ended 01:52 PM Jul 16, 2012

  • 06/07
    Impact
    Severity
    CR3 Aggregation Device Failover
    Status : Closed

    While our colocation company was running new cross connections to one of our core cabinets, a power cable dislodged from an aggregation layer device. This incident caused a loss of power to the device, requiring us to perform an emergency failover to our backup aggregation device. It took twelve minutes for the backup aggregation device to become active and bring our systems back online.

    Started 10:08 AM Jun 07, 2012

    Ended 10:20 AM Jun 07, 2012

  • 05/11
    Impact
    Severity
    Outage from Software Upgrade
    Status : Closed

    During a normal scheduled maintenance there was unexpected impact that caused an outage throughout the CHI-2 data center. A time line of the events is as follows:

    6:05 am - Customers experienced the outage that was detailed in the maintenance.
    6:25 am - A device failed while being updated which caused a flap in network services. This negatively impacted the core and thus brought down all services at CHI-2.
    6:47 am - Services were restored to 75% of the customers at CHI-2.
    7:25 am - Services were restored to 100% of the customers at CHI-2.
    8:20 am - During the cleanup of this outage, a secondary switch malfunctioned and created a layer 2 issue on the network that impacted the core once again.
    9:04 am - All services were restored to 100% of the customers at CHI-2.

    Started 06:12 AM May 11, 2012

    Ended 09:35 AM May 11, 2012

  • 12/31
    Impact
    Severity
    Phone System Maintenance
    Status : Closed

    Posted at: 08/28/2014 10:09 pm
    The phone system maintenance is now complete. If you have trouble reaching us by phone beyond this point in time, please be sure to open a support ticket via LEAP3.

    As always, we thank you for your patience in this process.

    Posted at: 08/28/2014 09:02 pm
    Greetings,

    The phone system maintenance that is scheduled tonight at 9:00PM CDT is now starting.

    Please stand by for further updates.

    Thank you.

    Started 06:00 PM Dec 31, 1969

    Ended 10:09 PM Aug 28, 2014

  • 09/11
    Impact
    Severity
    Connectivity Issues CHI-2
    Status : CLOSED

    Connectivity issues at CHI-2- techs are aware and working to resolve it.

    Posted at: Sep 11, 2012 at 06:31

    Austin Wilson

    We apologize for ending the maintenance late. The extra time was required to ensure the least possible impact for our customers. All work has now been completed. All maintenance tickets will automatically close within two days.

    Posted at: Sep 11, 2012 at 05:51

    Austin Wilson

    The resources have been refreshed. Migrating traffic back now.

    Posted at: Sep 11, 2012 at 05:07

    Austin Wilson

    Traffic migration complete, starting resource refresh now.

    Posted at: Sep 11, 2012 at 04:46

    Austin Wilson

    80% done with migrating traffic to secondary core.

    Posted at: Sep 11, 2012 at 04:05

    Austin Wilson

    Hello, We are starting our core maintenance now. Please standby for further updates.

  • 08/28
    Impact
    Severity
    Connectivity Issues CHI-2
    Status : CLOSED

    Connectivity issues at CHI-2- techs are aware and working to resolve it.

    Posted at: Aug 28, 2012 at 06:43

    Austin Wilson

    All maintenance work has been completed. No outage was experienced. Your maintenance tickets will automatically close within the next two days.

    Posted at: Aug 28, 2012 at 05:06

    Austin Wilson

    We are starting the maintenance now.

  • 08/25
    Impact
    Severity
    Cabinet issues
    Status : CLOSED

    Cabinet(s) #aswg-ej183-2.ord03 having connectivity issues our techs are working on it.

    Posted at: Aug 25, 2012 at 22:28

    Garett Murphy

    Greetings, The maintenance is complete. If you need further technical assistance, please open a ticket. Thank you.

    Posted at: Aug 25, 2012 at 22:01

    Garett Murphy

    Greetings, Maintenance for aswg-ej183-2.ord03 is now commencing. Please stand by for further updates. Thank you.

  • 08/24
    Impact
    Severity
    Connectivity issues in PHX-1
    Status : CLOSED

    Connectivity issues in PHX-1- techs are aware and working to resolve it.

    Posted at: Aug 24, 2012 at 02:06

    Mustafa Hussein

    This maintenance has been completed. Connectivity has been verified for each system in this cab. Please open a separate Technical Support ticket if you are having problems with your server. This maintenance window is now closed, and this status report is now closed. Thank you.

    Posted at: Aug 24, 2012 at 01:54

    Mustafa Hussein

    The scheduled maintenance for this cab is still underway. We apologize that the status report has not been posted properly. The maintenance tasks have been completed and the cab switch is online. We are now verifying connectivity for each server before closing this status report. Please stand by for further updates. Thank you.

  • 08/16
    Impact
    Severity
    Connectivity Issues CHI-1
    Status : CLOSED

    Connectivity issues at CHI-1- techs are aware and working to resolve it.

    Posted at: Aug 17, 2012 at 01:04

    Mustafa Hussein

    This scheduled maintenance has been completed. Connectivity has been verified. If you need technical help, then please open a Technical Support ticket. This maintenance status report is now closed. Thank you.

    Posted at: Aug 16, 2012 at 23:11

    Mustafa Hussein

    This is a scheduled maintenance for 207.16 The maintenance window is now open, during which, we will be performing hardware upgrade and other maintenance tasks. This status will be updated when the maintenance is finished. Thank you.

  • 08/15
    Impact
    Severity
    Cabinet issues
    Status : CLOSED

    Cabinet(s) #ED.108 having connectivity issues our techs are working on it.

    Posted at: Aug 15, 2012 at 22:31

    Garett Murphy

    The maintenance is completed and we are now closing this issue. If you need technical assistance, please open a new ticket. Thank you.

    Posted at: Aug 15, 2012 at 22:06

    Garett Murphy

    We are starting the Scheduled maintenance for ED.108 private network switch (priv-asw-ed108.ord03.singlehop.net). Please stand by for further updates.

  • 08/04
    Impact
    Severity
    Upstream Provider Issue
    Status : CLOSED

    Upstream providers is having technical issues. Route-reconvergence will occur in a few minutes.

    Posted at: Aug 05, 2012 at 00:17

    Garett Murphy

    This issue is resolved. If you need further technical assistance, please open a new ticket. Thank you.

  • 08/04
    Impact
    Severity
    Upstream Provider Issue
    Status : CLOSED

    Upstream providers is having technical issues. Route-reconvergence will occur in a few minutes.

    Posted at: Aug 05, 2012 at 00:16

    Garett Murphy

    This issue is resolved. If you need further technical assistance, please open a new ticket. Thank you.

    Posted at: Aug 04, 2012 at 19:00

    Garett Murphy

    Greetings, We are aware of some possible connectivity issues for traffic going over our upstream provider NLayer at the CHI-2 Datacenter. This is related to an issue with the upstream provider's hardware and they are aware and working to resolve it. Please stand by for further updates regarding this issue. We apologize for the inconvenience.

  • 08/02
    Impact
    Severity
    Connectivity Issues CHI-2
    Status : CLOSED

    Connectivity issues at CHI-2- techs are aware and working to resolve it.

    Posted at: Aug 02, 2012 at 22:19

    Garett Murphy

    Greetings, The maintenance is completed. Thank you for your patience. Should you require further technical assistance, please open a support ticket. Thank you.

    Posted at: Aug 02, 2012 at 22:02

    Garett Murphy

    We are starting the scheduled maintenance for SnapShield device sfw01.ord03.singlehop.net. Please stand by for further updates. Thank you.

  • 07/29
    Impact
    Severity
    Connectivity Issues CHI-2
    Status : CLOSED

    Connectivity issues at CHI-2- techs are aware and working to resolve it.

    Posted at: Jul 29, 2012 at 08:27

    Mustafa Hussein

    The issue has been resolved, and connectivity has been verified. The impact of this hardware malfunction was minimal as the fail-over system kicked in. We will be working with our hardware vendor to determine the root cause of this brief outage. This status update is now closed. Thank you for you patience.

    Posted at: Jul 29, 2012 at 08:18

    Mustafa Hussein

    The hardware issue has been resolved, and we are working on verifying connectivity at this time. Please stand by for further updates. Thank you.

    Posted at: Jul 29, 2012 at 08:09

    Mustafa Hussein

    We are now working on bringing the faulty hardware back on line. This hardware malfunction may have impacted traffic flow briefly, some customers may have noticed a brief disruption. Please stand by for further updates. Thank you.

    Posted at: Jul 29, 2012 at 07:53

    Mustafa Hussein

    We have been alerted by our monitoring system to a malfunctioning hardware. We are working on resolving the issue and gathering more information. Please stand by for updates.

What do those colored bars mean?

Client Impact indicates how many customers are impacted by the issue. The higher the Client Impact score, the more customers that will experience this issue.

Severity indicates how impactful the issue is. Remember, the issue's impact is a factor of both Client Impact and Severity.

Some issues impact a small number of customers in a very bad way. Those issues would have a low Client Impact score but a high Severity score.

Issues that impact many customers but are minor, such as minor network latency, would be reversed. Those issues would have a high Client Impact score but a low Severity score. Issues that impact a large number of clients in a very impactful way would have two very high scores.

  • Tech Support: Toll-Free: 1-866-817-2811
  • Submit a ticket to LEAP
  • Click for automatic reload every 30 seconds