• Increase font size
  • Default font size
  • Decrease font size
  • default color
  • cyan color
  • red color

ONLINE STATUS

Incident report: Network Outage - 5th August 2013
Dear customer,

The report below is a breakdown of the incident that was experienced across our network between approximately 10h00 and 10h45 on Monday 05th August 2013.

The fault was traced to one of edge routers.

Breakdown of the events:
At approximately 10h00 August 2013 we became aware of loss of service for customers in subnet 80.92.84.0/24, which appeared to be spread across all sites and services. Our systems and network team were immediately engaged to aid with the diagnostics and resolution.

Initial investigations indicated that whilst we were able to see that the network was available internally and we could contact individual devices on the core, any layer 3 traffic for 80.92.84.0/24 was not flowing into and out of our backbone.

The cause of the issue was traced at approximately 10h40 when our engineers identified a TCAM memory overrun in one of our edge router logs. This indicated this edge routers were not able to handle the OSPF routes, hence the overrun.

The routing process was migrated to another router and services were fully restored at 10h45.

Effects seen:
- IPv4 Layer 3 traffic into and out of our edge network 80.92.84.0/24 was unable to pass

Actions taken:
The immediate action taken to rectify this problem was to take out the defective edge router. This is now in place and will remain indefinitely.

Summary:
We understand that this was a major incident for us and our customers that rely on our network. We sincerely apologize for the incident and reassure you that it is being dealt with the greatest priority.