Ticket Number: HEA-NOC/20080522-4 Ticket Status: CLOSE
Ticket Type: unscheduled Resolver: HEA-NOC
Ticket Opened: 20080522 14:29 UTC+1 Problem Start: 20080522 08:03 UTC+1
Ticket Closed: 20080530 11:20 UTC+1 Problem End: 20080529 08:20 UTC+1
Site/line: Kilcarbery Park PoP
Due to issues encountered while upgrading the new core router in our
Kilcarbery PoP (cr2-kp) the system has been effectively off-line since
08:03 this morning. This has removed resiliency from the new IP network.
The progress of the resolution can be tracked in ticket 20080512-8 here:
No client outages have been caused, however the majority of clients will
have lost a level of resilience due to this outage. In a large number of
cases there are still two to three redundant circuits in place.
20080522 14:24 BN Opened ticket to reflect large issue caused by the
upgrade tracked in 20080512-8.
20080523 11:00 BN Cisco developers have been debugging this issue through
the night and have discovered a route injection corruption issue and
detailed a workaround. During the course of the upgrade a hardware fault
was also found with one of the router linecards, so this will be replaced.
It is expected that the router will be returned to full production before
17:30 today, Friday 23rd.
20080523 13:37 TG Replacement hardware is expected to be installed later
today. We will then perform further testing on Monday to confirm both
hardware and software issues are resolved, before being the device back
into full production.
20080526 10:02 BN Replacement hardware has been successfully installed.
The INEX LAN2 connection has been brought back up and this will be
20080528 11:26 TG We are scheduling another maintenance window for
tomorrow, Thursday 29th between 07:00 and 09:00. This is to complete the
router maintenance which was not completed last week due to the hardware
issue. Connections to INEX #2 will be effected, but all traffic will
re-reoute via INEX LAN #1. HEAnet customers will experience a loss of
resilience during this window.
20080529 08:24 BN This upgrade was completed successfully. Outages
occurred as expected during the window. The router will be monitored for
at least twenty-four hours before this ticket is closed.
20080530 11:20 BN Router has remained stable for over twenty-four hours.
Resilience considered to have been fully restored.
Time to Fix:
This ticket can be monitored at http://www.hea.net/tickets/20080522-4
HEAnet Limited. Registered in Ireland, No. 275301.