Ticket Number: HEA-NOC/20041029-3 Ticket Status: UPDATE
Ticket Type: unscheduled Resolver: HEA-NOC
Ticket Opened: 20041029 11:50 UTC+1 Problem Start: 20041029 11:37 UTC+1
Ticket Update: 20041029 14:44 UTC+1 Problem End:
Site/line: BH-Citywest GE (HEAN0015), BH-Citywest GE (HEAN0104),
BH-Cork GE (HEAN0117), BH-Kilcarbery GE ( HEAN0106), Boston
College (DS0882863), Boston College, Crumlin VEC, DCU,
DIAS, DIT, EPA, Fulbright, HEA, HEA-NOC, HETAC, HRB,
Hermes, IADT, ITnet, Kilester VEC, Liberties VEC, MLE,
NCAD, NCIRL, NCRI, NLI, NQAI, NUI, NUIM, RCSI, RIA, TCD,
UCD, Whitehall VEC
Cisco GSR supervisor crash.
20041029 11:43 JL
Hyperion, the core Dublin GSR router crashed. This caused an outage to all
traffic transiting the Dublin PoP as it switched over to the backup
Investigating cause of outage.
20041029 11:59 AH Another, lesser interruption to service on the same
router occurred at 11:57. HEAnet continue to investigate crash
20041029 12:14 AH Problem recurred at 12:12. Escalated to Cisco TAC for
20041029 12:30 AH Problem recurred at 12:29. Following discussion with
Lancomms, HEAnet are removing a supervisor card to try to stabilise the
20041029 12:46 AH Supervisor card removal fails to stabilise the router.
Further repeat at 12:46
20041029 13:00 AH Further failures at 12:56. HEAnet are preparing to
reroute gigabit ethernet connections between Brooklawn House and Citywest
via an alternative router, pending some additional troubleshooting.
20041029 13:09 AH Lancomms/Cisco report faults in multicast code
triggering memory errors which trigger bus errors, causing the supervisor
cards to fail. Multicast has been disabled on this router as a
20041029 13:30 AH Interval between failures is longer but problem recurred
at 13:27. Plan for repatching backbone links remains in place as
resolutions suggested have not been conclusive.
20041029 14:42 AH Router remained stable unil c. 14:29. Multicast has now
been disabled on the interface of connecting routers.
This ticket can be monitored at http://www.hea.net/tickets/20041029-3