Ticket Number: HEA-NOC/20020501-01 Ticket Status: UPDATE
Ticket Type: Unscheduled Ticket Source: HEA-NOC
Ticket Scope: Site Site/Line: ATM Client Access
Ticket Owner: [log in to unmask] Problem Fixer: HEA-NOC
Ticket Opened: Problem Start: 20020501 11:32 UTC+1
Ticket Closed: Problem End:
The ATM access router (uther.hea.net) not routing.
BGP route explosion seen by a number of other ISPs at 11:30 UTC+1
Traffic to and from Dublin ATM access customers: ucd, tcd, dias, dcu, dit, hea, itnet, nuim. Backbone link to Citywest.
20020501 11:35 UTC+1 investigate cause and switch clients to backup router (Teleute.hea.net)
20020501 11:55 UTC+1 Clients are now routing through teleute, access
20020501 12:25 WD Fault tracked to card on core router Miranda, isolating ATM access router Uther. Contacted Lancomms to report fault. ATM client access re-routes via alternative router at reduced capacity. Backbone link to Citywest re-routes via Galway.
20020501 15:45 WD: GigEthernet card reloaded on Miranda. Reconnected ATM access router Uther. OSPF updated ok. Original ATM client access routes and capacity restored (via Uther).
20020501 16:52 UTC+1 CW GE card in miranda crashed at 18:39 reload card at 18:49
20020501 19:10 UTC+1 CW ClientPVC's have been rerouted overnight as an intrim measure
20020502 10:35 UTC+1 CW Reloaded card at 10:31 crashed at 10:35. additional data captured and problem now with TAC. Still using backup routes.
2002050216:20 AH Case is being passed to Cisco US for further investigation. Possible intervention on the core router Miranda may be required throughout the evening as part of the investigation process, which may briefly interrupt service. Clients remain routed through the backup solution pending resolution of this case.
20020502 20:30 AH Cisco require the event to recurr to confirm diagnosis but have suggested a possible alternative IOS. Following a suitability check, a time for this work may be confirmed for as early as 3/5/02, if necessary. Clients remain on backup solution for stability.
20020503 10:47 AH Fault did not occur overnight but the situation is still 'at risk'. Therefore, emergency upgrade to recommended IOS is scheduled for 11:30. There will be a brief interruption to service for Dublin clients, expected to be no longer than 10 minutes.
Time to Fix: