Ticket Number: HEA-NOC/20060106-3 Ticket Status: UPDATE
Ticket Type: scheduled Resolver: HEA-NOC
Ticket Opened: 20060106 14:51 UTC Problem Start: 20060106 11:00 UTC
Ticket Update: 20060228 09:47 UTC Problem End:
Site/line: INEX, Schools Network, JANET GE Secondary, Kilcarbery Park
Due to continual errors on the line card in Slot 0 on the Kilcarbery Core
Router it has been decided to replace the card.
There may be a brief outage to Route Reflector 2, the Kilcarbery LAN and
one of the redundant connections to the Schools Network, Janet and the
INEX. These outage should last no more than 2 minutes.
20060106 14:47 BN Arranged for replacement card to be shipped to
20060110 13:08 BN Replacement card was inserted, but multiple errors
resulted in old card being left in place. This prolonged outages on
affected circuits by up to one hour. This work will be
rescheduled/altered after consultation with Lan Communications.
20060112 16:39 BN This work has been rescheduled to Monday 16th January at
20060116 17:22 BN New line card was inserted into the router but similar
crashes occured. The fault is now believed to be the 3port GE daughter
card and this will be replaced on Thursday 19th January. As new cards
will have been inserted into the GSR at that stage any disruption should
be kept to a minimum.
20060119 17:14 BN Further testing of new linecard produced errors that
have been referred to support providers. However as no connections are on
this card there was no disruption.
20060131 14:45 BN Linecard testing is ongoing and the next phase will
begin at 11:00 on Wednesday 1st February. While no production cards will
be touched the router should be considered at risk for the two hour
20060201 17:17 BN The testing today was far more intrusive than originally
expected and there were multiple instances of the router losing
connectivity to all peers. It has been returned to a stable state, but
more work - at the very least an IOS upgrade - will be required. Peers
will be informed of the timetable for this work as far in advance as
20060214 09:39 BN Cisco have indentified a series of tests to be carried
out on this router. A testing window will be identified and all relevant
20060223 14:48 BN This testing window with Cisco has now been identified
as 18:00 - 20:00 UTC on Monday 27th February. In order to ease the
interuptions caused by this work we will be shutting down the circuits
landing on the router. This will mean that resiliency will be lost for
the Schools Network, Janet (v4 & v6) and INEX v4 traffic, the primary
connections will still be in place. All of our INEX transitting v6
traffic uses this link without backup, so this traffic will cease for the
duration and reroute via the General Internet where possible. Route
Reflector 2 will also be unavailable. Cisco assure us that this testing
will produce an answer that will point to a final fix for this issue.
20060227 16:54 BN OSPF and ISIS metrics on the backbone have been
temporarily recosted to avoid interuptions to the traffic flow. External
peerings will begin to be shut down from 17:00 and restored once the work
20060228 09:46 BN This testing was completed by 21:00 last night and all
external peerings were restored. OSPF and ISIS metrics on the backbone
will be returned to normal this morning. Debug information is currently
with Cisco and we are awaiting further information.
Time to Fix:
This ticket can be monitored at http://www.hea.net/tickets/20060106-3