Ticket Number: HEA-NOC/20040519-11 Ticket Status: UPDATE
Ticket Type: unscheduled Resolver: HEA-NOC
Ticket Opened: 20040519 12:11 UTC+1 Problem Start: 20040519 12:00 UTC+1
Ticket Update: 20040805 09:40 UTC+1 Problem End:
Site/line: Boston College, DCU, DIAS, DIT, EPA, Fulbright, HEA,
HEA-NOC, HETAC, HRB, IRCSET, ITnet, MLE, NCAD, NCIRL, NLI,
NQAI, NUI, NUIM, RCSI, RIA, TCD, Teagasc, Tipp Inst, UCD
There is intermittent IPv6 packet loss on the network backbone.
Causes netsaint to log errors as packets are dropped.
20040519 12:08 JL
Investigations have shown that packet loss is indeed present and is
unlikely to be a problem with netsaint. First change to be made was ICMP
error interval which was added to all routers. Errors still there. During
investigation it was noticed that IPV6 CEF wasnt switched on on many of
our routers so stopping forwarding in FIB and subsequently hardware.
Preparing list of routers which need CEF enabled.
20040520 12:10 JL
IPv6 CEF needs to be enabled on Andel, Salinger, Leabhair, Tempus &
Lucien. Will add IPv6 cef to these routers on Wednesday 26th 09:00 during
20040524 12:22 JL
IPv6 CEF to be enabled on all routers listed, should not be service
20040526 09:11 JL
IPv6 CEF enabled on the listed routers. Monitoring performance.
20040611 10:14 JL
Enabling IPv6 has not fixed the problem. Further investigation has showed
that each of the links being pinged is taking a similiar route through the
Netsaint alarms are also occurring for the three problem links in 12 hour
intervals. Continuing to investigate.
20040615 10:36 CMC More ping6 alarms observed, however experimentation
reveals that a traceroute6 brings the path alive again however a ping6
does not. ICMP with incrementing ttl's was tried and also had no effect,
looks like a genuine UDP Vs ICMP difference.
20040616 17:17 CMC More ping6 alarms observed. Confirm UDP Vs ICMP issue
using "nc6 -u". Verbose mail sent to [log in to unmask]
20040716 16:46 JL
Lancomms still investigating issue. Possible IOS upgrade needed on Dublin
20040720 11:10 CW escalating case after discussions with Cisco
20040723 14:15 CW making config changes under direction from Cisco TAC
20040723 14:46 CW Cisco identify bug (CSCec15559), due to linecard resets
some disruption to connectivity in the Dublin POP but traffic did divert
via the backup UBR path to Citywest.
20040730 09:53 BN Cisco have identified 12.0.28S as the relevant IOS to
install as this is the latest GSR IOS and should have a fix for the listed
bug. This install will take place at 08:00 UTC +1 on Wednesday August 4
2004. As the router will be reloaded during this outage all customers
that connect to the Dublin PoP will reroute with reduced performance.
While the maintenance window is from 08:00 to 09:00 the actual routing
change is expected to last for c. 5 minutes.
20040804 09:14 BN IOS upgraded and all BGP & OSPF sessions came back as
expected. Some brief outages were observed, but router now appears to be
stable and running 12.0.28S. This situation will continue to be
20040804 17:19 BN Flaps during maintenance caused ongoing outage on one of
two links between hyperion and main access router mantova. This situation
has now been remedied and no traffic was affected.
20040805 09:39 BN While the IOS upgrade has removed any possible problems
relating to the SNMP bug it does not appear to have solved the problems
with IPv6 traffic. This will be referred back to the Cisco TAC.
This ticket can be monitored at http://www.hea.net/tickets/20040519-11