Ticket Number: HEA-NOC/20060926-3 Ticket Status: UPDATE
Ticket Type: unscheduled Resolver:
Ticket Opened: 20060926 11:52 UTC+1 Problem Start: 20060926 09:55 UTC+1
Ticket Update: 20061205 11:39 UTC Problem End:
Site/line: Crumlin VEC, Kilester VEC, Liberties VEC, NQAI, RIA,
Nagios reported ping problem with RIA, NQAI Crumlin VEC, Liberties VEC,
Whitehall VEC and Kilester VEC. Kuiper has not show any circuit
Issues with serial connections to RIA, NQAI, Crumlin VEC, Liberties VEC,
Kilester VEC and Whitehall VEC.
20060926 11:48 JH Call NQAI and they reported having slow access but did
not notice any outage.
20060926 11:55 BN Eircom report no known work or issue at the time, but
they will investigate. DCU are unaware of any issue on their network that
would cause packet loss, but they will also investigate. The PoP Access
router showed no signs of high CPU usage or any other errors. In addition
the packet loss was measured from two separate points on the network.
Everything was back by 12:01 but we will continue to monitor and
20060926 12:26 BN Problems merely paused in reporting at 12:01. Issue has
been extant on the serial connections since 09:55 this morning and packet
loss of between 10% - 20% is being recorded on all six circuits.
20060926 13:40 AH Eircom are unable to detect a problem. NQAI link was
brought down to test, as their ticket was looged as 'line down', almost
the opposite of what they were told at ticket logging time. They will keep
all tickets open until CoB.
20060926 17:07 BN No packet loss has been detected since around 13:00,
however some extreme changes in latency are still being detected and in
order to check the lines the link to the RIA will be taken down for
testing at 22:00. The line will remain down for c. 1 hour.
20060927 16:05 BN No errors were detected on the line to the RIA during
eight hours of testing. No similar levels of packet loss have been
detected since yesterday but monitoring and investigation will continue.
20061117 10:34 DW Smokeping and nagios report packet loss on the above
circuits (excepting Liberties VEC, currently the subject of a scheduled
outage) starting 10:00 today. Edited to add: faults reported to telco for
NQAI (#0838), RIA (#0839), Whitehall CFE, Killester VEC, Crumlin VEC (all
20061121 16:16 PC - Packet loss was seen on most of these circuits (all
except Killester) between 15:10-15:13 today, while the Liberties router
was rebooted. The Liberties CDVEC link was also disrupted on the 17th due
to recabling work carried out there and that also appears to have resulted
in packet loss on the links to the other sites. Eircom have checked the
circuits and do not believe that the lines are at fault, Lancomms are
20061122 12:55 PC - Lancomms believe that this may be a clocking issue
caused by the serial interface card using the clock from a single port on
all the lines. They are currently checking this with cisco.
20061125 09:11 JH Continuing problem with serial connections. Interface
flap on Crumlin VEC serial line at 2:32:01 to 2:32:37. Whitehall CFE
serial line is down. I will continue to investigate.
20061125 09:38 JH I have logged a call with BT Ireland about the Whitehall
line. Their ticket reference is 1-400736015
20061125 12:12 JH Whitehall CFE's serial line came back online at 09:54.
No call back yet from BT Ireland.
20061125 13:40 JH BT Ireland that the outage was caused by a power outage
in the area.
20061125 15:06 JH Further interface flaps on Crumlin VEC serial line
between 13:47:01 and 15:19:31.
20061125 18:08 JH Further flaps on the line, but Crumlin VEC's serial line
has been stable since 16:39:45. I will continue to monitor.
20061205 11:38 BB David Bannon of Lancomms requested a contemporary show
tech. Sent it on, and advised that the router has been renamed since this
case was opened.
This ticket can be monitored at http://www.hea.net/tickets/20060926-3