Ticket Number: HEA-NOC/20060519-1 Ticket Status: UPDATE
Ticket Type: unscheduled Resolver: HEA-NOC
Ticket Opened: 20060519 09:25 UTC+1 Problem Start: 20060519 03:30 UTC+1
Ticket Update: 20060522 09:55 UTC+1 Problem End:
Site/line: NUIG, Bluenet CW - GAL, Bluenet CW - KP
Monitoring shows that at approximately 03h30 this morning connectivity on
the backbone bluenet link between Galway and Limerick was lost. This link
carries no production traffic, but provides a backup link to NUIG.
Troubleshooting suggests that the issue is on the fibre link between
Galway and Limerick. ESB Telecoms have been contacted and we are awaiting
further information from them on the location and nature of the problem.
No impact on production traffic as this is a backup link.
20060519 09:04 LT Call logged with ESB Telecoms.
20060519 09:44 LT ESB Telecoms have checked their fibre between Galway and
Limerick and found no problems. E-net have checked their fibre in Galway
and there are no problems their either. A call has been logged with
Siemens to have them check the ADVA equipment, their call reference is
20060519 11:53 LT An outage has also been detected on E-net fibre within
Limerick. This fibre shares a common path, for part of its length, with
E-net fibre on the link between Limerick and Galway. Although E-net can
find no alarms on their systems they have sent out a crew to investigate.
In the meantime, Siemens are remotely checking the ADVA equipment. We have
asked them to apply software loops on the ADVA equipment in Limerick, to
allow us to try to identify whether connectivity from the Galway router to
that kit, and from the Limerick router to that kit, is available.
I have also edited the subject on this ticket, and the problem
description, to more accurately reflect the nature of the outage (i.e. a
bluenet backbone link outage rather than just an outage on backup link to
20060519 13:43 LT Problem still being investigated. We have lost inline
connectivity to the bluenet router in Limerick presumably as a result of
troubleshooting/testing by E-net (currently waiting for confirmation of
this from E-net). Siemens are still investigating the ADVA equipment for
possible issues there but the evidence strongly suggests a fibre break in
20060519 14:14 LT E-net have detected a likely fibre break on the UL
campus, between UL and the MAN. They are carrying out tests at present to
determine the exact location of the fault.
20060519 14:51 LT E-net believe they have narrowed down the problem to a
break in fibre between comms. rooms A and B on UL campus. They have found
a nest of rats in one of the chambers, apparently. They (E-net!) are
continuing to work on the problem.
20060519 16:48 OB e-net are experiencing difficulty opening some chambers
within UL which is hampering progress. Unable to provide a fix time as of
yet, but will provide out-of-hours updates.
20060519 17:03 LT E-net have reported that rats ate through the duct
containing the fibre, and through the fibre itself too. E-net are running
through new fibre and splicing it. They expect this work to take about 1.5
hours so they links should be live by 19h00 today.
20060519 19:20 OB e-net call to say that cable has been repaired. Verified
that Galway - Limerick link, and UL - LIT links are back up. Management
tools will report as still being down due to xenpak issue in Waterford.
20060521 09:45 EK Lancomms will deliver the replacement Xenpak to WIT
before luchtime on Monday.
20060522 09:54 COC
This ticket can be monitored at http://www.hea.net/tickets/20060519-1