Ticket Number: HEA-NOC/20100420-2 Ticket Status: UPDATE
Ticket Type: scheduled Resolver: NUIG
Ticket Opened: 20100420 09:56 UTC+1 Problem Start: 20100611 22:00 UTC+1
Ticket Update: 20100612 03:11 UTC+1 Problem End: 20100612 02:45 UTC+1
Site/line: CARRAROE, CASTLEBAR, DERI, GTI, IT Athlone, IT Sligo, LYIT,
LYIT-Killybegs, Marine Institute, MARTINRYAN, mhars, NIRAN,
NUIG, NUIG Clinical Sciences, NUIG-Gweedore, PAC, Schools
Network, Sean, St. Angela's
NUIG are expanding their Data Centre. Installation of a new 160KVA UPS
requires a new module to be added to the main Data Centre power
distribution board. This board needs to be powered off in order for the
modification to be made.
UPS and Generator and therefore all power to the data centre, including
HEAnet's PoP, will not be available during the works. The work will take
place over 8 hours, from 09.00-17.00 on Saturday 12th June 2010.
Loss of resiliency
Loss of Connectivity
GMIT (and all sub-sites)
NUIG (and all sub-sites)
-Open Learning Centre
-Sean Scoil Shailerna
-Aras Shorcha Ni Ghuairim
-Aras Mhairtin Ui Chadhain
20100420 09:29 BMA Opening ticket to track work. Clients will be contacted
individually to ensure date of outage is acceptable.
20100511 14:48 CS Since this outage is a problem for IT Sligo and the work
cannot be easily re-scheduled, NUIG will arrange for HEAnet kit to be
powered by the mains from nearby offices during the maintenance work
thereby avoiding a potential 8-hour long outage.
However, as not all of the HEAnet kit has dual power supplies, this will
result in two short outages: the first when the power supply is switched to
the temporary supply before maintenance begins and the second when
switching back after the work is complete.
We are currently waiting to confirm outage times with IT Sligo.
20100520 15:25 BN This work has now been scheduled to commence at 22:00 on
Friday 11th June. A short, ~3m, outage will take place just after 22:00
when replacement power will be put in place. Full power supply will be
swapped back in before 08:00 on the 12th with another short outage.
20100608 15:29 GL Work is to commence as planned on Friday 11th June at
22:00. However, full power supply will now be swapped back on the morning
of the 13th.
20100610 18:49 GL Updating ticket to reflect new end time
20100611 20:40 DL, power loss for 3 minutes at 20:01 as result of
preparation for work tonight. HEAnet systems recovered after power
20100612 00:09 DL, at 21:50 HEAnet systems disconnected from main power
supply and connected to alternative power supply with failed result since
alternative power supply was linked to the main. NUIG/HEAnet moves
equipment to different alternative power supply but fails due to overload.
NUIG/HEAnet connect systems to a third alternative power supply and restore
power. Systems powered up again at 23:50 and connectivity restored in
NUIG will be performing testing, originally scheduled during the day of
Saturday 13 June, during the night 12-13 June. Currently awaiting for
electrical engineer to arrive in NUIG, expected at 01:00 and perform
testing upon arrival. Once completed, HEAnet systems will be restored to
original power supply and remain as before the work had started.
20100612 00:20 DL, times and events
22:00 Switch equipment to alternative power
22:15 UPS Cut - equipment remains up
22:29 Main power cut
22:30 Alternative Power Failed
22:57 Alternative Power restored - another alternative power sourced
23:05 Alternative alternative power failed
23:20 Alternative alternative power restored - too much load on one
breaker, split to two breakers.
00:00 Alternative alternative power stable for 40 minutes.
However not confident of stability. NUIG are to carry out emergency button
test in estimated 2 hour at 02:00 instead of 8am to 9am. HEAnet engineer
will remain onsite and then restore HEAnet equipment to mains power and
return to normal.
20100612 02:08 GL NUIG have tested the emergency button test. Moving
HEAnet equipment back onto mains power. An outage is not envisaged as UPS
is back up.
20100612 02:30 GL Full connectivity restored and verified. Outage observed
to devices without dual power supply. No more outages expected over the
20100612 03:07 GL Just to note that NUIG themselves, both priamry and
secondary, remain down. They ae aware of this and will turn on their CPEs
Time to Fix:
This ticket can be monitored at http://www.hea.net/tickets/20100420-2
HEAnet Limited. Registered in Ireland, No. 275301.