Ticket Number: HEA-NOC/20080320-1 Ticket Status: CLOSE
Ticket Type: unscheduled Resolver: HEA-NOC
Ticket Opened: 20080320 08:52 UTC Problem Start: 20080320 00:09 UTC
Ticket Closed: 20080409 17:06 UTC+1 Problem End: 20080409 17:00 UTC+1
Coroebus (the mirror server) became unresponsive at approx 00:09 last
ftp.heanet.ie unavailable. Mirror content is unaffected.
20080320 08:50 RG Open ticket to monitor
20080320 12:40 RG It appears that 3 drives have failed - the two system
disks in the machine itself and one drive in the Powervault attached to
coroebus. As a result the system is not bootable until some replacements
can be sourced from Dell.
20080320 17:04 RG After talking to Dell support - it appears the 2 system
drives are not at fault. Removed and re-added each drive into the RAID1
set, the system now boots but gets wedged immediately after it loads the
20080321 09:38 LT Failure of a hardware component remains the most likely
cause of the problem, but further troubleshooting will be required to
isolate the problem. It is likely that the server will remain unavailable
until this work can be undertaken starting on Tuesday next (March 25th).
The content served/mirrored by the server may be retrieved from other
mirror servers in the meantime.
20080324 17:17 LT Following further remedial work, the server has been
available again since 16h44 running an older kernel version. Further
investigation of the core problem, and a longer term solution, will
20080325 09:25 RG ftp.heanet.ie will require a further reboot in order to
put the second disk in its RAID-1 mirror back into service, and to attempt
to (re)load the newer kernel. This reboot will be scheduled for tomorrow
morning at 09:15.
20080326 10:11 RG After re-enabling the second disk in the RAID-1 mirror,
it appears that the mirror has become corrupted. Investigating further.
20080326 10:57 RG Installed two replacement hard drives into coroebus and
rebuilt RAID-1 mirror. Currently re-installing the OS.
20080326 16:07 RG Setup a base install of Debian etch, currently
20080327 15:11 RG Most services have been re-enabled:
Will keep ticket open to document any further issues.
20080401 10:27 RG It appears ftp.heanet.ie has not been listed as a
sourceforge mirror since the downtime. Contacted sourceforge in order to
20080402 12:22 RG Public key for the sourceforge "syncuser" wasn't
present. Restored this along with their home directory from backups.
20080409 17:05 RG Closing ticket. Service is stable and sourceforge have
added us back into their mirror rotation.
This ticket can be monitored at http://www.hea.net/tickets/20080320-1
HEAnet Limited. Registered in Ireland, No. 275301.