Ticket Number: HEA-NOC/20050413-3 Ticket Status: CLOSE
Ticket Type: scheduled Resolver: HEA-NOC
Ticket Opened: 20050413 10:32 UTC+1 Problem Start: 20050414 08:00 UTC+1
Ticket Closed: 20050426 16:50 UTC+1 Problem End: 20050425 09:00 UTC+1
Site/line: Citywest PoP, Cork PoP, DCU-POP PoP, Dublin PoP, Galway PoP,
Kilcarbery Park PoP, Limerick PoP
A TCP/IP vulnerability has recently been released. Exploitation of this
issue could degrade or effectively cease traffic flow between sites on the
For more information on this vulnerability please see:
The recommended fix for effected systems is to upgrade the Cisco IOS or
This issue could degrade connectivity or cause some applications that use
TCP across the network to time out. The Juniper M20s in Cork and New York
are running versions of JUNOS not affected by this exploit. However
multiple Cisco systems will need to be upgraded. This will require a
reload of several routers across the network and brief (c. 10 mins)
customer outages will be unavoidable as PoP equipment is upgraded. It is
envisaged that these reloads will take place over the next week and a
timetable will be circulated.
20050413 10:14 BN Currently identifying correct upgrade path for affected
systems and compiling timeline for upgrades. This work will start at
08:00 on Thursday 14th April. Sites that will be affected will be
contacted as soon as possible.
20050413 14:47 BN A timeline has been organised to minimise customer
impact. The upgrades will take place over the next week, starting
tomorrow at 8am.
Thursday 14th April 08:00 - 10:00
There should be no customer impact, although some connections may be
degraded earlier in the morning and customer connectivity should be
considered at risk.
Outward connectivity to Geant, Janet and Global crossing will lose
during this window. All of these circuits will see BGP peerings go up and
Connection will also be lost to our New York PoP while Deimos.hea.net is
reloaded. This will affect connectivity to Sinet, CAnet, Abilene, MANLAN
Hyperion - Brooklawn House PoP Core Router
Deimos - Citywest PoP Core Router
Luna - Kilcarbery Park Core Router
Phobos - Citywest PoP Core Router
Alexander - Old Dub-Gal redundant link - Shut down
Hannibal - Old Dub-Gal redundant link - Shut down
Monday 18th April 08:00 - 09:00
There should be no customer impact, although some connections in Limerick
and Galway may be degraded earlier in the morning. This connections
should be considered at risk.
Rhea - UL PoP Core Router
Titan - NUIG PoP Core Router
Tuesday 19th April 08:00 - 09:00
Dublin Customers will have a brief outage during these upgrades. The
outages are expected to last no more than 10 minutes and will happen prior
Andel - Dublin sub-2Mb serial router
Mantova - Dublin atm/ethernet access router
Wednesday 20th April 08:00 - 10:00
Customers in Cork, Limerick and Galway will have a brief outage during
these upgrades. The outages are expected to last no more than 10 minutes
and will happen prior to 10am. HEAnet will endeavour to reload each
router as early as possible during this window.
Leabhair - Cork customer access router
Tempus - Limerick customer access router
Lucien - Galway customer access router
Lilith - INEX v4 router
Thursday 21th April 08:00 - 10:00
Hermes and RIFTTFS customers will have a brief outage during these
upgrades. DCU's primary circuit will also be down. The outages are
expected to last no more than 10 minutes and will happen prior to 10am.
HEAnet will endeavour to reload each router as early as possible during
Kuiper - DCU PoP access router
Spike - TCD PoP access router
Schiphol - IPv6 Tunnel router
Olympus - Experimental router
20050414 10:10 BN Maintenance Wednesday 14th completed without undue
outage. Four routers (Phobos, Deimos, Hyperion & Luna) upgraded as
planned. Old 34Mb Citywest - Galway link shutdown.
20050418 08:42 AH Rhea and Titan (Limerick and Galway core) upgraded to
plan. Traffic observed to reroute via ATM UBRs. Galway reloaded
08:08-08:11 and Limerick 08:31-08:33.
20050419 08:26 OMG Mantova and Andel upgraded according to plan. Mantova
reloaded at 07:56 - 08:01. Andel 08:53 - 08:57.
20050420 08:40 SOB Maintenance for wed 20th completed. Upgrade for lilith
postponed due to lack of space in the memory for the new IOS. leabhair,
lucien and tempus all returned to working order and all alarms cleared.
However the command sh isis neighbor no longer workes in the new IOS.
20050421 09:23 BN All work went as planned this morning. Olympus
(experimental router) was not reloaded due to other ongoing issues.
Kuiper, Spike and Schiphol were reloaded sucessfully. However due to
other issues the three PoP access routers in Cork, Limerick and Galway
will need to be reloaded before 9am on Monday 25th April. This will
cause an outage to UCC, NCRI, UL, NUIG, CAO and St Angelas. These outages
will last no more than five minutes. Affected institutions will be
20050421 10:44 COC contacted the main technical contacts for CAO, NUIG and
St Angela's College by phone to inform them that the outage would occur.
None of the contacts reported any issues with this planed outage.
20050425 08:44 DC All three PoP access routers reloaded successfully.
20050425 16:41 BN As all scheduled routers have been reloaded this work is
considered complete. Situation will be monitored for a further twenty-four
hours before closing ticket.
20050426 16:50 Auto Closed after 1 day.
Time to Fix:
This ticket can be monitored at http://www.hea.net/tickets/20050413-3