Re: [HEAnet #1528346] Faults List: National and Dublin core upgrades

Ticket Details

Ticket ID 1528346
Subject Re: [HEAnet #1528346] Faults List: National and Dublin core upgrades
Status open
Date Start Mon Feb 25 23:00:00 2019
Date End Fri Mar 01 03:00:00 2019
Scheduled YES

Problem Description

We are upgrading all MX480s on both National and Dublin core. Monday 25th of Feb 23:00 -> 03:00 ---------------------------------- core2-dcu core2-tcd core1-gal1 Tuesday 26th of Feb 23:00 -> 03:00 ---------------------------------- core1-tcd core2-ucd core1-cork2 Wednesday 27th of Feb 23:00 -> 03:00 ------------------------------------ core1-ucd core1-dcu core1-cork1 core1-lim2 Thursday 28th of Feb 23:00 -> 03:00 ----------------------------------- core1-gal2 core1-lim1

Problem Effects

All of the listed clients will see one of their two BGP sessions with AS1213 go down during reload. As we will only reload one at a time, you should not see a loss of connectivity. You will need to verify you are receiving 2 default routes on your routers form AS 1213, to ensure connectivity: sh ip bgp 0.0.0.0 [Cisco] show route protocol bgp 0.0.0.0/0 [JunOS] $user@$router> sh ip bgp 0.0.0.0 BGP routing table entry for 0.0.0.0/0 Paths: 2 available 1213 193.1.9.197 from 193.1.9.197 (193.1.9.197) Origin IGP, metric 5, localpref 100, IGP metric 1, weight 0, received 3d12h ago, valid, internal, best 1213 87.44.56.113 from 87.44.56.113 (87.44.48.36) Origin IGP, metric 10, localpref 100, IGP metric 1, weight 0, received 3d12h ago, valid, external The metric of 5 and 10 are important. If you have any queries please contact [log in to unmask]

Last Update Message

Tue Feb 26 01:06:38 2019

core2-dcu
core1-gal1 were upgraded ok.
We have an issue with core2-tcd, the downstream router edge2-tcd
had a technical issue when core2-tcd was rebooted, and routing process needed to be killed. Consequently it wasn't accepting traffic from the resilient path via edge1-tcd.
The following sites had a 15min approx outage while we troubleshooted the issue.
edge1-esri
edge1-ncad1
edge1-heanet-gd5
edge1-ucd-svuh
edge1-ichec
edge1-dit-mountjoy
edge1-revenue
edge1-tcd-stjames
edge1-depot
edge1-dit-bolton
edge1-tcd-stackb
edge1-oir
edge1-dit-chatham
edge1-ucd-eccles
edge1-tcd-ent
edge1-ndrc
edge1-educampus
edge1-ria
edge1-depot
edge1-ichec
edge1-ndrc
We plan to further investigate the issue tomorrow.
Please contact [log in to unmask] or the Service Desk on 01-6609040 for more information.

Affected Clients

DCU, NUIG, SCHOOLS, TCD, UCC, UCD, UL, RCSI

Affected Services

IPv4 Connectivity
IPv6 Connectivity

Transaction History

Date Created Message
Mon Feb 18 14:17:06 2019 Security patches
Mon Feb 18 14:21:39 2019 Link to public ticket: https://ticketing.heanet.ie/public/show.html?id=1528346
Mon Feb 18 14:42:18 2019 Updated date and time.
Tue Feb 26 01:05:56 2019 core2-dcu core1-gal1 were upgraded ok. We had an issue with core2-tcd, the downstream router edge2-tcd had a technical issue when core2-tcd was rebooted. A routing process needed to be killed. Consequently it wasn't accepting traffic from the resilient path via edge1-tcd. The following sites had a 15min approx outage while we troubleshooted the issue. edge1-esri edge1-ncad1 edge1-heanet-gd5 edge1-ucd-svuh edge1-ichec edge1-dit-mountjoy edge1-revenue edge1-tcd-stjames edge1-depot edge1-dit-bolton edge1-tcd-stackb edge1-oir edge1-dit-chatham edge1-ucd-eccles edge1-tcd-ent edge1-ndrc edge1-educampus edge1-ria edge1-depot edge1-ichec edge1-ndrc We plan to further investigate the issue tomorrow. Please contact [log in to unmask] or the Service Desk on 01-6609040 for more information.
Tue Feb 26 01:06:38 2019 core2-dcu core1-gal1 were upgraded ok. We have an issue with core2-tcd, the downstream router edge2-tcd had a technical issue when core2-tcd was rebooted, and routing process needed to be killed. Consequently it wasn't accepting traffic from the resilient path via edge1-tcd. The following sites had a 15min approx outage while we troubleshooted the issue. edge1-esri edge1-ncad1 edge1-heanet-gd5 edge1-ucd-svuh edge1-ichec edge1-dit-mountjoy edge1-revenue edge1-tcd-stjames edge1-depot edge1-dit-bolton edge1-tcd-stackb edge1-oir edge1-dit-chatham edge1-ucd-eccles edge1-tcd-ent edge1-ndrc edge1-educampus edge1-ria edge1-depot edge1-ichec edge1-ndrc We plan to further investigate the issue tomorrow. Please contact [log in to unmask] or the Service Desk on 01-6609040 for more information.