How a brand-new firewall took down international company for 37 minutes — and why LACP told us everything was fine.
It started the way these things always start: monitoring lighting up with BGP neighbor flaps on a pair of edge routers. Not one neighbor — several. iBGP between the routers, eBGP towards external peers, all of it bouncing on hold-time expiry. And then, just to make the morning more interesting, HSRP decided both routers should be Active at the same time.
When everything on a router starts failing at once, my first instinct is: the router is probably fine. Something is starving it.
Reading the router logs
I pulled syslog from both routers and sorted the noise. Two things jumped out immediately.
