Troubleshooting AWS Direct Connect BGP session flapping issues
AWS Direct Connect provides a private path between an on-premises network and AWS, but the BGP session over that path can repeatedly establish and drop. This behaviour, commonly called BGP flapping, causes route churn, packet loss and intermittent access to workloads in a VPC or through a transit gateway.
A stable physical circuit does not always mean a stable routing session. The fault may sit in the customer router, the Direct Connect virtual interface, an intermediate carrier, a BGP timer mismatch or an AWS-side event. Effective troubleshooting separates these layers instead of repeatedly resetting the entire connection.
The most useful evidence is time-correlated: router logs, Direct Connect state changes, interface counters, carrier notifications and CloudWatch metrics. Record timestamps in UTC as well as local Australian time, particularly when teams in Sydney, Melbourne and Perth are coordinating with an overseas network operations centre.
The workflow below focuses on isolating the failure domain, validating the BGP configuration and applying changes that reduce the chance of recurrence in a hybrid environment.
What flapping looks like
A BGP session is flapping when it transitions repeatedly between Established and Idle, Active or Connect. AWS may show the virtual interface as available while the BGP neighbour itself continues to reset. On the customer router, messages such as hold-time expiry, connection reset, cease notification or TCP port 179 failure provide valuable clues.
Start by collecting at least 30 minutes of events from both sides. Check whether the resets occur at a regular interval, only during busy periods or immediately after route changes. A predictable interval often points to a timer or reachability problem, while irregular resets can indicate packet loss, a port fault or control-plane overload.
Check AWS and carrier status first
In the AWS Direct Connect console, inspect the connection, virtual interface and BGP status independently. Confirm the correct region, VLAN, customer router IP, AWS peer IP and autonomous system number. A private, public or transit virtual interface can remain configured while its underlying cross-connect or hosted connection is degraded.
Review AWS Health Dashboard events and the carrier’s maintenance notices. For Australian deployments, a Sydney cross-connect may use a separate provider from a Melbourne backup, while a Perth office can traverse a longer carrier path before reaching the AWS region. Ask the provider for circuit alarms, optical levels, maintenance windows and packet-loss measurements rather than accepting a generic “link up” response.
Validate the BGP configuration
Compare the AWS-provided configuration with the live router configuration line by line. Verify the neighbour address, local ASN, remote ASN, VLAN tag, MD5 password and source interface. An incorrect update-source, duplicated peer address or stale password can produce a session that briefly connects before being rejected.
Check that the advertised prefixes are valid and within AWS service limits. Confirm that the router is not announcing an unexpected default route, overlapping networks or a large collection of more-specific prefixes. Route policies should explicitly control what enters and leaves AWS, with sensible local preference and AS-path handling for primary and backup circuits.
Separate transport faults from routing faults
A continuous ping to the directly connected BGP peer can help distinguish an IP transport issue from a BGP policy issue, although it should not be treated as proof of end-to-end application health. Examine interface errors, CRC counts, discards, MTU mismatches, VLAN counters and link renegotiation events. A peer that remains reachable while BGP resets is more likely to have a control-plane, authentication or timer problem.
The following pattern is useful when correlating evidence:
| Observation | Likely area | Useful next check |
|---|---|---|
| Physical interface drops | Fibre, optics or carrier | Transceiver levels, alarms and circuit logs |
| Peer ping fails with interface errors | Layer 1 or Layer 2 | CRC, MTU, VLAN and switch port counters |
| Peer reachable, hold timer expires | Packet loss or timer handling | Capture control traffic and review CPU load |
| Immediate reset after Established | Authentication or policy | MD5, ASN, route maps and neighbour logs |
| Only one of two paths flaps | Path-specific fault | Compare carrier, optics and router interfaces |
| Flapping during route updates | Control-plane or policy issue | Prefix count, routing process CPU and limits |
Review BGP timers and keepalives
BGP relies on keepalive messages and a hold timer. If the router fails to receive an acceptable update or keepalive within the hold period, it removes the neighbour. AWS Direct Connect commonly uses conservative BGP timing, so a customer router configured with unusually aggressive timers can make a marginal circuit appear unstable.
Compare the negotiated timers shown by both routers and avoid changing them during a live incident without recording the original values. Check whether firewall inspection, router CPU starvation or a control-plane policing policy is delaying TCP port 179 traffic. If Bidirectional Forwarding Detection is enabled in the design, verify that both ends and the carrier path support the chosen detection interval; overly sensitive detection can turn transient loss into repeated route withdrawal.
Inspect device health and packet handling
High CPU or memory pressure can prevent a router from processing BGP messages even when forwarding traffic continues. Review process CPU, routing-table size, interrupt load, control-plane policing drops and recent configuration commits. Virtual routers running on VMware or other hypervisors also need sufficient vCPU scheduling and reliable virtual NIC performance.
Use a packet capture where permitted to inspect the TCP session between the customer router and AWS peer. TCP resets, retransmissions and missing keepalives narrow the investigation considerably. Check firewalls and inspection devices in the path for session ageing, asymmetric routing or policies that block TCP 179 after a period of inactivity.
Apply controlled recovery actions
Once the fault is understood, make one change at a time and preserve the logs before clearing counters. Correct an ASN, password, VLAN or route policy first; then reset the specific BGP neighbour rather than bouncing the whole router or Direct Connect connection. Confirm that routes withdraw and return as expected, and verify application paths through the relevant VPC, virtual private gateway or transit gateway.
For production services, use two Direct Connect connections or virtual interfaces with genuinely diverse equipment, facilities and carriers. A Sydney primary and Melbourne backup can improve resilience, but only if the underlying paths do not share the same exchange, provider or maintenance window. Test failover during an approved change window and document the expected BGP and route-selection behaviour.
Prevent recurring session resets
Create monitoring for BGP state, route count, interface errors, packet loss, latency and circuit availability. CloudWatch, router telemetry and carrier portals should feed a shared incident timeline. Alert on a state change and on repeated flaps, not just a permanently down session, because a cycling peer can be more disruptive than a clean failure.
Keep a current Direct Connect design record containing peer addresses, ASNs, VLAN IDs, timers, prefixes, route policies and escalation contacts. Include Australian carrier support details and maintenance time conversions between AEST and AEDT. Regularly review firmware, optics, configuration backups and failover tests so that a BGP incident becomes a bounded operational task rather than a prolonged search across multiple providers.