BGP and OSPF Explained: Advanced Routing Protocols for Network Engineers

Most engineers can recite that OSPF is a link-state IGP and BGP is a path-vector EGP within their first year on the job. That definition is useless the first time you're debugging a route flap across three ABRs, or trying to figure out why a customer route is taking the long way around your backbone despite a "better" path existing on paper. At that point you need to know how these protocols actually make decisions, not what category they belong to.

This article assumes you've already deployed single-area OSPF and peered a basic eBGP session. We're going past that: multi-area design, the full BGP best-path algorithm, route reflectors, and where MPLS and SD-WAN actually sit in relation to all of it. If you run or design production networks — enterprise WAN, data center fabric, or service provider edge — this is the layer you need to operate at.

OSPF at scale: areas, LSAs, and route summarization

Single-area OSPF works fine until it doesn't. Every router in an area carries the full link-state database for that area and runs SPF (Dijkstra) against it. Push enough routers and prefixes into one area and you get SPF runs that are slow, CPU spikes on every topology change, and a database that's expensive to keep synchronized. Multi-area OSPF exists to bound that blast radius.

The design is hierarchical by requirement, not convention: area 0 (the backbone) must exist, and every non-backbone area must connect to it, either directly or via a virtual link when physical topology doesn't cooperate. Routers that sit on the boundary between area 0 and another area are Area Border Routers (ABRs); routers connecting OSPF to another routing domain (typically BGP) are Autonomous System Boundary Routers (ASBRs).

What actually moves through this hierarchy are Link-State Advertisements (LSAs), and knowing the types matters for troubleshooting:

• Type 1 (Router LSA) and Type 2 (Network LSA) describe intra-area topology — links, routers, and the DR/BDR relationship on multi-access segments. These stay within the area.
• Type 3 (Summary LSA) is generated by ABRs to advertise routes between areas. This is where route summarization happens — an ABR can collapse dozens of intra-area subnets into a single summarized Type 3 LSA using a command like area 1 range 10.1.0.0/16. This is not optional at scale; without summarization, every area's internal churn propagates as individual Type 3 updates across the whole domain, and every router recomputes routes for changes that shouldn't concern it.
• Type 4 (ASBR Summary LSA) tells routers outside an area how to reach an ASBR.
• Type 5 (External LSA) carries routes redistributed into OSPF from an external source (static, BGP, another IGP) and floods domain-wide by default.
• Type 7 (NSSA External LSA) is the Type 5 equivalent for Not-So-Stubby Areas, translated to Type 5 by the ABR at the NSSA boundary.

Area types are your other lever for controlling what floods where. A stub area blocks Type 5 LSAs and relies on a default route instead — useful for spoke sites with no need to see full external routing detail. A totally stubby area (a Cisco-specific term, implemented via area X stub no-summary on the ABR) goes further and blocks Type 3 summaries too, leaving only a default route — the tightest option for a simple branch topology. An NSSA exists for the awkward case: an area that needs to stub off external routing from the rest of the domain but still has a local ASBR that needs to inject its own external routes (Type 7, translated at the boundary).

Get the area boundaries wrong and you'll see it as SPF instability that seems to come from nowhere — a flapping link three hops away in another area shouldn't touch your router's CPU, and if it does, your summarization is missing or your area boundaries are drawn in the wrong place.

Area 0 is not a special area — it's the only mandatory hallway in a building where every other room's door has to open onto it, whether directly or through a tunnel.

BGP path selection: how routers actually choose a route

BGP doesn't pick "the shortest path." It picks the first route that wins a strict, ordered sequence of tie-breakers, evaluated top to bottom, stopping at the first one that isn't a tie. Every BGP engineer needs this list memorized, because every routing anomaly you'll debug traces back to one of these steps:

1. Weight (Cisco-proprietary, locally significant, not exchanged) — highest wins.
2. Local preference (local-preference) — highest wins. This is exchanged between iBGP peers within the AS and is your primary tool for controlling outbound traffic flow.
3. Locally originated routes — routes the local router originated (via network, aggregate-address, or redistribution) are preferred over received routes.
4. AS-path length — shortest as-path wins. This is the one everyone remembers and over-relies on; it's step 4, not step 1.
5. Origin type — IGP beats EGP beats incomplete (incomplete usually means redistributed from another protocol).
6. MED (Multi-Exit Discriminator) — lowest wins. Only compared between routes from the same neighboring AS by default (unless always-compare-med is configured), which trips people up constantly.
7. eBGP over iBGP — a route learned via eBGP is preferred over one learned via iBGP.
8. Lowest IGP metric to the next hop — this is where OSPF and BGP actually interact directly. Your IGP's cost to reach the BGP next hop breaks the tie.
9. Oldest route (eBGP paths only, and only if bgp bestpath compare-routerid isn't forcing router-ID comparison earlier) — path stability is preferred over a newer, possibly less stable, alternative.
10. Lowest router ID (or originator ID for reflected routes) — a pure tie-breaker among otherwise identical paths.
11. Lowest cluster list length, then lowest neighbor address — the final tie-breakers, mostly relevant in route-reflector topologies.

The practical implication: if you set local-preference on inbound routes to steer egress traffic, that decision overrides AS-path length every time. Engineers who only tune AS-path prepending are working one lever below where the real control is.

Route reflectors and confederations at scale

iBGP has a hard rule that trips up everyone coming from OSPF: routes learned via iBGP are never re-advertised to another iBGP peer. This split-horizon-style rule prevents loops within the AS, but it means iBGP requires a full mesh — every iBGP speaker peered directly with every other one — to guarantee full route visibility. That's n(n-1)/2 sessions. At 10 routers that's 45 sessions; it stops being maintainable long before you hit 50 routers.

Route reflectors (RRs) break this constraint by relaxing the re-advertisement rule for a designated router. A route reflector can take a route learned from one iBGP client and reflect it to other clients, collapsing the mesh into a hub-and-spoke of RR-to-client sessions. To keep loop prevention intact, reflected routes carry two extra attributes: ORIGINATOR_ID (the router ID of the route's originator, used to prevent a route from being reflected back to its source) and CLUSTER_LIST (a list of cluster IDs the route has passed through, checked the same way AS-path is checked in eBGP loop prevention).

The split-horizon rule for reflected routes specifically: a route from a non-client peer is reflected only to clients; a route from a client is reflected to both other clients and non-client peers. This is what lets you deploy RRs incrementally without a full topology rebuild — non-client (regular) iBGP peers keep working as before.

Confederations solve the same full-mesh problem differently: split the AS into sub-ASes, each running its own iBGP full mesh internally, connected by what looks like eBGP (but preserves some iBGP attributes like local-preference) between sub-ASes. The whole confederation appears as a single AS number to the outside world via BGP_CONFED_SEQUENCE handling in the AS-path. In practice, route reflectors are the far more common choice — they're simpler to operate and don't require renumbering into sub-ASes. Confederations show up mainly in very large service-provider backbones where administrative boundaries between sub-groups of routers are a design goal, not just a scaling workaround.

Where MPLS and SD-WAN fit in

MPLS and BGP intersect most directly in L3VPN deployments. Each customer (or department) gets a VRF (Virtual Routing and Forwarding) instance — an isolated routing table on the PE (Provider Edge) router. Routes from a VRF are tagged with a Route Distinguisher (RD) to make them unique in MP-BGP (multiprotocol BGP, address family vpnv4), and Route Targets (RTs) control which VRFs import which routes — this is what lets you build hub-and-spoke, full-mesh, or extranet topologies between sites purely through RT import/export policy, without touching the underlying MPLS label-switched path. The label switching itself (LDP or RSVP-TE-signaled labels) handles forwarding through the provider core; BGP handles the customer-facing routing intelligence layered on top.

This is where a lot of "BGP vs OSPF" framing breaks down in practice: inside an MPLS VPN site, you're commonly still running OSPF (or another IGP) for intra-site routing, and BGP for VPN-wide reachability between sites — they're not competitors, they operate at different layers of the same architecture.

SD-WAN changes the picture at the WAN edge rather than replacing this model. Traditional MPLS L3VPN gives you a private, provider-guaranteed path with predictable latency but at a real cost per site and per megabit. SD-WAN overlays routing decisions on top of multiple underlying transports — MPLS, broadband, LTE/5G — and picks paths dynamically based on real-time metrics like jitter, loss, and latency rather than static BGP attributes alone. Underneath, most SD-WAN fabrics still use a control-plane protocol that borrows heavily from BGP path-selection concepts (Viptela/Cisco SD-WAN's OMP is a clear example), so the mental model you've built for BGP best-path transfers directly — the difference is that path decisions can now react to live performance data instead of only administrative policy. It doesn't remove the need to understand BGP; it adds a layer of application-aware policy on top of it. Sites still commonly run BGP toward the SD-WAN edge to exchange routes with the fabric, and still run an IGP internally.

A real-world example

Consider an enterprise with three sites — HQ and two branches — each multihomed to two ISPs, with an MPLS L3VPN connecting sites via the provider core as well. Inside each site, OSPF runs between the core and access layers, with the site's edge routers as OSPF area 0 and each floor or building as a separate area feeding into it, summarized at the ABRs. This keeps intra-site convergence fast and contained.

At the WAN edge, each site's edge routers run eBGP to both ISPs and to the MPLS PE. Internally, the three edge routers at each site peer via iBGP (or, if the network has grown past a handful of sites, via a pair of route reflectors placed centrally) to share externally learned routes across the site without a full mesh.

Now trace a specific decision: HQ needs to reach a prefix at Branch B. That prefix is reachable both via the MPLS VPN (a BGP vpnv4 route with a good IGP-metric next hop) and via the internet through Branch B's ISP (learned as an eBGP route re-advertised through iBGP, arriving with a longer AS-path and no local preference boost). Assuming local-preference hasn't been explicitly set to prefer the internet path, the MPLS path wins — not primarily because of AS-path length, but because it's typically set with a higher local preference at ingress as deliberate policy (private path preferred over public internet), and it would win on AS-path length regardless in this case. If Branch B's MPLS circuit fails, that VPN route withdraws, BGP reconverges to the eBGP-learned internet path within the normal BGP convergence window, and OSPF inside Branch B is completely unaffected because the failure never touched the IGP domain. That isolation — WAN failures don't destabilize IGP domains and vice versa — is the entire point of running BGP at the edge and OSPF internally rather than trying to run one protocol everywhere.

Common mistakes to avoid

1. Redistribution loops between OSPF and BGP with no route tagging. Mutual redistribution (OSPF routes into BGP and BGP routes back into OSPF at a different ASBR) without tagging creates a real risk of routes looping back into the protocol they came from, sometimes with a worse metric that wins locally and causes suboptimal or unstable routing. Fix: tag routes on redistribution (route-map with set tag) and filter on re-redistribution to explicitly block routes carrying your own tag from re-entering the source protocol.

2. No maximum-prefix limit on eBGP sessions. A misconfigured or compromised peer that suddenly advertises the full internet table (or a leaked private table) into a session with no limit can exhaust memory and take down a router. Fix: set neighbor X maximum-prefix <limit> on every eBGP session, sized realistically above expected route counts, with alerting before the hard limit tears the session down.

3. Ignoring asymmetric routing consequences. Multihomed, multi-path designs routinely produce traffic that egresses one path and returns via another — normal in BGP, but it breaks stateful firewalls and NAT devices that expect to see both directions of a flow on the same interface, and it complicates troubleshooting because a traceroute in one direction tells you nothing about the return path. Fix: design stateful inspection points to either see both directions (e.g., via policy-based routing or by placing firewalls where symmetry is guaranteed) or use stateless/flow-aware devices that tolerate asymmetry, and document expected asymmetric paths so on-call engineers don't chase a phantom problem.

Frequently asked questions

Q: Should I use OSPF or BGP inside my data center fabric?
A: For large leaf-spine fabrics, many operators now run eBGP even internally (one AS per leaf/rack) because it scales more predictably than OSPF's flooding domain and gives finer per-link policy control. For small-to-mid data centers, single or dual-area OSPF is still simpler to operate and perfectly adequate. There's no universal answer — it's a function of fabric size and how much per-path policy control you need.

Q: How many prefixes can one OSPF area realistically handle?
A: There's no hard protocol limit, but operational guidance generally targets a few hundred routers and a few thousand routes per area before SPF run times and LSA flooding overhead start to matter — the real constraint is router CPU and how frequently the topology changes, not a fixed number.

Q: Why would I choose route reflectors over confederations?
A: Route reflectors are simpler to deploy incrementally on an existing flat AS — no renumbering, no restructuring of AS-path semantics. Confederations require carving your AS into sub-ASes, which is a bigger structural change. Most networks should default to route reflectors and reach for confederations only when there's a genuine administrative or policy reason to partition the AS itself.

Mastering BGP and OSPF at this level isn't about memorizing more commands — it's about understanding why each mechanism exists: areas and summarization exist to bound SPF's blast radius, the best-path algorithm exists to make policy deterministic across a network you don't fully control, and route reflectors exist because full-mesh iBGP simply doesn't scale. Once those design pressures make sense, MPLS VPNs and SD-WAN stop looking like separate technologies and start looking like what they are — BGP and IGP concepts applied at the WAN edge with a few extra labels and metrics layered on top.