BGP for Engineers: How the Internet Routes Itself

BGP is the protocol that holds the internet together. Every packet that crosses an autonomous system boundary is guided by BGP decisions. Yet for most engineers, BGP remains mysterious — something ISPs manage, not something you’d ever touch. That changes the moment you work on cloud infrastructure, build a multi-site network, or run a datacenter fabric. This guide cuts through the academic abstractions and gives you a working mental model.

What Problem Does BGP Solve?

Routing protocols like OSPF and IS-IS work brilliantly inside a single organization. They flood topology information, run shortest-path algorithms, and converge quickly. They assume you trust every router in the network and that traffic engineering policy is uniform.

The internet doesn’t look like that. It’s tens of thousands of independent networks — banks, universities, cloud providers, ISPs — each with different policies, each only sharing what they choose to share. You can’t flood your internal topology to AT&T. You can’t let Amazon apply their routing policy inside your network.

BGP was designed for this reality. It’s a path-vector protocol: instead of advertising topology, it advertises reachability with the full path of autonomous systems the advertisement passed through. Policy is first-class — you decide what you accept, what you reject, and what you prefer.

Autonomous Systems

An Autonomous System (AS) is a network under a single administrative domain using a common routing policy. Each AS has a unique ASN (Autonomous System Number).

• Public ASNs: 1–64511, assigned by RIRs (ARIN, RIPE, APNIC). You need one if you’re connecting to the public internet with BGP.
• Private ASNs: 64512–65534 (16-bit), 4200000000–4294967294 (32-bit). Use these for internal BGP deployments, datacenters, labs.
• 16-bit vs 32-bit: Modern BGP supports 4-byte ASNs. Write them in dot notation (e.g., 1.1000) or plain decimal.

Your ISP (AS 12345) ---- Your Network (AS 65001) ---- Colocation (AS 65002)

BGP sessions between ASes are eBGP (external BGP). BGP sessions within a single AS are iBGP (internal BGP). They behave differently in important ways.

eBGP vs iBGP: The Critical Differences

eBGP (External BGP)

• Peers are in different ASes
• Sessions typically between directly connected routers (TTL=1 by default — ebgp-multihop needed otherwise)
• Routes learned via eBGP get the local router’s AS prepended to the AS_PATH
• eBGP routes are redistributed to iBGP peers and other eBGP peers (with policy)

iBGP (Internal BGP)

• Peers are in the same AS
• No distance requirement — sessions over loopback addresses across the network
• Routes learned via iBGP are not re-advertised to other iBGP peers (the split-horizon rule)
• The AS_PATH is not modified when forwarding iBGP routes
• Requires either full mesh or a route reflector (or confederation) to avoid the split-horizon problem

The iBGP Full Mesh Problem

If you have 10 routers running iBGP, each needs a session to every other: n(n-1)/2 = 45 sessions. At 100 routers, that’s 4,950 sessions. This doesn’t scale.

Route Reflectors solve this. One or more routers are designated as reflectors. All other routers (clients) peer only with the reflector. The reflector breaks the split-horizon rule and re-advertises iBGP routes to all clients.

         Route Reflector (RR)
        /         |          \
      R1          R2          R3
     (client)  (client)    (client)

Route reflectors add a CLUSTER_LIST attribute to prevent routing loops between reflectors, and ORIGINATOR_ID to prevent a client from accepting its own reflected route.

BGP Session Establishment

BGP runs over TCP port 179. A session goes through these states:

Idle → Connect → Active → OpenSent → OpenConfirm → Established

Once established, peers exchange their full routing tables (UPDATE messages), then send incremental updates as routes change. KEEPALIVE messages (default every 60s) maintain the session; if no keepalive arrives within the hold timer (default 180s), the session drops and all routes from that peer are withdrawn.

The BGP UPDATE Message

Each UPDATE carries:

• WITHDRAWN ROUTES: prefixes being removed
• PATH ATTRIBUTES: properties of the advertised routes
• NLRI (Network Layer Reachability Information): the prefixes being advertised

Path Attributes: The Heart of BGP Policy

BGP routes carry attributes that drive selection and propagation decisions.

Well-Known Mandatory (must be in every UPDATE)

Well-Known Discretionary (recognized by all, included when relevant)

Optional Transitive (pass through even if not understood)

Optional Non-Transitive (not propagated beyond the next AS)

BGP Route Selection Algorithm

When multiple paths exist to the same prefix, BGP picks the best one. The algorithm evaluates attributes in strict order — first decisive comparison wins:

1.  Highest WEIGHT (Cisco proprietary, local to router only)
2.  Highest LOCAL_PREF
3.  Locally originated (network statement > aggregate > redistribution)
4.  Shortest AS_PATH
5.  Lowest ORIGIN (IGP < EGP < Incomplete)
6.  Lowest MED (only compared between paths from same AS)
7.  eBGP over iBGP
8.  Lowest IGP metric to NEXT_HOP
9.  Oldest eBGP route (for stability)
10. Lowest Router ID
11. Lowest neighbor IP address (tiebreaker)

This order is why LOCAL_PREF is the primary tool for inbound traffic engineering within your AS, and AS_PATH prepending is the primary tool for influencing which AS your traffic comes from.

Communities: BGP’s Tagging System

Communities are labels you attach to routes to trigger policy actions. They’re the BGP equivalent of k8s labels — powerful because both sender and receiver agree on meaning.

Standard community format: ASN:value (e.g., 65001:100)

Common uses:

• Signal to a peer not to advertise a route further (no-export: 65535:65281)
• Trigger traffic engineering at a peer’s routers
• Mark routes by origin region or service type
• Control route propagation in complex topologies

Large communities (RFC 8092) use ASN:function:value format and avoid the 16-bit limitation:

65001:1:100   # Set LOCAL_PREF 100
65001:2:50    # Prepend AS 2 times toward peer 50
65001:3:0     # Do not export to any peer

Running BGP with FRRouting

FRRouting (FRR) is the open-source routing suite used in production at major networks and by network vendors (Cumulus Linux, SONiC). It runs as a daemon on Linux and provides a Cisco/Juniper-like CLI.

Installation

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

# Ubuntu 22.04/24.04
curl -s https://deb.frrouting.org/frr/keys.asc | sudo apt-key add -
echo "deb https://deb.frrouting.org/frr $(lsb_release -s -c) frr-stable" | \
  sudo tee -a /etc/apt/sources.list.d/frr.list

sudo apt update
sudo apt install frr frr-pythontools

# Enable BGP daemon
sudo sed -i 's/bgpd=no/bgpd=yes/' /etc/frr/daemons
sudo systemctl enable --now frr

Basic FRR Configuration

FRR configuration lives in /etc/frr/frr.conf. You can also configure interactively via vtysh:

1

sudo vtysh

Simple eBGP peering (two routers exchanging routes):

! Router A — AS 65001, IP 10.0.0.1
!
router bgp 65001
 bgp router-id 10.0.0.1
 !
 neighbor 10.0.0.2 remote-as 65002
 neighbor 10.0.0.2 description "Router B"
 !
 address-family ipv4 unicast
  network 192.168.1.0/24
  neighbor 10.0.0.2 activate
  neighbor 10.0.0.2 soft-reconfiguration inbound
 exit-address-family
!

! Router B — AS 65002, IP 10.0.0.2
!
router bgp 65002
 bgp router-id 10.0.0.2
 !
 neighbor 10.0.0.1 remote-as 65001
 neighbor 10.0.0.1 description "Router A"
 !
 address-family ipv4 unicast
  network 10.10.0.0/16
  neighbor 10.0.0.1 activate
  neighbor 10.0.0.1 soft-reconfiguration inbound
 exit-address-family
!

iBGP with Route Reflector

! Route Reflector — AS 65001, loopback 10.255.0.1
!
router bgp 65001
 bgp router-id 10.255.0.1
 bgp cluster-id 10.255.0.1
 !
 ! Client 1
 neighbor 10.255.0.2 remote-as 65001
 neighbor 10.255.0.2 update-source lo
 neighbor 10.255.0.2 description "Leaf-01"
 !
 ! Client 2
 neighbor 10.255.0.3 remote-as 65001
 neighbor 10.255.0.3 update-source lo
 neighbor 10.255.0.3 description "Leaf-02"
 !
 address-family ipv4 unicast
  neighbor 10.255.0.2 activate
  neighbor 10.255.0.2 route-reflector-client
  neighbor 10.255.0.3 activate
  neighbor 10.255.0.3 route-reflector-client
 exit-address-family
!

Route Maps: Applying Policy

Route maps are the primary BGP policy tool. They match routes and set attributes.

! Prefer paths from AS 65002 over AS 65003
!
ip prefix-list CUSTOMER-ROUTES seq 10 permit 10.20.0.0/16
ip prefix-list CUSTOMER-ROUTES seq 20 permit 10.21.0.0/16
!
route-map PREFER-65002 permit 10
 match ip address prefix-list CUSTOMER-ROUTES
 set local-preference 200
!
route-map PREFER-65002 permit 20
 ! catch-all: permit everything else with default preference
!
router bgp 65001
 neighbor 10.0.1.1 remote-as 65002
 !
 address-family ipv4 unicast
  neighbor 10.0.1.1 route-map PREFER-65002 in
 exit-address-family
!

! Filter outbound — only send specific prefixes to a peer
!
ip prefix-list EXPORT-TO-PEER seq 10 permit 192.168.0.0/22
ip prefix-list EXPORT-TO-PEER seq 20 deny 0.0.0.0/0 le 32
!
route-map PEER-OUT permit 10
 match ip address prefix-list EXPORT-TO-PEER
!
router bgp 65001
 address-family ipv4 unicast
  neighbor 10.0.2.1 route-map PEER-OUT out
 exit-address-family
!

AS_PATH Prepending

To make a path less preferred by other ASes, prepend your own ASN multiple times:

route-map DEPREF-THIS-PATH permit 10
 set as-path prepend 65001 65001 65001
!
router bgp 65001
 address-family ipv4 unicast
  neighbor 10.0.2.1 route-map DEPREF-THIS-PATH out
 exit-address-family
!

The neighbor sees a longer AS_PATH and prefers the other path. Don’t prepend more than 3-4 times — excessive prepending is ignored by some networks.

Community Tagging

! Tag routes from a customer with a community
route-map CUSTOMER-IN permit 10
 set community 65001:100 additive
!

! Export only routes tagged as customer routes
ip community-list standard CUSTOMER-ROUTES permit 65001:100
!
route-map PEER-OUT permit 10
 match community CUSTOMER-ROUTES
!
route-map PEER-OUT deny 20
!

BGP in the Datacenter

Modern datacenter networks increasingly run BGP instead of traditional spanning tree or OSPF. The Clos fabric topology — spines connected to leaves, leaves connected to servers — maps cleanly to BGP’s eBGP model.

Why BGP in the Datacenter?

• Equal-Cost Multipath (ECMP): BGP supports ECMP natively, enabling all spine-leaf paths to be used simultaneously
• Policy and filtering: route maps let you control exactly what gets advertised
• Failure isolation: BGP’s explicit policy model contains failures better than flooding protocols
• Familiarity: operators who work with internet BGP can apply the same skills
• Vendor interoperability: standardized protocol, unlike some proprietary fabric solutions

Typical Spine-Leaf BGP Design

                 Spine-01 (AS 65000)     Spine-02 (AS 65000)
                /    \                  /    \
               /      \                /      \
   Leaf-01 (AS 65001)  Leaf-02 (AS 65002)  Leaf-03 (AS 65003)
       |                    |                    |
   Servers              Servers              Servers

Each leaf has a unique private ASN. Spines share an ASN (or use unique ASNs — both approaches work). Servers may run BGP directly (BGP unnumbered with link-local only peering is popular).

BGP Unnumbered (RFC 5549 + RFC 7404): BGP peers over link-local IPv6 addresses auto-assigned by the kernel. No IP address management needed on point-to-point links:

! FRR BGP unnumbered config
router bgp 65001
 bgp router-id 10.255.1.1
 !
 neighbor SPINES peer-group
 neighbor SPINES remote-as external
 !
 neighbor swp1 interface peer-group SPINES
 neighbor swp2 interface peer-group SPINES
 !
 address-family ipv4 unicast
  redistribute connected route-map LOOPBACK-ONLY
  neighbor SPINES activate
  neighbor SPINES next-hop-self
 exit-address-family
!

! Only advertise loopbacks, not transit links
ip prefix-list LOOPBACK seq 10 permit 10.255.1.1/32
ip prefix-list LOOPBACK seq 20 deny 0.0.0.0/0 le 32
!
route-map LOOPBACK-ONLY permit 10
 match ip address prefix-list LOOPBACK
!

ECMP Configuration

router bgp 65001
 bgp bestpath as-path multipath-relax
 maximum-paths 64
 maximum-paths ibgp 64
!

multipath-relax allows ECMP across paths with different AS_PATHs — essential in a Clos fabric where every path has a different spine AS in it.

BFD: Fast Failure Detection

BGP’s default hold timer is 180 seconds — too slow for datacenter failure detection. Bidirectional Forwarding Detection (BFD) provides sub-second link failure detection.

! Enable BFD for a BGP neighbor
router bgp 65001
 neighbor 10.0.0.2 remote-as 65002
 !
 address-family ipv4 unicast
  neighbor 10.0.0.2 activate
  neighbor 10.0.0.2 bfd
 exit-address-family
!

! BFD timers (in milliseconds)
bfd
 peer 10.0.0.2
  detect-multiplier 3
  receive-interval 300
  transmit-interval 300
!

With detect-multiplier 3 and 300ms intervals, a link failure is detected in ~900ms.

Monitoring and Troubleshooting

Key vtysh Commands

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

# Session state
show bgp summary
show bgp neighbors

# Routing table
show bgp ipv4 unicast
show bgp ipv4 unicast 10.20.0.0/16
show bgp ipv4 unicast 10.20.0.0/16 detail

# Path attributes for a prefix
show bgp ipv4 unicast 10.20.0.0/16 bestpath

# Routes from a specific neighbor
show bgp ipv4 unicast neighbors 10.0.0.2 received-routes
show bgp ipv4 unicast neighbors 10.0.0.2 advertised-routes

# Route reflector cluster
show bgp ipv4 unicast cluster-list

# BFD
show bfd peers
show bfd peers detail

# Community-filtered view
show bgp community 65001:100

# Statistics
show bgp ipv4 unicast statistics

Reading show bgp summary

Neighbor        V         AS   MsgRcvd   MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd
10.0.0.2        4      65002     12345     12300        1    0    0 1d02h34m            5
10.0.0.3        4      65003         0         0        0    0    0    never       Active

• State/PfxRcd shows the number of prefixes received when established, or the current FSM state when not
• Active means BGP is trying to connect but the TCP session hasn’t completed
• Idle often means the session is administratively down or waiting to retry after a failure

Debugging

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

# Watch BGP events (use carefully — very verbose on busy routers)
debug bgp updates in
debug bgp updates out
debug bgp neighbor-events

# Soft reset — re-apply inbound policy without resetting the session
clear bgp ipv4 unicast 10.0.0.2 soft in

# Hard reset — drops and re-establishes the session
clear bgp ipv4 unicast 10.0.0.2

Prometheus Metrics with FRR Exporter

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

# Install frr_exporter
wget https://github.com/tynany/frr_exporter/releases/latest/download/frr_exporter_linux_amd64
chmod +x frr_exporter_linux_amd64
sudo mv frr_exporter_linux_amd64 /usr/local/bin/frr_exporter

# Systemd unit
cat > /etc/systemd/system/frr-exporter.service <<'EOF'
[Unit]
Description=FRR Prometheus Exporter
After=network.target frr.service

[Service]
ExecStart=/usr/local/bin/frr_exporter
User=frr
Restart=on-failure

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl enable --now frr-exporter

Key metrics:

• frr_bgp_peer_state — session state (1=Established)
• frr_bgp_peer_prefixes_received_count — routes received per peer
• frr_bgp_peer_message_received_total — keepalives, updates
• frr_bfd_peer_state — BFD session state

Practical Lab: BGP Between Two Namespaces

Test BGP locally without hardware using Linux network namespaces:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

# Create two namespaces
sudo ip netns add ns1
sudo ip netns add ns2

# Create a veth pair connecting them
sudo ip link add veth1 type veth peer name veth2
sudo ip link set veth1 netns ns1
sudo ip link set veth2 netns ns2

# Assign IPs
sudo ip netns exec ns1 ip addr add 10.0.0.1/30 dev veth1
sudo ip netns exec ns2 ip addr add 10.0.0.2/30 dev veth2
sudo ip netns exec ns1 ip link set veth1 up
sudo ip netns exec ns2 ip link set veth2 up

# Add loopbacks (simulate router IDs and networks to advertise)
sudo ip netns exec ns1 ip addr add 192.168.1.0/24 dev lo
sudo ip netns exec ns2 ip addr add 10.10.0.0/16 dev lo

# Run FRR in each namespace
sudo ip netns exec ns1 /usr/lib/frr/frrinit.sh start
sudo ip netns exec ns2 /usr/lib/frr/frrinit.sh start

Or use Containerlab for a proper virtual topology:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33

# clab-bgp-lab.yml
name: bgp-lab

topology:
  kinds:
    linux:
      image: frrouting/frr:latest

  nodes:
    router1:
      kind: linux
      exec:
        - >
          vtysh -c "conf t" -c "router bgp 65001"
          -c "bgp router-id 1.1.1.1"
          -c "neighbor 172.20.20.3 remote-as 65002"
          -c "address-family ipv4 unicast"
          -c "network 192.168.1.0/24" -c "neighbor 172.20.20.3 activate"
          -c "exit-address-family"

    router2:
      kind: linux
      exec:
        - >
          vtysh -c "conf t" -c "router bgp 65002"
          -c "bgp router-id 2.2.2.2"
          -c "neighbor 172.20.20.2 remote-as 65001"
          -c "address-family ipv4 unicast"
          -c "network 10.10.0.0/16" -c "neighbor 172.20.20.2 activate"
          -c "exit-address-family"

  links:
    - endpoints: ["router1:eth1", "router2:eth1"]

1
2
3
4
5
6
7
8

# Install containerlab
bash -c "$(curl -sL https://get.containerlab.dev)"

# Deploy
sudo containerlab deploy -t clab-bgp-lab.yml

# Connect and verify
sudo docker exec -it clab-bgp-lab-router1 vtysh -c "show bgp summary"

Security Considerations

Route Origin Validation with RPKI

RPKI (Resource Public Key Infrastructure) cryptographically validates that an ASN is authorized to originate a prefix. Without RPKI, any network can accidentally (or maliciously) originate your prefixes.

! Configure RPKI validator in FRR
rpki
 rpki polling_period 300
 rpki cache 192.0.2.1 3323 preference 1
!

! Apply RPKI validation in route-map
route-map BGP-IN permit 10
 match rpki valid
 set local-preference 200
!
route-map BGP-IN permit 20
 match rpki notfound
 set local-preference 100
!
route-map BGP-IN deny 30
 match rpki invalid
!

BGP Session Security

! MD5 authentication
router bgp 65001
 neighbor 10.0.0.2 password s3cureP4ss!
!

! TTL security (generalized TTL security mechanism)
! eBGP peer must be exactly N hops away, prevents remote attacks
router bgp 65001
 neighbor 10.0.0.2 ttl-security hops 1
!

Prefix Filtering

Always filter prefixes from external peers. Accept only what you expect:

! Bogon prefix list — never accept these
ip prefix-list BOGONS deny 0.0.0.0/8 le 32
ip prefix-list BOGONS deny 10.0.0.0/8 le 32
ip prefix-list BOGONS deny 100.64.0.0/10 le 32
ip prefix-list BOGONS deny 127.0.0.0/8 le 32
ip prefix-list BOGONS deny 169.254.0.0/16 le 32
ip prefix-list BOGONS deny 172.16.0.0/12 le 32
ip prefix-list BOGONS deny 192.0.0.0/24 le 32
ip prefix-list BOGONS deny 192.0.2.0/24 le 32
ip prefix-list BOGONS deny 192.168.0.0/16 le 32
ip prefix-list BOGONS deny 198.18.0.0/15 le 32
ip prefix-list BOGONS deny 198.51.100.0/24 le 32
ip prefix-list BOGONS deny 203.0.113.0/24 le 32
ip prefix-list BOGONS deny 240.0.0.0/4 le 32
ip prefix-list BOGONS permit 0.0.0.0/0 le 32

! Apply inbound
route-map PEER-IN permit 10
 match ip address prefix-list BOGONS
!

Quick Reference

Where to Go Next

BGP is deep. After the basics above:

• EVPN over BGP: the modern way to build overlay networks in datacenters — VXLANs with BGP control plane for MAC/IP learning
• BGP Flowspec (RFC 5575): distribute firewall rules via BGP — used for DDoS mitigation
• Segment Routing with BGP: traffic engineering without RSVP, using SR-MPLS or SRv6
• Internet Exchange Points: how BGP peering at IXPs like DE-CIX and Equinix IX works
• BGP communities at scale: how large networks use communities for traffic engineering across thousands of peers

The NANOG presentation archives and Ripe NCC’s routing courses are excellent for going deeper. The bgp.he.net looking glass lets you explore live BGP data for any prefix or ASN.