LUNAROPS · OPERATIONAL UPLINK 100% UPTIME 1,247d POSTS 893 JEFF.MOON@LUNAROPS.DEV UTC --:--:--

FRRouting in Production: The Linux Router That Replaces Cisco for Many Use Cases

14 min read Networking Deep Dives
frroutingfrrnetworkingbgpospflinuxrouting
Contents

FRRouting — FRR for short — is the routing stack that quietly runs under some of the most demanding modern network deployments: Cumulus Linux (now NVIDIA), SONiC, Cilium’s BGP mode, MetalLB, Vyatta/VyOS forks, Calico’s route reflector, and the Linux routers inside countless small ISPs and enterprises. If you’ve used any of those, you’ve used FRR. And if you’ve ever looked at Cisco or Juniper pricing for a medium-sized branch router and winced, FRR on commodity hardware is often a serious alternative.

This post covers FRR as a production tool: what it is architecturally, which protocols you’d use it for, the operational patterns that work, the ones that don’t, and the trade-offs against vendor kit.

What FRR is

FRR is a suite of userspace routing daemons that speak routing protocols and program the Linux kernel’s forwarding table via netlink. It forked from Quagga in 2017 after Quagga’s development stalled, and has since become the de facto open source routing stack — development is active, vendors contribute, and the release cadence is predictable.

The daemons are separate processes, one per protocol:

  • zebra — the central process that talks to the kernel, holds the RIB, and coordinates the other daemons. Always required.
  • bgpd — BGP4, BGP4+, MP-BGP with EVPN, VPNv4/v6.
  • ospfd / ospf6d — OSPFv2 (IPv4) and OSPFv3 (IPv6).
  • isisd — IS-IS.
  • ripd / ripngd — RIP v1/v2 and RIPng.
  • ldpd — LDP for MPLS.
  • bfdd — BFD for fast failure detection.
  • pimd — PIM for multicast.
  • pbrd — policy-based routing.
  • staticd — static routes configured via vtysh (kept separate from kernel static routes).
  • vrrpd — VRRP.
  • babeld, nhrpd, eigrpd, pathd — for more niche uses.

You enable only the daemons you need. A typical deployment might run zebra + bgpd + bfdd, or zebra + ospfd + bfdd, not all of them.

Configuration is Cisco IOS-like. If you know show ip route and router bgp syntax, you’ll feel at home.

Installation

FRR packages are available on every major distro but the distro packages lag. For production, use the upstream FRR repository:

1
2
3
4
5
6
7

# Debian/Ubuntu
curl -s https://deb.frrouting.org/frr/keys.gpg | \
    tee /usr/share/keyrings/frrouting.gpg > /dev/null
FRRVER="frr-stable"
echo "deb [signed-by=/usr/share/keyrings/frrouting.gpg] https://deb.frrouting.org/frr \
    $(lsb_release -s -c) $FRRVER" | tee /etc/apt/sources.list.d/frr.list
apt update && apt install -y frr frr-pythontools

1
2
3

# RHEL/Fedora — enable the Copr repository
dnf copr enable -y @frrouting/frr-stable
dnf install -y frr frr-pythontools

Official container images exist at quay.io/frrouting/frr.

Enabling the right daemons

Edit /etc/frr/daemons:

bgpd=yes
ospfd=yes
bfdd=yes
staticd=yes
# others=no

Only start what you need. Each disabled daemon is fewer things that can crash, fewer things that can consume CPU, fewer things in ps. Reload with systemctl restart frr.

Global daemon options live in the same file:

bgpd_options="   -A 127.0.0.1 -M rpki"
ospfd_options="  -A 127.0.0.1"

-M rpki loads the RPKI (Resource Public Key Infrastructure) module for BGP origin validation — useful if you peer publicly, skip for internal-only deployments.

vtysh: the operational interface

vtysh is the integrated shell. It connects to the running daemons over Unix sockets and presents a unified Cisco-like CLI. Everything you’d do with the CLI on a real router, you do here.

# vtysh
Hello, this is FRRouting (version 10.2).
Copyright 1996-2005 Kunihiro Ishiguro, et al.

router01# show ip route
router01# configure terminal
router01(config)# router bgp 65001
router01(config-router)# neighbor 10.0.0.2 remote-as 65002
router01(config-router)# end
router01# write memory

write memory persists the config to /etc/frr/frr.conf. FRR uses integrated config mode by default (controlled by service integrated-vtysh-config in the config file), where one frr.conf holds all daemon configs. The alternative — a separate config file per daemon — exists for legacy reasons; use integrated mode.

You can also edit /etc/frr/frr.conf directly and reload:

1

systemctl reload frr

reload uses FRR’s frr-reload.py tool to compute a diff and apply changes live without dropping adjacencies. This is the right way to manage FRR from Ansible or Puppet — configs are idempotent text files, apply them with reload.

The big use cases

1. Edge routing for small ISPs and campuses

A Linux box with 4× 10 GbE, running FRR with BGP to an upstream and OSPF to internal routers, replaces a mid-range Cisco ASR or MX80 for a fraction of the hardware cost. Not a match for carrier-grade hardware at millions of pps with ACLs, but more than enough for regional ISPs and campus deployments.

2. Datacenter leaf/spine with BGP unnumbered

The modern datacenter fabric pattern: every ToR switch runs FRR, every leaf peers BGP with every spine, and the fabric does ECMP across equal-cost paths. BGP unnumbered removes the need for IP addressing on transit links — peer by interface name:

router bgp 65001
 neighbor fabric peer-group
 neighbor fabric remote-as external
 neighbor fabric capability extended-nexthop
 neighbor swp1 interface peer-group fabric
 neighbor swp2 interface peer-group fabric
 address-family ipv4 unicast
  redistribute connected
  neighbor fabric activate
 exit-address-family

Zero IPs on transit interfaces. IPv6 link-local auto-configures. ECMP handles load balancing. This is the Cumulus/SONiC/Arista pattern that FRR makes available on any Linux box.

3. BGP peering with the internet

A dual-homed edge router with two upstream providers, announcing your prefixes and accepting a full table (or just defaults) from each:

router bgp 65001
 bgp router-id 203.0.113.1
 bgp log-neighbor-changes
 no bgp default ipv4-unicast
 !
 neighbor 198.51.100.1 remote-as 64500
 neighbor 198.51.100.1 description upstream-a
 neighbor 198.51.100.1 password shared-secret
 !
 neighbor 203.0.113.254 remote-as 64501
 neighbor 203.0.113.254 description upstream-b
 !
 address-family ipv4 unicast
  neighbor 198.51.100.1 activate
  neighbor 198.51.100.1 prefix-list ACCEPT-DEFAULT in
  neighbor 198.51.100.1 prefix-list MY-PREFIXES out
  neighbor 198.51.100.1 route-map UPSTREAM-A-IN in
  neighbor 198.51.100.1 route-map UPSTREAM-A-OUT out
  neighbor 203.0.113.254 activate
  ...
  network 203.0.113.0/24
 exit-address-family
!
ip prefix-list MY-PREFIXES seq 5 permit 203.0.113.0/24
ip prefix-list ACCEPT-DEFAULT seq 5 permit 0.0.0.0/0

no bgp default ipv4-unicast is mandatory in modern BGP configs — it forces you to explicitly activate each address family per neighbor. Prevents the classic mistake of enabling a neighbor and accidentally redistributing v6 routes into v4 sessions.

Always use prefix-lists for filtering. Never trust a full table from an upstream without filtering. Accepting bogon routes (RFC1918, unallocated space, your own prefixes) is a common misconfiguration that causes outages.

4. VPN overlay meshes

A mesh of WireGuard tunnels between sites, each tunnel running OSPF or BGP, gives you site-to-site routing that auto-recovers from tunnel failures:

interface wg0
 ip address 10.99.0.1/30
 ip ospf area 0
 ip ospf network point-to-point
 ip ospf hello-interval 2
 ip ospf dead-interval 8

The WireGuard tunnel is just a Linux interface; OSPF sees it like any other. For dynamic mesh membership, consider BGP with route reflectors instead — better at scaling to tens of sites than full-mesh OSPF.

5. Kubernetes BGP with Cilium or MetalLB

Cilium’s BGP mode runs FRR in a pod on each node and peers with the upstream network, advertising service IPs and pod IPs into the fabric. MetalLB’s BGP mode is the simpler version of the same pattern — announce LoadBalancer service IPs via BGP to make them reachable from outside the cluster.

The FRR under Cilium is configured by Cilium’s operator based on CRDs; you don’t hand-write it. But knowing FRR helps you debug when a BGP session isn’t coming up.

6. EVPN for L2-over-L3

BGP EVPN (Ethernet VPN) lets you extend VLANs across a routed fabric without STP spanning tree. A typical EVPN VXLAN setup:

router bgp 65001
 address-family l2vpn evpn
  neighbor fabric activate
  advertise-all-vni
 exit-address-family
!
vni 100
 rd auto
 route-target import auto
 route-target export auto

Combined with Linux bridge + VXLAN interfaces, you get datacenter-grade fabric on commodity hardware. This is Cumulus/SONiC territory and involves more moving parts than plain BGP — evaluate whether you actually need it.

Route maps and prefix lists: the policy language

Prefix lists filter by prefix. Route maps filter and transform — match on various attributes, set others:

ip prefix-list CUSTOMER-ROUTES seq 5 permit 192.0.2.0/24 le 32
ip prefix-list CUSTOMER-ROUTES seq 10 permit 198.51.100.0/24 le 32
!
route-map FROM-CUSTOMER permit 10
 match ip address prefix-list CUSTOMER-ROUTES
 set local-preference 200
 set community 65001:100
!
route-map FROM-CUSTOMER deny 99
!
router bgp 65001
 neighbor 10.0.1.1 route-map FROM-CUSTOMER in

The implicit-deny at the end is explicit here (the deny 99 catches anything not matched earlier). Always end route-maps with an explicit deny or permit — the default behavior in FRR (deny) can surprise you.

Communities are arbitrary tags you can attach and filter on later — a standard pattern is to tag routes at ingress with “received from customer X”, then make policy decisions at egress based on those tags.

BFD for fast convergence

BFD (Bidirectional Forwarding Detection) detects link failures in hundreds of milliseconds by exchanging fast heartbeats. Register BGP or OSPF adjacencies with BFD:

router bgp 65001
 neighbor 10.0.0.2 bfd
!
bfd
 peer 10.0.0.2
  detect-multiplier 3
  receive-interval 150
  transmit-interval 150
 exit

Three missed 150 ms heartbeats = 450 ms failure detection. Without BFD, BGP takes 180 seconds (default hold time) to notice a neighbor is gone. Huge difference for production.

BFD works with OSPF, BGP, static routes, IS-IS — any protocol that registers. On high-speed links, tune down to 50 ms intervals for sub-second convergence.

Graceful restart and NSF

When bgpd or ospfd restarts (crash, upgrade, etc.), you don’t want to drop routes. FRR supports graceful restart with BGP:

router bgp 65001
 bgp graceful-restart
 bgp graceful-restart preserve-fw-state

Neighbors that also support graceful restart keep forwarding based on stale routes while bgpd recovers — typically a few seconds. preserve-fw-state tells zebra to keep the kernel FIB populated across the restart.

For OSPF, graceful-restart (RFC 3623) is also supported, but historically less reliable — verify with your vendor ecosystem.

Integration with the Linux stack

Things FRR does that pure Cisco can’t:

  • VRFs via Linux network namespaces or ip-vrf tooling. FRR runs routing for any number of VRFs natively.
  • Any kernel data path. XDP, DPDK (via VPP and the vpp-frr integration), eBPF programs — all coexist with FRR’s routing decisions.
  • Standard Linux tools. ip route show, ip route get, ss, tcpdump work exactly as expected. FRR just programs the kernel; the kernel forwards.
  • MTU path discovery works because Linux handles it, not FRR.
  • ECMP is a kernel feature; FRR installs multi-nexthop routes and the kernel does the hashing.

Things FRR doesn’t do as well as Cisco/Juniper:

  • Per-port QoS. Linux tc is available but no match for vendor QoS chipsets.
  • ACLs at line rate. nftables works for edge-case policing; for heavy ACLs at 100+ Gbps, hardware offload is required.
  • Flow collection. softflowd, goflow, or sFlow agents bolt on but aren’t as polished.
  • Visibility. No native SNMP MIB coverage as complete as vendors. Use Prometheus with the FRR exporter instead.

Monitoring

The frr_exporter Prometheus exporter scrapes vtysh output and exposes metrics. Point Prometheus at it and import a Grafana dashboard.

Key metrics:

  • frr_bgp_peers with state labels — alert on any Active or Idle peer that should be Established.
  • frr_bgp_received_prefixes_total — alert on sudden spikes (a bad prefix filter just accepted the internet) or drops.
  • frr_ospf_neighbors with state labels.
  • frr_bfd_session_up.

Also tail /var/log/frr/*.log — adjacency state changes, route installation failures, RIP/OSPF/BGP events land here. Ship to your log pipeline.

Performance

FRR is userspace control plane, kernel data plane. That means:

  • Routing decisions are as fast as zebra and the kernel can install routes. Typical BGP convergence on a full table (1M+ prefixes) takes 60–120 seconds on modern hardware — competitive with commercial routers, sometimes faster.
  • Forwarding performance is kernel forwarding performance. With default Linux routing, expect millions of packets per second on modest hardware; with XDP or DPDK offload, line rate at 100 Gbps is feasible.
  • Route table scale is kernel-limited. Linux handles 1M+ routes in FIB fine. 10M+ routes starts to strain memory.

For a ~1M-prefix BGP full feed, budget 2–4 GB RAM for bgpd, 1 GB for zebra, and a few GB of headroom.

Configuration as code

Treat FRR configs like any other infrastructure config:

  • Store frr.conf in git.
  • Apply via Ansible: push the config, then systemctl reload frr or vtysh -c 'copy running-config startup-config'.
  • Test in a netlab environment (containerlab works great for FRR) before rolling to production.
  • Use frr-reload.py for idempotent live updates.

A reasonable Ansible pattern:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

- name: deploy FRR config
  ansible.builtin.template:
    src: frr.conf.j2
    dest: /etc/frr/frr.conf
    owner: frr
    group: frr
    mode: '0640'
  notify: reload frr

- name: reload frr
  ansible.builtin.command: /usr/lib/frr/frr-reload.py --reload /etc/frr/frr.conf
  listen: reload frr

frr-reload.py computes the diff from running config and applies it incrementally. No adjacency drops on config changes unless the change itself requires it.

Common pitfalls

  1. Running unneeded daemons. Each adds CPU and attack surface. Disable in /etc/frr/daemons.
  2. Accepting full BGP tables without filtering. You will eventually accept garbage. Always filter.
  3. Missing no bgp default ipv4-unicast. Modern BGP requires explicit activation per family.
  4. Not using BFD. Default timers are too slow for modern failures.
  5. Peer with password but forgetting to set it on the other side. Silent failure mode.
  6. Over-reliance on redistributing connected. Redistributes management interfaces, IPMI VLANs, loopbacks you didn’t intend. Use selective redistribution with route-maps.
  7. Forgetting to write memory after vtysh changes. Reboot loses it.
  8. Running on kernels with firewall tables that block OSPF (89) or LDP/BFD. FRR is userspace; firewalls eat its traffic.
  9. Mixing IPv4 and IPv6 configs in one address family block. They’re distinct address families. Keep them separate.
  10. Using FRR as both router and firewall without thinking about asymmetric paths. Linux is stateful at the conntrack layer; multipath + stateful firewalls + asymmetric returns = dropped packets.

When FRR is the right choice

  • Budget-conscious multi-router deployments. The money saved vs Cisco/Juniper is real and significant.
  • Linux-first shops. If you’re already running Linux and Ansible for everything else, FRR fits the toolchain.
  • Cloud-native networking. Cilium, MetalLB, VyOS, SONiC — you’re using FRR whether you know it or not.
  • Container-based network labs. containerlab makes FRR the obvious choice for experimenting with BGP/OSPF topologies.
  • Homelab with 3+ routers. Static routes don’t scale; commercial routers are overkill.

When vendor kit is still worth the money

  • ISP backbone with 400 GbE and deep buffers. Vendor ASICs win.
  • Campus with strong QoS and line-rate ACLs. Vendor hardware offload.
  • Carriers needing TR-069/Netconf/Yang integration with existing OSS/BSS systems. Vendor integration ecosystem.
  • Shops without Linux operational expertise. If your team is CCNA-heavy and Linux-light, the vendor CLI familiarity matters.
  • Certifications and support contracts. Sometimes the auditor or the contract wins over the tech.

The mental shift from Cisco

If you’re coming from Cisco and picking up FRR:

  • show commands are nearly identical. show ip route, show ip bgp summary, show interface — all work.
  • configure terminal / end / write memory all work.
  • Cisco-style no prefixes work (no ip address, no neighbor X).
  • But config is a Linux file under /etc/frr/frr.conf. You can edit it in a text editor, diff it in git, template it with Ansible. Huge ergonomic win over TFTP config archives.
  • Interfaces are Linux interfaces. eth0, swp1, bond0, vlan100 — not GigabitEthernet1/0/1.
  • VRFs are net namespaces or vrf-lite. Different operational model than Cisco VRFs.

A reasonable starter config

For a dual-homed edge router with one BGP upstream and an internal OSPF domain:

frr version 10.2
frr defaults datacenter
hostname edge01
service integrated-vtysh-config
!
interface lo
 ip address 10.99.0.1/32
!
interface eth0
 description upstream-ISP
 ip address 198.51.100.2/30
!
interface eth1
 description internal-fabric
 ip address 10.0.0.1/30
 ip ospf area 0
 ip ospf network point-to-point
 ip ospf bfd
!
router ospf
 ospf router-id 10.99.0.1
 auto-cost reference-bandwidth 100000
 passive-interface default
 no passive-interface eth1
 redistribute bgp route-map BGP-TO-OSPF
!
router bgp 65001
 bgp router-id 10.99.0.1
 no bgp default ipv4-unicast
 neighbor 198.51.100.1 remote-as 64500
 neighbor 198.51.100.1 password xxxxx
 neighbor 198.51.100.1 bfd
 !
 address-family ipv4 unicast
  network 203.0.113.0/24
  neighbor 198.51.100.1 activate
  neighbor 198.51.100.1 prefix-list FROM-UPSTREAM in
  neighbor 198.51.100.1 prefix-list TO-UPSTREAM out
 exit-address-family
!
ip prefix-list FROM-UPSTREAM seq 5 permit 0.0.0.0/0
ip prefix-list TO-UPSTREAM seq 5 permit 203.0.113.0/24
!
route-map BGP-TO-OSPF permit 10
 match ip address prefix-list FROM-UPSTREAM
 set metric-type type-2
!
bfd
 peer 198.51.100.1
  receive-interval 150
  transmit-interval 150
  detect-multiplier 3
 exit

frr defaults datacenter is a preset that enables modern defaults (fast timers, proper BGP behavior) appropriate for datacenter/edge use. Use it.

FRR is not Cisco IOS. It’s close enough that your muscle memory works, different enough that you can template it, and open enough that you can read the source when something misbehaves. For a wide range of routing needs — homelab, small ISP, datacenter fabric, Kubernetes BGP, VPN meshes — it’s genuinely the right tool. Pair it with BFD, git-managed configs, Ansible-driven reloads, and Prometheus metrics, and you have a routing platform that holds up against hardware costing 20× more.

Comments