Proxmox SDN: Software Defined Networking
Proxmox SDN: Software Defined Networking
Traditional Proxmox networking uses Linux bridges configured per-node. This works for small setups, but as clusters grow you hit real limitations: network config must be manually replicated across nodes, VLANs require physical switch configuration, and there’s no built-in way to create isolated tenant networks that span nodes without pre-provisioning them on every host.
Proxmox SDN (introduced as stable in PVE 8.1) solves this with a cluster-wide networking abstraction. You define networks once in the datacenter-level configuration and Proxmox propagates them to all nodes automatically. More importantly, it supports VXLAN and EVPN overlay networks that let you create virtual L2 segments spanning the entire cluster without touching your physical switches.
This post covers the full SDN stack: the zone and VNet model, simple VLAN zones, VXLAN overlays, EVPN with BGP for proper multi-site routing, and practical configurations for multi-tenant isolation.
Core Concepts
Proxmox SDN introduces three abstractions stacked on top of each other:
Zone: defines a networking technology and its scope. A zone represents how networks are implemented — plain Linux bridge, VLAN-aware bridge, VXLAN overlay, or EVPN overlay. Each zone maps to one network implementation type.
VNet: a virtual network within a zone. Think of it as a virtual switch or a network segment. VMs attach to VNets the same way they attach to regular bridges.
Subnet: optional L3 configuration on a VNet — CIDR, gateway, DHCP settings. Proxmox can optionally run IPAM (IP Address Management) to track allocations.
The relationship: one Zone contains many VNets. Each VNet is one network segment. VMs connect to VNets.
Datacenter SDN
├── Zone: vlan-zone (Simple/VLAN)
│ ├── VNet: production (VLAN 100)
│ ├── VNet: staging (VLAN 200)
│ └── VNet: management (VLAN 10)
├── Zone: overlay-zone (VXLAN)
│ ├── VNet: tenant-a (VNI 10100)
│ └── VNet: tenant-b (VNI 10200)
└── Zone: evpn-zone (EVPN)
├── VNet: corp-net (VNI 20100)
└── VNet: dmz (VNI 20200)
Prerequisites and Installation
Installing SDN Dependencies
|
|
Network Requirements
SDN works best with:
- A dedicated VLAN trunk port per node for VLAN-based zones
- Routed underlay connectivity between nodes for VXLAN/EVPN (all nodes must be able to reach each other’s VTEP IP)
- For EVPN: BGP peering between nodes (or to external route reflectors)
Zone Types
Simple Zone
The simplest zone type: just creates a Linux bridge per VNet, no VLAN or overlay. Useful for creating named network segments that are automatically replicated across nodes.
|
|
After applying, every node will have a Linux bridge named isolated-net. VMs attach to it like any other bridge. Traffic between VMs on the same node flows through the local bridge; traffic between nodes requires routed connectivity or a physical switch trunk.
Simple zones are primarily useful for documentation and config-management purposes — the real power comes with VLAN and overlay zones.
VLAN Zone
Maps each VNet to a VLAN tag on a VLAN-aware trunk interface. This is the straightforward replacement for manually configured VLAN bridges.
|
|
Your physical switch port connected to vmbr0 must be a VLAN trunk carrying VLANs 10, 100, 200. Traffic is tagged at the bridge and carried over the physical infrastructure to other nodes.
Adding Subnets and DHCP
|
|
With DHCP configured, VMs on the VNet receive addresses automatically. Proxmox tracks allocations in its IPAM database.
VXLAN Zones: Overlay Networking
VXLAN (Virtual Extensible LAN) encapsulates L2 Ethernet frames inside UDP packets, creating virtual L2 segments that span routed IP networks. In Proxmox SDN, a VXLAN zone creates VTEP (VXLAN Tunnel Endpoint) interfaces on each node and bridges them to VNet interfaces.
When to use VXLAN over VLAN:
- You need more than 4094 network segments (VXLAN supports ~16 million VNIs)
- Your physical switches don’t support VLAN trunking to hypervisor hosts
- You want network segments that span multiple sites connected by IP routing
- You’re building a multi-tenant environment with arbitrary tenant isolation
VXLAN limitations (without EVPN):
- Uses multicast or flooding for BUM (Broadcast, Unknown unicast, Multicast) traffic — requires multicast in the underlay, or falls back to unicast flooding to all peers
- No L3 routing between VNets — requires an external router or a VM acting as gateway
- MAC learning is per-VTEP, which can cause issues at scale
|
|
After applying, each node has:
- A VXLAN interface
vxlan10100listening on UDP port 4789 - A bridge
tenant-aconnected to the VXLAN interface - VMs on
tenant-acan communicate across nodes via VXLAN encapsulation
Verifying VXLAN Operation
|
|
EVPN Zones: BGP-Controlled Overlays
EVPN (Ethernet VPN, RFC 7432) is the control plane for VXLAN that replaces multicast/flooding with BGP-distributed MAC/IP information. When a VM on node A wants to reach a VM on node B, instead of flooding frames to all VTEPs, the local VTEP looks up the destination MAC in its BGP EVPN table and sends the VXLAN packet directly to the correct remote VTEP.
EVPN advantages over plain VXLAN:
- No multicast requirement in the underlay — works over any IP network
- Scales to thousands of VNets without flooding overhead
- Supports L3 routing between VNets via Symmetric IRB (Integrated Routing and Bridging)
- MAC mobility: when a VM migrates, BGP distributes the MAC update immediately
- ARP suppression: VTEPs answer ARP requests locally, eliminating broadcast storms
EVPN requires FRR (Free Range Routing) running on each Proxmox node for BGP.
EVPN Architecture Options
Route reflector topology (recommended for 4+ nodes):
┌─────────────────────┐
│ Route Reflector │ (can be a dedicated VM or dedicated node)
│ (FRR, no VTEPs) │
└──────┬──────┬───────┘
│ │
┌────────┘ └────────┐
│ │
┌────▼────┐ ┌────▼────┐
│ pve1 │──────────────│ pve2 │
│ VTEP │ │ VTEP │
└────┬────┘ └────┬────┘
│ │
┌────▼────┐
│ pve3 │
│ VTEP │
└─────────┘
Full mesh topology (for 3 nodes, simpler):
- Every node has BGP sessions to every other node
- No route reflector needed
- Doesn’t scale beyond ~5 nodes
Configuring EVPN in Proxmox SDN
|
|
FRR BGP Configuration
Proxmox SDN generates the FRR configuration automatically when you apply SDN config. The generated config (at /etc/frr/frr.conf) looks like:
frr version 9.1
frr defaults traditional
hostname pve1
log syslog informational
no ipv6 forwarding
router bgp 65000
bgp router-id 10.0.1.1
no bgp default ipv4-unicast
no bgp ebgp-requires-policy
neighbor 10.0.1.2 remote-as 65000
neighbor 10.0.1.2 update-source 10.0.1.1
neighbor 10.0.1.3 remote-as 65000
neighbor 10.0.1.3 update-source 10.0.1.1
address-family l2vpn evpn
neighbor 10.0.1.2 activate
neighbor 10.0.1.2 next-hop-self
neighbor 10.0.1.3 activate
neighbor 10.0.1.3 next-hop-self
advertise-all-vni
exit-address-family
ip nht resolve-via-default
Verify BGP sessions are up:
|
|
L3 Routing Between VNets (Symmetric IRB)
With EVPN and L3VNI (a dedicated VNI for routing), traffic between VNets is routed at the ingress VTEP rather than needing a dedicated router VM:
|
|
Exit Nodes: Reaching the Physical Network
EVPN VMs by default can only reach other VMs in the EVPN domain. To reach the internet or on-premises networks, configure exit nodes:
|
|
Multi-Tenant Isolation Patterns
Pattern 1: VLAN Zones per Tenant
Simple and auditable. Each tenant gets its own VLAN zone and VLAN range:
|
|
Limitation: physical switches must carry all VLANs on trunk ports. Doesn’t scale beyond ~100 tenants cleanly.
Pattern 2: EVPN with Per-Tenant VNIs
More scalable. Each tenant gets VNets in a shared EVPN zone but different VNIs (effectively different broadcast domains):
|
|
VMs on ta-net1 and tb-net1 cannot communicate — they’re in completely separate broadcast domains. Neither can reach the other without an explicit routing policy.
Pattern 3: EVPN Zones per Tenant (Full Isolation)
Strongest isolation model. Each tenant has its own EVPN zone with its own VRF and L3VNI:
|
|
With separate VRFs, even if tenants use overlapping IP ranges (10.0.0.0/24 for both), there’s no conflict — they’re in isolated routing tables.
Firewalling SDN Networks
Proxmox’s built-in firewall integrates with SDN. You can apply firewall rules at the VNet level (applied to all VMs on that VNet) or at the VM NIC level.
|
|
For more complex microsegmentation, use the per-VM firewall configuration in conjunction with SDN:
# /etc/pve/firewall/100.fw (VM 100)
[OPTIONS]
enable: 1
policy_in: DROP
policy_out: ACCEPT
[RULES]
IN ACCEPT -p tcp -dport 80 -comment "allow HTTP"
IN ACCEPT -p tcp -dport 443 -comment "allow HTTPS"
IN ACCEPT -source 10.100.0.0/24 -p tcp -dport 22 -comment "allow SSH from management"
Troubleshooting SDN
SDN Config Not Applying
|
|
VXLAN Connectivity Issues
|
|
BGP/EVPN Not Working
|
|
VM Can’t Reach Gateway
|
|
Operational Notes
Applying SDN Changes
SDN config changes are staged until you explicitly apply them. This is a safety feature — misconfigured SDN applied to all nodes simultaneously could disconnect all VMs:
|
|
Performance Considerations
VXLAN encapsulation adds overhead:
- CPU: software VXLAN offload uses CPU cycles. With hardware offload (modern NICs support VXLAN TX/RX offload), overhead is minimal.
- MTU: VXLAN adds 50 bytes of overhead. If your underlay MTU is 1500, effective VM MTU is 1450. Set the underlay MTU to 1600 or use jumbo frames (9000) to avoid fragmentation.
|
|
BGP Route Reflector for Large Clusters
For clusters beyond 5 nodes, a full-mesh BGP topology becomes unwieldy (each node needs sessions to every other node). Set up a route reflector:
|
|
SDN vs Traditional Proxmox Networking
| Aspect | Traditional (/etc/network/interfaces) |
Proxmox SDN |
|---|---|---|
| Config management | Manual per-node | Cluster-wide, single point of change |
| VLAN support | Manual bridge per VLAN or VLAN-aware bridge | Declarative, auto-provisioned |
| Overlay networks | Not supported natively | VXLAN and EVPN built-in |
| Multi-tenancy | Manual firewall rules | Zone/VNet isolation with EVPN VRFs |
| L3 routing | External router VM or physical router | Distributed routing with Symmetric IRB |
| Troubleshooting | Simple, well-understood | More complex, requires BGP knowledge |
| Stability | Mature | Newer, improving rapidly (stable in PVE 8.1+) |
For homelabs and simple clusters, traditional networking is fine. For multi-tenant environments, clusters spanning multiple sites, or any environment requiring more than ~20 isolated network segments, SDN becomes the right tool.
The learning investment is real — VXLAN and EVPN concepts take time to internalize. But once the mental model clicks, Proxmox SDN gives you the kind of network flexibility that previously required expensive SDN appliances or dedicated networking hardware.
Comments