IPMI and Redfish: Out-of-Band Management for Humans
Every server you own has a second computer inside it that you rarely think about. It’s on the same chassis, it draws a few watts whether the main CPU is running or not, and it has its own Ethernet port, its own web UI, its own CPU, and its own operating system. It’s called the BMC — Baseboard Management Controller — and it’s the reason you can reboot a wedged server at 3 a.m. from your couch instead of driving to the datacenter.
Most engineers who meet a BMC meet it through the vendor’s web UI: iLO on HP, iDRAC on Dell, XClarity on Lenovo, SuperDoctor on Supermicro. That’s fine for the first server. It does not scale. By the time you have twenty or two hundred, clicking through JavaScript-heavy iDRAC 8 consoles is actively bad for your sanity. The two protocols that let you automate and the tools that speak them are what this post is about.
IPMI (Intelligent Platform Management Interface) is the old protocol — 1998 vintage, a weird binary wire format, UDP-based, and plagued by security problems that became public scandals in 2013. It still runs on essentially every server made since 2000 and will for the foreseeable future.
Redfish is the replacement — a REST/JSON API defined by the DMTF and supported on any server shipped since around 2017. It’s everything IPMI isn’t: modern auth, rich object model, self-describing, actually pleasant to script against.
In practice you’ll use both. IPMI for sensors and power control on older gear; Redfish for anything remotely modern and anything where you’re writing real automation. Let’s walk them end-to-end.
What the BMC actually is
A BMC is a small ARM SoC (usually ASPEED AST2500 or AST2600 — that same chip powers most of the industry) running embedded Linux. It’s wired into every major subsystem of the motherboard:
- Platform Environment Control Interface (PECI) — a sideband bus to the CPU for thermal and power telemetry that doesn’t require OS help.
- Intelligent Platform Management Bus (IPMB) — an I²C bus connecting sensors, fans, PSU monitoring, and front-panel controls.
- Low Pin Count (LPC) bus — gives the BMC access to the “host” side: video framebuffer (for KVM console redirection), SuperIO (for serial console capture), and the BIOS flash.
- Dedicated management NIC — a separate physical port (or shared with the host NIC via “NCSI”) that exposes the BMC’s IP even when the OS is off.
The BMC runs whether or not the host CPU is powered — that’s the key property. Plug in power, don’t even press the power button, and you can log in to the BMC and configure the server, update firmware, or power it on remotely. On enterprise servers, the BMC has its own dedicated 2W power rail that’s always live.
Inside the BMC you’ll find:
- A web server serving the management UI
- An IPMI server (UDP/623)
- An SSH daemon serving SMASH/CLP or a vendor CLI
- A Redfish server on HTTPS (port 443 by default)
- A virtual media proxy (remote ISO mount)
- A KVM console server (HTML5 noVNC on modern gear, Java applet nightmare on older)
- Log storage (System Event Log, audit log)
- Sensor polling daemons
All of that runs independently of anything the host OS is doing.
IPMI: what you need to know in 2026
The wire protocol
IPMI on the network is RMCP+ (Remote Management Control Protocol), UDP port 623. It supports a handful of authentication ciphers (the relevant ones today are RAKP+HMAC-SHA1 and MD5; you want SHA1 or better) and a small set of command classes:
- Chassis commands — power on/off, reset, get/set boot device
- Sensor commands — read temperatures, fan speeds, voltages
- SEL (System Event Log) — the hardware event log
- FRU (Field Replaceable Unit) — asset info, serial numbers
- LAN/User/SOL configuration — manage BMC users, network, serial-over-LAN
- Raw commands — escape hatch for vendor-specific stuff
The native client is ipmitool (and the somewhat more modern freeipmi-tools). Most sysadmins alias these with a wrapper that embeds credentials.
Baseline commands
Power management, the thing you actually use 90% of the time:
|
|
-I lanplus is RMCP+. The older -I lan is plain RMCP with weaker auth and you shouldn’t use it. -C3 forces cipher suite 3 (HMAC-SHA1/HMAC-SHA1-96/AES-128). Avoid cipher 0 (no auth) — some vendors ship with it enabled.
Boot device selection for the next boot (useful before a PXE re-image):
|
|
options=efiboot is the one that matters in a UEFI world — without it, some BMCs will boot the target in legacy/BIOS mode regardless of the host’s normal boot setting.
Sensors and the SEL
Read all sensors:
|
|
The System Event Log is where the hardware itself records interesting events — ECC errors, fan failures, thermal events, boot-time POST errors:
|
|
If you have any automated monitoring worth the name, scraping sdr list and sel list on a schedule is the floor. Prometheus exporters like ipmi_exporter do this per-host.
Serial over LAN (SOL)
This is the single most useful IPMI feature — a TTY console, redirected over the network, that survives host crashes:
|
|
For this to work end-to-end:
- BIOS must redirect the console to a serial port. Usually
Serial Redirection = On,Console Redirection Port = COM2,Baud = 115200, 8N1. - GRUB must emit to the same serial port. Add to the kernel cmdline:
console=tty1 console=ttyS1,115200n8. - BMC must have SOL enabled (
ipmitool sol info 1to verify;ipmitool sol set enabled true 1to enable).
Once it works, you can watch a kernel panic scroll by, interact with grub from your laptop, log into a single-user root shell — anything you could do at the physical console.
User management
|
|
The privilege levels are: CALLBACK(1), USER(2), OPERATOR(3), ADMINISTRATOR(4). OPERATOR can read sensors and power-cycle. ADMINISTRATOR can change everything including firmware.
BMC network config
|
|
Channel 1 is the dedicated management NIC on most Supermicro/Dell/HP; channel 8 is often the shared/NCSI port. ipmitool channel info tells you which is which.
The 2013 IPMI security incident
In 2013, Dan Farmer published research showing that every IPMI 2.0 implementation in circulation was vulnerable to a straightforward offline password cracking attack via the RAKP protocol. An attacker could request an authentication handshake and receive a hash of the password that they could then crack offline. This is inherent to the protocol, not an implementation bug — it’s in the spec.
The mitigations are still what you must do now:
- Never expose BMCs to the internet. Full stop. Management network only, with firewall rules and VPN for remote access.
- Use strong passwords. 20+ characters, random. The whole attack hinges on crackable passwords.
- Disable cipher 0.
ipmitool lan set 1 cipher_privs XaaaaXXaaXXaaXX(theXmarks cipher 0 as disabled). - Disable unused accounts. Many vendors ship with a default
admin/adminorADMIN/ADMIN. Change both the username and password, or create a new account and disable the default. - Prefer Redfish for anything new. Redfish uses HTTPS with normal TLS, normal session auth, no protocol-level password leak.
If your security audit flags IPMI, these controls are the defensible answer. “We turned it off” isn’t realistic on older gear; “we isolated and hardened it” is.
Redfish: the modern replacement
Redfish is a REST API. Every object (system, chassis, BMC, NIC, disk, power supply, firmware inventory) has a URI. Every operation is an HTTP method. Responses are JSON with embedded hypermedia (links to related objects). It’s self-describing — you can explore a BMC you’ve never seen before with curl and figure out its capabilities.
First contact
|
|
The root is publicly readable (no auth needed) — everything else requires either Basic auth or a session token.
Authentication
Basic auth works and is fine for scripts:
|
|
For longer sessions, create a session token:
|
|
Subsequent calls use -H "X-Auth-Token: abcdef...". Delete the session with DELETE /redfish/v1/SessionService/Sessions/1 when done.
Power control
|
|
Valid ResetType values: On, ForceOff, GracefulShutdown, ForceRestart, GracefulRestart, PowerCycle, Nmi (send an NMI for crash dumps).
Boot order override for next-boot-only
|
|
BootSourceOverrideEnabled: "Once" — the override applies to the next boot only. "Continuous" makes it persist until cleared. BootSourceOverrideMode must match the firmware’s boot mode; mixing UEFI and Legacy on modern UEFI-only firmware fails silently.
Virtual media: mount a remote ISO
This is the feature that pays for Redfish by itself. Point a server at a URL-hosted ISO (or IMG), and the BMC mounts it as a virtual CDROM. Boot from it. Install your OS, run a rescue image, whatever.
|
|
Combined with a boot override to CDROM and a power cycle, that’s a complete headless re-image — no PXE, no need to be anywhere near the machine, no JavaScript required.
The Image URL must be reachable from the BMC’s network. HTTP or HTTPS. Some BMCs support CIFS/NFS URLs too. Dell iDRAC and HP iLO both support this. Supermicro’s older BMCs (SIM lite) only support IPMI/Java applet virtual media — check before you rely on it.
Sensors and telemetry
|
|
For monitoring, the redfish_exporter (Prometheus) is the usual choice — it walks the tree and exposes metrics. Alternatively, many BMCs expose SNMP for the common counters and that’s less demanding than Redfish polling.
Firmware updates
Redfish’s UpdateService is the spec’s answer to “I want to update BMC firmware and BIOS without dropping into a vendor-specific tool.” Upload a firmware image, start the SimpleUpdate action:
|
|
Poll the returned task URI to track progress. In practice, vendors are still rolling out Redfish update support and many critical updates (BIOS on older gear, RAID firmware) still require vendor tools. But for BMC firmware itself and newer platforms, it works.
The SSE event stream
Redfish supports server-sent events — a long-lived HTTP connection that streams events (hardware errors, state changes, thermal trips). Subscribe with:
|
|
This replaces IPMI SEL polling for real-time alerting. Combined with something like Alertmanager-webhook on the receiving end, you get hardware events into your pager fast.
sushy and python-redfish
Writing curl everywhere gets old. Two libraries worth knowing:
sushy— OpenStack’s Python client. Object-oriented, handles pagination and error classes. The foundation of Ironic’s BMC management.python-redfish-library(DMTF) — more faithful to the spec, less opinionated.
|
|
For one-offs, curl + jq is unbeatable. For multi-host automation, use a library.
Automating bare-metal provisioning
The common workflow for a rack of new machines:
- Discover — scan the management network for BMCs.
nmap -sU -p 623to find IPMI; an HTTP probe for Redfish. (In enterprise environments, DHCP scope reservations and MAC-to-rack-position mappings give you a more trustworthy inventory than network scans.) - Set credentials — rotate the factory default to your per-host secret via Redfish
AccountServiceoripmitool user set. - Configure BMC — static IP, VLAN, NTP, syslog target, SNMP trap destination.
- Configure BIOS — Redfish’s
Biosendpoint exposes BIOS settings as key-value pairs. Set boot order, virtualization bits, NUMA behavior, SR-IOV, power profile. Apply takes a reboot. - Mount install media — insert the live installer ISO via virtual media.
- One-time boot override —
BootSourceOverrideEnabled: Once, BootSourceOverrideTarget: Cd. - Power on —
ResetType: On. - Boot the installer — cloud-init / kickstart / preseed runs.
- After install — unmount the ISO, clear the override, set boot order to disk.
OpenStack Ironic does all of this. Metal³ (Kubernetes on bare metal) does it too. At smaller scales, a few hundred lines of Python on top of sushy is enough. The first time you provision 40 servers in parallel from a laptop, it feels like a cheat code.
Tinkerbell and Foreman are the common open-source frameworks that tie Redfish control to OS provisioning. Sidero Metal / Talos integrate Redfish for running Kubernetes clusters on physical hosts.
Vendor quirks you’ll hit
- Dell iDRAC — Redfish is mature and complete.
racadmis the legacy CLI; it’s fine but Redfish supersedes it. The OME (OpenManage Enterprise) product wraps the whole fleet. - HP iLO — also mature.
hpiloCLI is still used. Some iLO versions have quirks in virtual-media URL handling (URLs must be exactly the right form). - Supermicro — Redfish support is newer and coverage is thinner. Older boards with SIM-lite BMCs are IPMI only. The newer X12/X13/H13 boards are solid. Supermicro’s free license tier limits BMC features; the paid license adds SSO and virtual media improvements.
- Lenovo XClarity — Redfish works; XClarity Controller has good multi-host management if you have many Lenovos.
- ASRock Rack / ASUS — prosumer gear. IPMI always works; Redfish support varies wildly by model and firmware version. Check before buying if Redfish matters.
- OpenBMC — a Linux Foundation project providing an open-source BMC firmware, used by Facebook, IBM, and others on their in-house hardware. Cleanly exposes Redfish as the primary API. If your servers are from a hyperscaler-adjacent vendor, you may already have OpenBMC under the hood.
Security posture for BMCs
Treating BMCs seriously is non-negotiable once you understand that they are, effectively, a root shell into every server that runs regardless of OS state. A compromised BMC owns the server; it can reflash firmware, manipulate BIOS settings, bridge to the host via virtual media or KVM, and persist indefinitely.
The minimal posture:
- Separate management network. Dedicated VLAN, dedicated switches if possible, no bridging to production traffic.
- Jump host / bastion for access. No direct routes from user VLANs to the BMC subnet.
- Per-BMC credentials. Not a shared admin password across the fleet.
- Credential rotation. BMC passwords stored in a secret manager, rotated periodically.
- Firmware updates. BMCs have CVEs. Treat them like any other internet-facing infrastructure even though they’re not on the internet. Test updates in a canary rack before fleetwide.
- Audit logging. Forward the BMC’s audit log to your SIEM. Many BMCs support syslog forwarding directly.
- TLS certs. Replace the factory self-signed cert with one from your internal CA. Not strictly necessary but it makes browser access less painful and removes a class of “is this really the BMC I think it is?” doubt.
- Disable unneeded services. Turn off IPMI if you’re only using Redfish. Turn off SNMPv1/v2c if not needed. Disable the console USB redirection if it’s not used.
Serial console aggregation at rack scale
Once you have tens or hundreds of servers, you don’t want to ipmitool sol activate them one at a time. The pattern:
- Conserver or sercd — classic serial console aggregators. Each managed server’s SOL session is multiplexed into a persistent logged stream. You can
console -f host42from anywhere and drop into its serial console; when you disconnect, the log keeps recording. ipmiconsole(from freeipmi) — lighter-weight SOL client with session persistence.- Vendor aggregators — Dell’s OME Console Aggregator, HP’s OneView, Lenovo XClarity all offer UI-based console access, but scripted Conserver is usually what seasoned ops people end up with.
The win: every server’s serial console is continuously logged. When something panics, the backtrace is already captured — not lost in the scrollback of a terminal you didn’t open in time.
A pragmatic fleet setup
For a dozen-to-a-hundred server homelab or small production, an opinionated setup:
- Dedicated mgmt VLAN (10.x.x.x/24), accessible via VPN or jumpbox.
- Inventory in git — a YAML file listing hostname, BMC IP, BMC MAC, rack position, role. Source of truth.
- Ansible
community.general.redfish_*modules — for firmware, BIOS, and user management. ipmi_exporter+redfish_exporter— Prometheus metrics; alerts on thermal, PSU redundancy loss, DIMM errors, SEL anomalies.- Conserver running on one utility host — every server’s SOL multiplexed into a persistent log directory.
- A small Python CLI wrapper —
mgmt power <host> cycle,mgmt console <host>,mgmt image <host> <iso>. Reads the inventory YAML, emits Redfish/IPMI calls. One hundred lines of code saves a lot of typing. - Alertmanager webhook → ticket creation for any SEL entry of type “Critical” — a bad DIMM or failing PSU turns into an ops ticket without human intervention.
That stack is enough to run a real fleet without logging into any BMC web UI for months at a time. Which is the point.
Closing
BMCs are the invisible infrastructure. They’re the reason you can sleep while your servers run in a building you’ve never visited. Learning IPMI and Redfish isn’t glamorous, and it’s the kind of knowledge that pays out in emergencies — when a filesystem is wedged and the host is unresponsive and you have two choices: drive three hours to the datacenter, or type ipmitool chassis power cycle and have the server back in 90 seconds.
Get the tooling set up now, before you need it. Automate the things you’d otherwise click. Document your BMC credentials somewhere other than the machine they unlock. The first time you use virtual-media Redfish boot to reimage a dead-disk server from your hotel room, every hour you spent learning this will have been repaid.
Comments