FlexLM and RLM License Server Internals: Debugging Denials, Borrowing, and Building License-Aware Schedulers

Every serious EDA shop eventually hits the license server problem. You own N seats of a tool, you have 2N engineers, and somehow the answer to “can I run my simulation right now?” depends on a TCP service that was designed in the early 1990s, is configured through text files that look like Lisp forms stapled to Fortran, and returns errors that read like “-97,121:2 "Can't connect to license server system." System Error: 115” when things go wrong.

Two license managers dominate EDA: FlexLM (now FlexNet Publisher) from Flexera, and RLM (Reprise License Manager) from Reprise Software. Between them, they run the licensing for Synopsys, Cadence, Siemens/Mentor, Xilinx, Intel FPGA, Altium, MATLAB, ANSYS, Mentor Calibre, and most of the rest of the industry. If you admin an EDA environment, you will read lmstat output on a regular basis. If you build CI or cluster schedulers for EDA teams, you will integrate with these servers’ queueing behavior, or you will fight them forever.

This post is the “everything a working EDA admin needs to know” version: how these servers actually work at the protocol level, what every daemon in a deployment does, the file formats and what each field means, how license denials get issued, how queueing and borrowing work, and how to build a scheduler that is license-aware rather than hope-based.

The big picture: licenses, daemons, and checkouts

A license manager has one job: hand out a limited pool of tokens to clients, track who has what, refuse requests when the pool is exhausted, and reclaim tokens when clients die. Everything else is details.

The core flow:

1. Vendor generates a signed license file containing the pool sizes, expiration dates, features, and a hostid (MAC address, disk serial, or dongle) that pins the license to a specific server.
2. License server (one or a three-server quorum) reads the file, starts listening on TCP, and waits.
3. Tool (VCS, DC, Innovus, etc.) at startup contacts the server, asks for a token for a specific feature, and either gets a token (uses the tool) or a denial (tool exits with an error).
4. Tool holds the token for its entire runtime, returning it when it exits cleanly — or when the server times it out after detecting the client is gone.

The servers themselves are small — a few MB of code, a few thousand tokens in memory, low CPU. The operational complexity is entirely in the vendor-specific rules: what counts as one checkout, how features combine, how denial messages are worded, and how clients recover from failures.

FlexLM: the incumbent

FlexLM (now officially “FlexNet Publisher”) has been the EDA license standard since the late 80s. It is used by Synopsys (most products), Cadence (some), Siemens EDA (many), MATLAB, ANSYS, and many others.

The three daemons

A FlexLM deployment involves:

• lmgrd — the main license daemon. The process you start at boot. It listens on TCP (default port 27000–27009) for client connections and delegates feature-specific requests to vendor daemons.
• Vendor daemon (snpslmd, mgcld, cdslmd, …) — one per vendor. lmgrd forks these. They own the actual pool accounting for that vendor’s features.
• lmutil — the swiss-army admin tool (lmstat, lmreread, lmdown, lmhostid, lmdiag).

Why two daemons? Historically, FlexLM let vendors ship their own daemon with vendor-specific logic (feature bundling, usage accounting, node-locked checks) while lmgrd handled the universal bits (TCP, health checks, fallover).

The license file

A FlexLM license file is ASCII and looks like this:

SERVER licsrv1.example.com 00112233AABB 27000
VENDOR snpslmd /opt/synopsys/bin/snpslmd PORT=27020

INCREMENT VCS-Simulator-RT snpslmd 2026.06 01-jun-2027 25 \
    VENDOR_STRING="LIMIT=25 VENDORSTAMP=abcd" \
    NOTICE="Contract 12345 License expires 01-jun-2027" \
    DUP_GROUP=UH SUPERSEDE ISSUED=15-jun-2026 \
    ck=120 SIGN="0B0F 8F8F 3C3C ..."

INCREMENT DesignCompiler snpslmd 2026.06 01-jun-2027 10 \
    VENDOR_STRING="..." \
    DUP_GROUP=UHD SUPERSEDE ISSUED=15-jun-2026 \
    ck=121 SIGN="..."

PACKAGE design-tools-pkg snpslmd 2026.06 \
    VENDOR_STRING="..." COMPONENTS="VCS-Simulator-RT DesignCompiler" \
    ck=122 SIGN="..."

Decoding it:

• SERVER line — hostname, hostid (MAC or custom ID), port. For a redundant three-server setup, you have three SERVER lines.
• VENDOR line — path to the vendor daemon binary and optionally a specific port for that daemon.
• INCREMENT (or FEATURE) — one line per licensed feature: name, vendor daemon, version, expiration, count, and a signature. DUP_GROUP=UH means duplicate checkouts from the same User+Host (counted once), UHD means User+Host+Display. The signature prevents tampering; editing any field invalidates the license.
• PACKAGE — groups multiple features under one name, used for bundle accounting.

You will see hundreds of INCREMENT lines on a real site. The ck= checksum is for the parser; SIGN= is the cryptographic signature.

Redundant triad

FlexLM’s flagship HA pattern is a triad: three license servers, at least two must be up for checkouts to succeed. License files identify all three servers:

SERVER srv1 0011... 27000
SERVER srv2 2233... 27000
SERVER srv3 4455... 27000
VENDOR snpslmd

Clients connect to any of the three (round-robin or LM_LICENSE_FILE order). If the “master” server fails, the remaining two elect a new master and continue. When the failed server returns, it rejoins.

Triads work, but beware: all three machines must have the same time (run NTP), the hostids must match the license file exactly, and network partitions can still produce denied-service windows. Some sites run triads inside a single datacenter and rely on a secondary site for DR rather than trying to stretch a triad across datacenters.

Clients: LM_LICENSE_FILE and SNPSLMD_LICENSE_FILE

Clients discover license servers through environment variables:

1
2

export LM_LICENSE_FILE=27000@licsrv1.example.com
export SNPSLMD_LICENSE_FILE=27000@licsrv1.example.com:27000@licsrv2.example.com

FlexLM searches:

1. Vendor-specific env var (SNPSLMD_LICENSE_FILE for Synopsys, MGLS_LICENSE_FILE for Siemens, CDS_LIC_FILE for Cadence).
2. LM_LICENSE_FILE.
3. Optional system config.

A colon-separated list tries each in order. Clients fall back quickly on connection failures, but if a server is reachable but slow or returning denials, startup can be slow.

Checkout: what actually happens

When a tool starts, it does:

1. TCP connect to the license server port on lmgrd.
2. lmgrd forwards the connection to the correct vendor daemon’s port.
3. Client sends a checkout request: CHECKOUT VCS-Simulator-RT version=2026.06 features=1.
4. Vendor daemon checks:
   • Is the feature licensed?
   • Has the license expired?
   • Is there a token available?
   • Does this client satisfy the rules (hostid restrictions, user/host restrictions from the options file)?
5. If yes, vendor daemon decrements the pool, records the checkout (user, host, display, checkout time), and returns a token handle.
6. Client keeps the connection open. Heartbeats flow. If the connection drops, the server frees the token after a grace period.

The grace period is usually around 4 hours. That is why, when a tool crashes, you see the license “stuck” for a long time — the server thinks the tool might still be there. You can force-release with lmremove, but the better fix is usually to make sure tools exit cleanly.

The options file: the sysadmin’s weapon

FlexLM ships an options file (options.dat, typically referenced from the license file via VENDOR ... OPTIONS=...) that lets you apply policy without touching the signed license:

# options.dat for snpslmd

# Reserve licenses for specific users/groups
RESERVE 2 VCS-Simulator-RT USER alice
RESERVE 5 VCS-Simulator-RT GROUP signoff

# Exclude problem users/hosts
EXCLUDE VCS-Simulator-RT USER spammer

# Exclude entire groups from a feature
EXCLUDE DesignCompiler GROUP interns

# Queue jobs when licenses are busy
LINGER VCS-Simulator-RT 60        # hold license 60s after release (avoid thrash)

# Report the activity
REPORTLOG /var/log/flexlm/report.log
DEBUGLOG /var/log/flexlm/debug.log

# Timeout idle checkouts
TIMEOUT VCS-Simulator-RT 14400    # 4 hours of idle = release

# Priority-based queuing
GROUP signoff alice bob carol
GROUPCASEINSENSITIVE ON
MAX 10 VCS-Simulator-RT GROUP signoff

The options file is how you turn a static license pool into something usable under contention. RESERVE keeps licenses available for critical users; EXCLUDE keeps trouble users out; TIMEOUT reclaims licenses from idle-but-alive tools; MAX caps consumption per group.

One trap: RESERVE does not mean “give these tokens to these users if they ask.” It means “always hold these N tokens back from the general pool; only give them to the named users/groups.” So if you have 25 licenses and you RESERVE 5 for signoff, everyone else sees a 20-license pool. This is not intuitive — budget accordingly.

lmstat and friends

The commands you will run daily:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

# Status of all features:
lmstat -a

# Status of one feature:
lmstat -f VCS-Simulator-RT

# Who's using this feature right now:
lmstat -f VCS-Simulator-RT -i

# Force a reread of the license file without restarting:
lmreread -c /opt/flexlm/license.dat

# Diagnose why a checkout is failing (from the client machine):
lmdiag -c $LM_LICENSE_FILE VCS-Simulator-RT

# Force release a hung license (use carefully):
lmremove -c $LM_LICENSE_FILE VCS-Simulator-RT bob bobs-workstation 42

# Graceful shutdown:
lmdown -c /opt/flexlm/license.dat -q

# Get this machine's hostid (what the license was pinned to):
lmhostid

lmstat -a is the go-to. Sample output:

Users of VCS-Simulator-RT:  (Total of 25 licenses issued;  Total of 18 licenses in use)

  "VCS-Simulator-RT" v2026.06, vendor: snpslmd, expiry: 01-jun-2027
  floating license

    alice  dev-lab-01  /dev/pts/2 (v2026.06) (licsrv1/27020), start Thu 4/18 08:32
    bob    dev-lab-02  /dev/pts/0 (v2026.06) (licsrv1/27020), start Thu 4/18 09:14
    carol  farm-node23  /dev/pts/0 (v2026.06) (licsrv1/27020), start Thu 4/18 10:05 (linger: 300)
    ...

Each line is a checkout. The (linger: 300) tag means the license has been released but is still reserved for this user for 300 more seconds (via the LINGER option) so that rapid re-invocations do not bounce.

RLM: the modern alternative

RLM was created by Matt Christiano (one of the original FlexLM authors) at Reprise Software, starting in 2006, as a smaller and simpler license manager. Cadence, Keysight, and numerous others use it. It shares the conceptual model with FlexLM but is cleaner in its implementation, smaller in daemon footprint, and ships with a built-in web UI.

The daemons

• rlm — the main daemon. Analogous to lmgrd.
• Vendor daemon (cdslmd_rlm, etc.) — one per vendor, started by rlm.
• rlmutil — the admin CLI (rlmstat, rlmreread, rlmswitch, etc.).

The license file

Similar structure, slightly different syntax:

HOST licsrv1.example.com 00112233AABB 5053
ISV cdslmd /opt/cadence/bin/cdslmd

LICENSE cdslmd Innovus 2026.06 01-jun-2027 10 \
    _ck=ab12 sig="..." issued=15-jun-2026

LICENSE cdslmd Genus 2026.06 01-jun-2027 10 _ck=cd34 sig="..."

Three SERVER lines on a triad, ISV (Independent Software Vendor) instead of VENDOR, LICENSE instead of INCREMENT. Otherwise analogous.

Web UI

The killer feature of RLM is a built-in admin web UI on port 5054 (or configurable). You browse to the URL and see:

• License pool usage live.
• Current checkouts.
• Historical usage graphs (built-in).
• Configuration editor (set reservations, exclusions, etc., from the browser).

No Flexera-style commercial monitoring tool needed. For smaller sites this is a huge win.

RLM options

Configured via .opt files, similar to FlexLM options.dat:

# rlm options file for cdslmd

RESERVE 2 Innovus user alice
GROUP implementation alice bob
RESERVE 4 Innovus group implementation
EXCLUDE Genus user intern_bob
TIMEOUT Innovus 7200

The syntax is different but the concepts map one-to-one.

rlmstat

1
2
3
4
5
6
7

rlmstat -a                       # all status
rlmstat -c port@host -a          # specific server
rlmstat -i                       # ISV info
rlmstat -l Innovus               # detailed feature info
rlmreread -c port@host           # reread license file
rlmswitch -c port@host           # switch/rotate log files
rlmdown -c port@host             # shutdown

RLM’s output is slightly more readable than FlexLM’s, in my opinion — but you learn to read both.

Debugging license denials

A license denial from a tool is an inscrutable error like:

Error: Unable to check out a license.
Error code: -1 Feature: Innovus
FlexNet License Finder: -15,570
  System Error: 115 "Operation now in progress"
  License path: 27000@licsrv1.example.com
  FlexNet Licensing error: -97,121
  For further information, refer to...

The error numbers do have meaning:

The pattern for diagnosing:

1. Get the exact error code. Two numbers: the overall error and a sub-error. Both matter.
2. Run lmdiag from the same machine as the tool. It walks through the same checkout logic and tells you in plain English why it failed:

1
2
3
4
5
6
7
8
9

lmdiag -c 27000@licsrv1.example.com Innovus

License server machine is licsrv1.example.com (licsrv1)
License file(s) on licsrv1: /opt/flexlm/cadence.dat:

  "Innovus" v2026.06, vendor: cdslmd
  License server: licsrv1

  Innovus is on EXCLUDE list for this user

That last line is often the whole answer.

3. Check lmstat -a from the client machine. Can the client even reach the server? If lmstat hangs, it is a network or firewall issue, not a license issue.

4. Check the debug log on the server. debuglog.cdslmd or wherever you pointed DEBUGLOG:

14:23:45 (cdslmd) IN: "Innovus" alice@farm23
14:23:45 (cdslmd) DENIED: "Innovus" alice@farm23 (All licenses in use.)
14:23:46 (cdslmd) OUT: "Innovus" bob@farm07

DENIED lines tell you exactly what went wrong. Grepping for DENIED in the debug log over time is a key metric — spikes in denial rate mean something changed.

5. For -97 errors: network between client and server is slow or dropping. Check firewall rules (vendor daemon port is often dynamic, not 27000), MTU issues, or a flaky switch.

Common root causes, in order of frequency

• Wrong LM_LICENSE_FILE (pointing to a server that does not have that feature, or a server that is down).
• All licenses in use (just wait, or buy more).
• Tool / license version mismatch — the license file is for 2026.06 but someone is trying to run 2027.06 with a SUPPORT_EXPIRE_DATE that has passed.
• Hostid mismatch after a hardware change (new NIC, new dongle, MAC address changed on a VM).
• Firewall blocking the vendor daemon port (not just lmgrd’s port) — the client gets an lmgrd connection but the subsequent port-to-vendor-daemon connection fails.
• Options file excludes the user without the user knowing.
• Expired license — nobody watched the dates.
• Client-side clock skew — the license is valid, but the client’s clock is far enough in the past that the license “has not started yet.”

Queueing

Both FlexLM and RLM support license queueing: when all licenses are in use, the client can wait instead of exiting. The tool has to support it (vcs supports it with -lmqueue; dc_shell does not natively but can be wrapped).

On the server side, queueing is enabled with options:

# FlexLM
QUEUE VCS-Simulator-RT 100  # queue up to 100 requests

When a license frees, the first queued request gets it. Priorities work with MAX, RESERVE, and queue-specific options.

Queueing is a blunt instrument. It works for small teams where latency is not critical. For a 1000-person CI farm, you want the scheduler (SLURM, LSF) to manage the queue with full awareness of resource policies, not the license server’s FIFO.

Borrowing: running offline

Some EDA tools support license borrowing — the client checks out a license and promises to return it N hours later, even if it disconnects. Useful for running on a laptop for a week.

1
2
3
4
5
6
7
8
9

# Client requests a borrow:
export LM_BORROW=18-apr-2026:25-apr-2026

# Start the tool; it borrows a license for the specified period.
vcs ...

# After the tool exits, the license stays borrowed.
# Manually return early:
lmborrow return VCS-Simulator-RT

For centralized farm use, borrowing is usually off — you do not want a dead job holding a license for a week. But for traveling engineers or situations where you intermittently disconnect from the license network, borrowing is the supported workflow.

Building a license-aware scheduler

The interesting work is integrating license awareness with a cluster scheduler. A naive approach submits farm jobs without regard for license availability; jobs get denied; SLURM counts them as completed; humans see 300 failures in the morning. Good.

Approach 1: lmstat polling in the scheduler

Have the scheduler periodically parse lmstat -a output and refuse to start jobs when the license pool is near empty. Most schedulers do not have this built in; you add a job submission filter:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

#!/usr/bin/env python3
# lic-submit-filter.py: called by SLURM's job_submit plugin
import subprocess, re, sys, json

def available(feature):
    out = subprocess.check_output(['lmstat', '-f', feature], text=True)
    m = re.search(r'Total of (\d+) licenses issued;\s*Total of (\d+) licenses in use',
                  out)
    if not m: return 0
    issued, used = int(m.group(1)), int(m.group(2))
    return issued - used

job = json.load(sys.stdin)
for lic in job.get('licenses', []):
    feat, count = lic.split(':')
    if available(feat) < int(count):
        print(f'DEFER: insufficient licenses for {feat}', file=sys.stderr)
        sys.exit(99)       # tell SLURM to defer
sys.exit(0)

This is crude (race conditions, polling latency) but prevents the job from starting into a guaranteed denial.

Approach 2: SLURM’s built-in Licenses= feature

As covered in the regression-at-scale post, SLURM (and LSF, PBS, etc.) have a built-in Licenses= configuration:

# slurm.conf
Licenses=vcs:25,innovus:10,dc_shell:20

Jobs declare consumption:

1

#SBATCH --licenses=vcs:1

SLURM holds the job in queue until it can reserve the license. Then it starts — and the tool itself does another checkout from the license server, which the scheduler already accounted for. The scheduler’s count and the server’s count track each other.

The weakness: SLURM’s count is static unless you update it. If licenses go offline (server crash, network partition, external team consuming from the same pool), SLURM still thinks the full pool is there.

Approach 3: Sync SLURM with actual license state

Run a cron job every minute that queries the license server, computes available tokens, and updates SLURM’s view:

1
2
3

#!/bin/bash
# /etc/cron.d/sync-lic-to-slurm
* * * * * root /usr/local/bin/sync_lic_to_slurm.sh

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

#!/bin/bash
# /usr/local/bin/sync_lic_to_slurm.sh
for feature in vcs innovus dc_shell; do
    avail=$(lmstat -f "$feature" | awk '
        /Total of [0-9]+ licenses issued/ {
            issued=$3
        }
        /Total of [0-9]+ licenses in use/ {
            used=$6
        }
        END { print issued }')
    scontrol update licenses="${feature}=${avail}"
done

Now SLURM’s view is eventually consistent with the license server. Jobs queue according to actual availability.

This is rough but works. For production, a deeper integration with the license server’s logs (watching OUT: and IN: events) is more responsive.

Approach 4: License-aware queue ranking

Beyond just not starting jobs when the pool is empty, you can rank jobs in the queue by license profile. A job that uses one license seat for 10 minutes is more “valuable” to run now than a job that uses four seats for 10 hours. SLURM’s fairshare plugin can be extended with factors based on declared license usage.

Ops patterns that save you

Monitor usage trends

Every license server logs usage. Parse those logs into a time-series database (Prometheus, InfluxDB, Graphite). Graph “licenses in use per feature over time.” Graph “denials per feature per day.” When someone asks “should we buy more VCS licenses?” you can answer with data instead of vibes.

Many sites run an lmstat-scraper that exports Prometheus metrics:

eda_licenses_issued{feature="VCS-Simulator-RT"} 25
eda_licenses_in_use{feature="VCS-Simulator-RT"} 22
eda_licenses_denied_total{feature="VCS-Simulator-RT"} 1423

Grafana dashboard, alerting on 95% utilization, automatic tickets when a feature is pegged for more than an hour.

Keep the server logs forever

Debug logs from the vendor daemon are gold for retrospective analysis: which runs got denied, when, how often, under what circumstances. Rotate them nightly, compress, archive to S3 or Glacier, keep for 2 years. Storage is cheap; historical data is priceless when a vendor asks you to justify a contract renewal.

Keep a runbook for common failures

• “All lmstat commands hang” → ping the license server, check its process, check network.
• “One feature suddenly goes to zero” → check for expiration that was not renewed, or an options file change.
• “Licenses released slowly” → look at LINGER setting, consider reducing.
• “Client says feature does not exist” → verify LM_LICENSE_FILE and the correct vendor env var.
• “Users complain of slowness” → check for a runaway job holding checkouts open.

Have a plan for server failure

If the license server goes down on a Friday evening and nobody notices until Monday morning, that is 48 hours of blocked engineers. Monitor lmgrd and every vendor daemon with your ordinary infrastructure monitoring (Nagios, Prometheus, Datadog). Page someone on service failure. Test restart procedures quarterly.

Coordinate with security

License servers listen on network ports that must be firewalled carefully:

• lmgrd port (configured, typically 27000).
• Vendor daemon port (sometimes dynamic; pin it with PORT=27020 in the VENDOR line).
• Both must be reachable from every client host.

Many outage stories are “security rolled out a new firewall rule that blocked the vendor daemon port.” The fix is to always specify static vendor ports in license files so firewall configs can pin them.

Beware of VM and cloud hostids

If your license server runs in a VM, the hostid is often a MAC address that changes if the VM is migrated or if the NIC is replaced. Some vendors accept dongles (USB hardware keys); others tie to a host-specific string that is stable across migration. Check with each vendor before moving to cloud.

A note on vendors

• Synopsys — FlexLM (via snpslmd). Usage reports via SmartKeys. Check out is per-feature; features bundle via INCREMENT packages. Heavy users of “consumable” time-based licenses in recent years.
• Cadence — Historically FlexLM (cdslmd); now many tools on RLM. “Suite” licensing with checkout rules that combine features.
• Siemens EDA (Mentor) — FlexLM (mgcld). Variable per tool; Calibre is hungry.
• Xilinx/AMD, Intel FPGA, Altium — FlexLM.
• MATLAB / ANSYS — FlexLM via MathWorks’s own License Manager variant.

Each has quirks (Cadence license suites, Synopsys feature bundling, Siemens Tessent’s per-die licensing). The underlying license manager is always FlexLM or RLM; the quirks are in the vendor daemon’s logic.

Wrapping up

License management is one of the few parts of EDA where there is no “open-source alternative” and no path around the incumbent — FlexLM and RLM are your world. The good news is that they are understandable. Underneath the cryptic error codes and the 1990s ergonomics, they are simple TCP services that hand out and reclaim tokens, with a rich rules engine for policy.

Learn to read lmstat output fluently. Build monitoring that turns license events into metrics. Model licenses as a first-class resource in your scheduler. When a tool says "-1,121: Cannot connect to license server" at 2 AM, you want to already know whether lmgrd is up, whether the port is reachable, and whether the specific feature has available tokens — not have to start learning from zero.

Every hour you spend making your license infrastructure boring is an hour the entire team is not blocked by a mystery denial. That leverage compounds more than any other tool-chain investment in an EDA environment.