Parallel Filesystems Compared: Lustre vs BeeGFS vs GPFS vs Weka

There is a moment in the growth of any compute cluster when NFS stops working. You can feel it: the scheduler shows 500 jobs running, but they are all waiting on I/O; your file server is at 100% CPU; the ls on /scratch/projects takes 40 seconds to return. What you need is not a bigger NFS server. You need a parallel filesystem — one that distributes both data and metadata across many servers and lets many clients read and write simultaneously without funnelling through a single gateway.

The four parallel filesystems you will actually evaluate are Lustre (the incumbent for HPC), BeeGFS (the easy-to-deploy challenger), IBM Spectrum Scale / GPFS (the enterprise-grade veteran), and Weka (the newcomer built for flash and low latency). CephFS and DAOS come up in adjacent conversations but sit slightly outside this category — CephFS is a general-purpose filesystem on top of a unified object/block/file stack, DAOS is an object store with a POSIX shim for persistent memory. This post focuses on the four that real HPC sites put on their shortlist.

What follows is how each one is architected, how it actually feels to deploy and operate, what workloads it is good at, and — most importantly — which traps each one sets for you.

The Shared Vocabulary

Before the comparisons, the pieces every parallel filesystem has:

• Metadata — the directory tree, file names, permissions, timestamps, mappings from file to data location. Metadata operations (open, stat, readdir, unlink) are often the bottleneck on real workloads, not throughput.
• Data / object servers — the machines that hold the actual bytes. Files are striped across many of them so one file can be read or written faster than any single server could deliver.
• Clients — the compute nodes that see the filesystem as a POSIX mount. The client is a kernel module (Lustre, GPFS, Weka) or a FUSE layer (BeeGFS has both; its native client is kernel-mode) that talks over the network to the metadata and data servers.
• Network fabric — InfiniBand, RoCE, 100/200/400 GbE, Omni-Path. The network is not a detail; it is half the filesystem.
• Stripe — how a file’s bytes are spread across data servers. Stripe count is the number of servers a file touches; stripe size is the chunk size per server before wrapping to the next. A 1-stripe file lives on one server (fine for small files); a 32-stripe file has 32 servers serving one file in parallel (essential for big files).
• HA pairs — most parallel filesystems use active/passive or active/active pairing of servers with shared storage (dual-ported SAS, NVMe-oF, or SAN) so a server failure does not take data offline.

Lustre

Lustre is the elephant. It powers most of the TOP500: Frontier, Aurora, LUMI, Leonardo, Fugaku-class sites. If your workload is “thousands of MPI ranks streaming huge checkpoints,” Lustre is what everyone else is running.

Architecture

• Metadata Servers (MDS) host one or more Metadata Targets (MDT). Early Lustre had one MDS and one MDT — a classic bottleneck. Modern Lustre supports DNE (Distributed Namespace) where directories are spread across multiple MDTs (DNE Phase 1) and individual directories can be striped across MDTs (DNE Phase 2).
• Object Storage Servers (OSS) host one or more Object Storage Targets (OST). Each OST is a block device (typically ldiskfs, a patched ext4, or ZFS in modern deployments).
• Clients mount the filesystem and talk LNet (Lustre Networking) to MDS and OSS nodes. LNet supports TCP, InfiniBand (IB verbs), RoCE, and others.
• MGS (Management Server) holds configuration for the cluster — typically colocated with the first MDS.

Striping

Striping is the first thing you tune on Lustre:

1
2
3
4
5
6
7
8

# Set default stripe for a directory: 8 OSTs, 1 MiB chunks
lfs setstripe -c 8 -S 1M /scratch/big_files

# Inspect a file's stripe layout
lfs getstripe bigfile.dat

# Migrate a file to a new stripe pattern
lfs migrate -c 16 -S 4M hugefile.h5

Progressive File Layout (PFL) lets you stripe small portions on one OST and big files wide:

1
2
3
4
5

lfs setstripe \
  -E 1M   -c 1      \
  -E 1G   -c 4      \
  -E -1   -c 16 -S 4M \
  /scratch/projects

Files written there live on 1 OST for the first megabyte, 4 for up to 1 GiB, and 16 OSTs for anything larger. Small files stay small-file-fast; big files are wide.

Strengths

• Maximum scale. Lustre is what you deploy when you need multiple hundreds of GB/s aggregate throughput and dozens of PB of capacity.
• Mature. Every HPC admin has war stories, which means the documentation, debugging tools (lctl, lfs, lctl dk), and tuning knowledge are deep.
• Open source, with commercial support from DDN, HPE, and Whamcloud.
• Strong striping model — PFL and Data-on-MDT (small files live on the MDT itself) cover a wide workload spectrum.

Weaknesses

• Operational complexity is real. Failover requires shared block storage and pacemaker/corosync. A bad e2fsck on an MDT can take a long, anxious night.
• Metadata scaling is additive, not linear. DNE helps, but you do not get the “just add servers” metadata scaling that GPFS and Weka can offer.
• Small-file workloads are rough without DoM (Data on MDT). Millions of 4 KiB files will slaughter your performance.
• Client kernel module. Upgrading kernels without testing client compatibility is a cluster-down event. Match client version to server version carefully.
• Tooling is UNIX-grade, not enterprise-grade. Web UIs exist (EXAScaler from DDN, ClusterStor from HPE) but the core remains CLI-centric.

When to pick Lustre

Pick Lustre when you are deploying hundreds of TB to hundreds of PB, your workload is large-file sequential I/O (checkpoint/restart, climate models, genomics pipelines with big intermediate files), you have dedicated storage admins, and you value the enormous HPC knowledge base. Lustre is the safe choice at scale — nobody gets fired for buying Lustre.

BeeGFS

BeeGFS (originally “FhGFS” out of the Fraunhofer Institute) is the friendly parallel filesystem — simpler to install, simpler to operate, and scaling to mid-range deployments that are the sweet spot for academic clusters, pharma, genomics, and engineering sims.

Architecture

Four daemons, each independently scalable:

• Management daemon (beegfs-mgmtd) — single instance, tracks cluster membership. Not on the data path.
• Metadata daemons (beegfs-meta) — hold directory entries and file inodes. Multiple metadata daemons scale the namespace horizontally; each daemon owns a set of metadata targets.
• Storage daemons (beegfs-storage) — hold the file data. Files are striped (round-robin, by default) across storage targets on different servers.
• Client (beegfs-client) — kernel module on compute nodes.

Unlike Lustre, there is no special backing filesystem. Each metadata daemon stores its metadata on a local ext4 or XFS filesystem; each storage daemon stores chunks as files on local ext4/XFS/ZFS. That makes recovery straightforward — if a node dies, you mount the underlying filesystem on another machine and rebuild.

Deployment

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

# On metadata server:
apt install beegfs-meta beegfs-utils
beegfs-setup-meta -p /data/meta -s 1 -m mgmt.cluster.local
systemctl enable --now beegfs-meta

# On storage server:
apt install beegfs-storage beegfs-utils
beegfs-setup-storage -p /data/storage -s 1 -i 101 -m mgmt.cluster.local
systemctl enable --now beegfs-storage

# On client:
apt install beegfs-client beegfs-helperd
# Edit /etc/beegfs/beegfs-client.conf: sysMgmtdHost = mgmt.cluster.local
systemctl enable --now beegfs-client

That is a functional deployment in about 20 minutes. No shared storage, no pacemaker.

Features

• BeeOND (BeeGFS On Demand) — spin up a parallel filesystem on the local SSDs of your compute job’s allocated nodes, mount it just for the job, destroy it when the job ends. Great for staging shuffle-heavy workloads like GPU training on top of a slower backing filesystem.
• Storage and Metadata Mirroring — active-active replication of metadata and data for HA.
• Quotas, RDMA (IB/RoCE), Storage Pools — standard parallel FS features.
• Commercial support from ThinkParQ (the company behind BeeGFS).

Strengths

• Easy to install and operate. By far the gentlest learning curve of the big four.
• No shared storage requirement for basic deployment — just servers with local disks.
• Native buddy mirroring for HA without external cluster software.
• BeeOND is a genuinely unique feature — on-demand per-job parallel FS.

Weaknesses

• Smaller community than Lustre or GPFS.
• License model changed — the server components moved to a paid license for production use in recent releases (the client remains open). Check current terms before committing.
• Tops out earlier. BeeGFS at hundreds of servers works, but most deployments live in the tens of servers / PB range.
• Small-file performance is decent but not as good as Weka or GPFS with SSD metadata.

When to pick BeeGFS

Pick BeeGFS for mid-size clusters (10s of compute nodes, 100s of TB to low PB), academic or research labs without dedicated storage staff, or where “works on day one with sensible defaults” matters more than “absolute top-end performance.” BeeOND is also the deciding factor if your workload benefits from per-job scratch on local SSD.

IBM Spectrum Scale (GPFS)

GPFS has been around since the early 90s under several IBM brands. It is the enterprise parallel filesystem — deployed in banks, government, weather, life sciences, and the Summit/Sierra-class HPC systems. Unlike Lustre’s “HPC first, enterprise bolted on” feel, GPFS was designed for both from day one.

Architecture

• NSD (Network Shared Disk) servers expose block devices (typically SAN-attached) as NSDs to the cluster. Data is striped across NSDs at the block level.
• Filesystem daemon (mmfsd) runs on every node — servers and clients both. GPFS is symmetric in a way Lustre and BeeGFS are not: any node can be a client, a server, or both, and the metadata work is distributed rather than concentrated on specific MDS nodes.
• Cluster manager and filesystem manager roles float among nodes; quorum requires at least three manager-eligible nodes.
• Policy engine controls file placement, migration, and deletion based on rules (age, size, extended attributes, etc.).

Distinguishing features

• Information Lifecycle Management (ILM) — write policies that move cold files from SSD to HDD to tape automatically. This is the feature enterprises buy GPFS for.
• Active File Management (AFM) — cache remote filesystems locally, hybrid on-prem/cloud bursting, multi-site replication with eventual consistency.
• Encryption, compression, erasure coding (GNR / Native Raid), snapshots, clones, quotas by user/group/fileset — all first-class.
• AFM to S3 — GPFS can expose an S3 endpoint and cache S3 content.
• CES (Cluster Export Services) — re-export GPFS as NFS, SMB, or object (S3/Swift) with HA.
• Strong NFSv4 ACL support. Windows file server compatibility is real, not an afterthought.

Strengths

• Most mature enterprise storage features of any parallel FS: snapshots that actually work at scale, clones, compression, encryption, ILM policies, audit.
• Symmetric architecture — metadata scales by adding nodes; there is no MDS bottleneck to engineer around.
• Better small-file performance than Lustre at the same price point, especially with SSD metadata pools.
• IBM support is a phone call away; if you have a legal or regulatory requirement for commercial-grade support, this matters.
• Multi-protocol access (POSIX + NFS + SMB + S3) is genuinely usable, not a half-integrated bolt-on.

Weaknesses

• Proprietary and expensive. Licensing is node-based, and for clusters of any size the cost is non-trivial. Academic and small deployments rarely justify the price.
• Learning curve is as steep as Lustre in a different direction — the mm* command family (mmlscluster, mmlsfs, mmchpolicy, mmapplypolicy) is huge, and the error messages are cryptic until you have worked with it for a year.
• Upgrade paths require attention: rolling upgrades work but across major versions you need to read IBM’s release notes carefully.
• Kernel module — like Lustre, a new kernel requires a new GPFS build.

When to pick GPFS

Pick GPFS when you need a single storage platform for mixed workloads (HPC + general file server + S3 gateway), when you have a budget that makes the licensing a non-issue, when you need enterprise features like WORM, encryption, compliance, and audit, or when you are already an IBM shop with support relationships in place.

Weka

Weka (WekaFS / Weka Data Platform) is the youngest of the four and built on a fundamentally different premise: NVMe flash is cheap enough to build the whole filesystem on it, and if you do that, you can deliver sub-millisecond latency that Lustre and GPFS cannot match on their spinning-disk-primary architectures.

Architecture

• Weka cluster runs on commodity x86 servers with local NVMe drives. The software is packaged as containers and deployed via CLI or Ansible.
• Weka reserves CPU cores per server (typically 6–19 cores, depending on role) that bypass the kernel scheduler entirely, pinning threads to cores and polling NVMe queues in userspace. This is the core of the low-latency design.
• Distributed erasure coding across servers (4+2, 6+2, 8+2, etc.) — no separate metadata/data tier; every server holds both metadata and data.
• Client is a DPDK-based userspace client that talks to the cluster over Ethernet or InfiniBand.
• S3 gateway and tiering to object storage — Weka can tier cold data to S3-compatible backends (native S3, Ceph RGW, MinIO), giving you a hot NVMe tier backed by an unlimited object tier.

Features

• Sub-millisecond latency at scale. Weka’s design target is small, random I/O over tens of thousands of clients — the GPU training workload, the VDI workload, the EDA regression workload.
• Snapshot, clone, and replication with copy-on-write semantics.
• POSIX, NFS, SMB, S3 — multi-protocol with one namespace.
• GPUDirect Storage support — NVMe data can go directly to GPU memory, bypassing the CPU entirely. This is increasingly important for large-scale AI training.

Strengths

• Latency and small-file performance are in a different class from Lustre and GPFS. If your workload is “many clients, small random reads” (think AI training data loaders, EDA simulations with millions of small files), Weka is what sets the bar.
• Tiering to S3 means you can store the cold half of your data cheaply and still see it as one mount.
• GPU workloads: Weka is the de facto shared filesystem behind most big production GPU clusters because of GPUDirect and low latency.
• Operational simplicity for what it does — no block-device arcana, no pacemaker.

Weaknesses

• Proprietary and expensive — Weka’s pricing reflects both the engineering and the market position (think NetApp-range pricing per useful TB, plus an infrastructure cost because you need NVMe).
• Network-sensitive — you need good NICs (100 GbE minimum, typically 200 GbE) and a well-tuned fabric, or the userspace fast path will disappoint you.
• Less established than the others. Plenty of production deployments, but not decades of war stories.
• Reserved cores mean you give up 10–20 cores per storage node to Weka. For some workloads that is the right trade; for others it is not.

When to pick Weka

Pick Weka when latency is the problem, when your workload is GPU training or EDA regression (many clients, small random I/O), when you have the budget for high-end flash and high-speed networking, or when you want a single unified hot/cold namespace and are comfortable with a proprietary vendor.

The Comparison at a Glance

Choosing: A Decision Tree

What is your primary workload?

• Large sequential I/O at enormous scale (checkpoint/restart, climate, HEP) → Lustre.
• GPU training with small-random access patterns → Weka, possibly GPFS if budget pushes you there.
• Mixed workload, enterprise governance, multi-protocol → GPFS.
• Academic cluster or mid-size lab, simple ops → BeeGFS.
• HPC EDA (semiconductor design, millions of small files) → Weka if budget allows; GPFS with SSD metadata pool as fallback.

What is your budget per useful PB?

• Low-hundreds of thousands: Lustre with commodity storage, or BeeGFS.
• Low millions: GPFS, Lustre on enterprise appliances (DDN/HPE).
• Millions+: Weka, GPFS on ESS (IBM’s appliance).

What is your ops team?

• Dedicated storage team with decades of HPC experience → Lustre or GPFS, whichever matches workload.
• General Linux admins, no storage specialists → BeeGFS or Weka (vendor-supported).

Do you need S3 semantics?

• Yes, as a tier → Weka or GPFS with AFM.
• Yes, as a primary interface → don’t use a parallel FS; use Ceph RGW or MinIO and bolt a CSI driver on.

Deploying: What You Actually Learn the Hard Way

A few lessons from clusters I have seen stumble during the first six months:

1. The fabric is half the filesystem

A 200 GbE NIC on a misconfigured switch will give you 5 GB/s when the datasheet says 25. Diagnose with:

• ib_write_bw, ib_read_bw, ib_write_lat (perftest) for InfiniBand.
• iperf3 -P 8 for Ethernet (multiple streams to avoid single-flow limits).
• ethtool -S and the switch counters for discards, pause frames, CRC errors.
• MTU consistency across every host and switch — one host with MTU 1500 in a 9000-MTU network sprays fragments everywhere.

Before you tune the filesystem, tune the fabric. Burn-test it for 48 hours under synthetic load.

2. Metadata is the silent killer

Your users will not notice 10% slower throughput. They will scream when ls takes 20 seconds on their results directory with 100k files in it. When designing for a parallel FS, plan for metadata-heavy workloads:

• On Lustre, use DNE and consider DoM for small files.
• On BeeGFS, scale meta daemons early; they are cheap and avoid hot-spotting.
• On GPFS, put metadata on dedicated SSD NSDs.
• On Weka, this is usually a non-issue; the whole thing is flash.

3. Benchmark with your workload, not someone else’s

dd and iozone tell you nothing useful. For HPC, use IOR (for throughput) and mdtest (for metadata):

1
2
3
4
5

# Streaming write, 16 nodes, 8 ranks each, 4 GiB per rank, aligned 1 MiB I/O
mpirun -n 128 -ppn 8 ior -a POSIX -t 1m -b 4g -w -r -e -F -o /scratch/ior_test

# Metadata: create/stat/remove 100k files per rank
mpirun -n 64 -ppn 4 mdtest -I 100000 -z 0 -u -d /scratch/mdtest_dir

Then run your actual application. The gap between synthetic IOR numbers and real-world performance is often 2–5×.

4. Monitor

You want continuous visibility into:

• Per-server throughput and IOPS (where are the hot spots?).
• Per-client bandwidth (which node is saturating things? often a single misbehaving job).
• Metadata ops/sec (the real-world bottleneck).
• Network errors (retransmits on TCP, symbol errors on IB, pause frames on RoCE).

Every vendor has its own stack (Lustre exposes stats via lctl; GPFS has mmperfmon and Grafana exporters; BeeGFS has beegfs-ctl; Weka has a built-in web UI and Prometheus endpoints). Wire them to Prometheus and Grafana on day one. Dashboards before complaints.

5. Have a restore plan before you need one

Snapshots are not backups. Parallel filesystems make it easy to destroy a lot of data with one command. Before production:

• What is your backup strategy? (Tape via HSM, sync to object storage, replication to a second cluster?)
• How long does a restore actually take? Have you tested it?
• Do you have a documented procedure for “rm -rf / on the wrong mount”?

6. Tune stripe and block size to workload

• Large sequential workloads: big stripe count (16–32 OSTs or NSDs), big stripe size (4–16 MiB).
• Small file workloads: stripe count 1, data on metadata target where supported.
• Medium files (100 MiB–few GiB): 4–8 stripes, 1 MiB stripe size.
• Random access to huge files (databases, HDF5): match stripe size to application block size; stripe wide.

Wrong stripe settings give you 10% of the throughput you paid for. This is tuning the vendors cannot do for you because it is workload-dependent.

When NOT to Use a Parallel Filesystem

Parallel filesystems are expensive, operationally heavy, and designed for a specific class of problem. If you are not in that class, a simpler tool is better:

• Small scale (< 10 compute nodes, < 100 TB): a well-tuned NFS server on a beefy box with SSDs will serve you better and cost a tenth as much.
• Mostly object-accessed data (ML datasets, archive): S3-compatible object store (MinIO, Ceph RGW, commercial) with a caching layer on the clients.
• Small scratch needs on individual nodes: per-node local NVMe. Nothing beats it for latency.
• Git repositories, source trees, small configs: a filesystem optimized for millions of tiny files, not a parallel FS. Even Weka, which handles small files well, is not the right store for your git repos.

Use a parallel filesystem because you have measured a problem that only a parallel filesystem can solve — aggregate bandwidth, many-client metadata, or latency under concurrent load — not because “HPC = parallel filesystem.”

Wrapping Up

If you take one thing from this post: pick the filesystem that fits the workload, the ops team, and the budget — in that order. Lustre is the default for maximum-scale HPC. BeeGFS is the default when you want parallel-FS benefits without the operational burden. GPFS is the default when you need enterprise features or are an IBM shop. Weka is the default when latency and small-random performance are the primary problem and flash budget is available.

None of them are magic. All of them will bite you if the fabric is not right, if the metadata layout does not match the workload, or if you skip benchmarking with realistic input. The time you spend on fabric tuning, metadata design, and representative load testing is the time that determines whether your filesystem is the foundation of your cluster or the thing your users curse at standups.