Information
Throughput charts tell only half the story of a file protocol. A server that saturates a fast link while consuming every core has little left for the applications that need the data. A better yardstick is CPU cost per gigabyte NAS protocols impose, meaning how much processor time it takes to move each gigabyte. Here is how to think about it and measure it on your own gear.
Why throughput alone misleads
Two protocols can both reach line rate while one burns twice the processor time. On a lightly loaded storage server that hardly matters, but on a busy host running virtual machines, analytics or machine learning training, the cycles spent copying and checksumming data compete with real work. Efficiency decides how many clients a node can serve before the CPU, not the network, becomes the limit.
Memory bandwidth is part of the picture too. Every extra copy of a payload consumes memory traffic as well as cycles, so a path that touches the data fewer times can scale better across many concurrent streams. This is one reason the CPU cost per gigabyte NAS protocols incur can differ more than raw bandwidth would suggest.
Where the cycles go
Conventional TCP paths spend CPU on interrupt handling, kernel and user-space buffer copies, checksum work and protocol processing, partly reduced by offloads in modern network adapters. Encryption and signing add more per-byte cost. RDMA changes the shape by letting the adapter place data directly into memory, bypassing much of the copying and kernel network stack.
Cores also matter in a less obvious way. A single TCP connection is often handled largely by one core, so a protocol path that cannot spread work across connections can hit a ceiling well before the network does. Multichannel and multistream approaches address that by creating parallel flows.
The protocol paths worth comparing
On the Windows side, SMB Multichannel spreads a session across multiple connections or interfaces, while SMB Direct runs SMB over RDMA. On the Linux side, NFS over RDMA offers a similar bypass, and multistream TCP, commonly enabled with the nconnect mount option, opens several TCP connections to the same server to improve parallelism.
A conference session at the Storage Developer Conference 2026 from Xinnor compares these approaches using the MLPerf Storage benchmark. The published abstract gives no results, so this article makes no claims about which path wins.
Each option has prerequisites. SMB Direct depends on RDMA-capable adapters on both ends and supported operating system versions, and NFS over RDMA depends on matching client and server support. Verify that the path you think you are testing is actually in use, for example by checking the connection details reported by the client.
Building a fair test
Keep everything constant except the protocol. Use identical servers, clients, network adapters, firmware and file system settings, and the same dataset, block sizes and thread counts. Warm the cache the same way for each run, or bypass it, and repeat each test several times so you can see variance.
Run workloads that resemble yours. Sequential large reads behave differently from many small random operations, and a training workload differs from a file share full of office documents.
Do not compare an optimized configuration of one protocol with a default configuration of another. Tune each with equal care, document every setting, and keep the configuration alongside the results so that colleagues can reproduce them.
Measuring CPU per gigabyte yourself
Record CPU utilization on both client and server with tools such as mpstat or sar while a benchmark tool such as fio generates load. Multiply average utilization by core count and duration to get CPU-seconds, then divide by gigabytes transferred. Track client and server figures separately, because RDMA may shift cost between them.
Also note latency and tail behavior, since a cheap protocol that stalls under contention is not a bargain.
A short table helps: for each protocol and workload, list throughput, server CPU-seconds per gigabyte, client CPU-seconds per gigabyte and the 99th percentile latency. Patterns in that table, not a single number, should guide the decision.
Cost, compatibility and operations
Efficiency has to be weighed against practicalities. RDMA needs compatible adapters, switches configured for lossless or tuned operation, and client support, all of which add complexity. Standard multistream TCP is easier to deploy and may be good enough if your processors have headroom. Signing and encryption requirements should be included in the test, because they are often mandatory in real deployments.
Failure behavior deserves a test as well. Pull a link, restart a server and see how each protocol recovers, because efficient steady-state performance is of limited value if reconnection is slow or disruptive.
Tying results back to architecture
Protocol efficiency is one input to a larger design. If a single node cannot deliver the bandwidth you need even at low CPU cost, scale-out NAS architecture lets you add nodes and spread clients across them. For general planning, purpose-built NAS storage is built around file-serving workloads, and a review of how SAN, NAS and DAS differ helps confirm that file protocols are the right fit before you tune them.
Once you know your own figures for CPU cost per gigabyte NAS protocols produce, capacity planning becomes arithmetic. Divide available CPU by the per-gigabyte cost to estimate how much traffic a node can sustain, then leave headroom for backups, snapshots and rebuilds.
Final thoughts
Choosing a protocol on headline speed alone hides the real expense. Measure CPU per gigabyte under conditions that resemble your workload, keep the comparison fair, and weigh the savings against operational complexity. The numbers you collect yourself will be more trustworthy than any generic ranking.
Add comment
Comments