Parallel NFS (pNFS) Flexible File Layout Version 2

Introduction In Parallel NFS (pNFS) (see Section 12 of ), the metadata server returns layout type structures that describe where file data is located. There are different layout types for different storage systems and methods of arranging data on storage devices. defined the Flexible File Version 1 Layout Type used with file-based data servers that are accessed using the NFS protocols: NFSv3 , NFSv4.0 , NFSv4.1 , and NFSv4.2 . A metadata server that supports the Flexible File Version 2 Layout Type MUST be an NFSv4.2 server. The new operations defined by this document for the metadata server (the TRUST_STATEID family on the metadata server / storage device control session, and the CB_CHUNK_REPAIR back-channel callback to clients) are NFSv4.2 operations and have no representation in NFSv4.1 or earlier minor versions. Storage devices can speak NFSv3, NFSv4.1, or NFSv4.2, but some encoding types and coupling configurations narrow that choice; see for the exact rules. To provide a global state model equivalent to that of the files layout type, a back-end control protocol might be implemented between the metadata server and NFSv4.1+ storage devices. An implementation can either define its own proprietary mechanism or it could define a control protocol in a Standards Track document. The requirements for a control protocol are specified in and clarified in . The control protocol described in this document is based on NFS. It does not provide for knowledge of stateids to be passed between the metadata server and the storage devices. Instead, the storage devices are configured such that the metadata server has full access rights to the data file system and then the metadata server uses synthetic ids to control client access to individual data files. In traditional mirroring of data, the server is responsible for replicating, validating, and repairing copies of the data file. With client-side mirroring, the metadata server provides a layout that presents the available mirrors to the client. The client then picks a mirror to read from and ensures that all writes go to all mirrors. The client only considers the write transaction to have succeeded if all mirrors are successfully updated. In case of error, the client can use the LAYOUTERROR operation to inform the metadata server, which is then responsible for the repairing of the mirrored copies of the file. This client side mirroring provides for replication of data but does not provide for integrity of data. In the event of an error, a user would be able to repair the file by silvering the mirror contents. I.e., they would pick one of the mirror instances and replicate it to the other instance locations. However, lacking integrity checks, silent corruptions are not able to be detected and the choice of what constitutes the good copy is difficult. This document defines the Flexible File Version 2 Layout Type, an independent layout type that adds error-detection integrity (checksum) for erasure coding. It does not modify the Flexible File Version 1 Layout Type (); the two coexist. Data blocks are transformed into a header and a chunk. This document also introduces new operations that allow the client to roll back writes to the data file. Using the process detailed in , the revisions in this document become an extension of NFSv4.2 . They are built on top of the external data representation (XDR) generated from . This document defines LAYOUT4_FLEX_FILES_V2, a new and independent layout type that coexists with the Flexible File Version 1 Layout Type (LAYOUT4_FLEX_FILES, ). The two layout types are NOT backward compatible: a flexible file v2 layout cannot be parsed as a flexible file v1 layout and vice versa. A server MAY support both layout types simultaneously; a client selects the desired layout type in its LAYOUTGET request.

Requirements Language The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here.

Motivation Workloads that need both the throughput of parallel pNFS data servers and the durability of erasure coding have driven the work in this draft. The deployments span scientific-instrumentation pipelines that produce petabytes of detector data per year, training-checkpoint files written hot from machine-learning jobs, archive workloads in which a single file may hold the only copy of an experiment's result, and ordinary production filesystems where read patterns evolve across a file's lifetime. These deployments are documented in detail in ; the protocol shape that follows is the result of looking at them and asking what a pNFS layout type would have to provide. The first thing the deployments share is that erasure coding moves work off the data servers that are already the bottleneck in the write-heavy parallel case. Server-side erasure coding makes each data server compute its share of the parity transform on every write, multiplying the per-write CPU cost by (k + m) across the storage tier and serialising on the data server's limited compute. Client-side erasure coding shifts that compute to the writers, which scale horizontally with the workload, and lets the data servers stay close to their strength -- storing and serving bytes. Flexible file v1 layout () already chose client-side compute by placing replication at the writer; this draft extends that choice to client-side erasure coding. Benchmark measurements summarised in confirm that the resulting overhead is competitive with server-side encoding on realistic workloads and that the encoding compute scales with the writer population rather than with the data-server count. Client-side erasure coding has a corollary that protocol designers cannot avoid: when a client fans a stripe out across multiple data servers and fails mid-write, no single data server has whole-transaction visibility. The state left behind on each data server is a partial fragment of a write that may or may not have completed elsewhere. A server-side coordinator that holds the whole stripe -- the flexible file v1 case -- can resilver from a surviving copy without any client involvement. In the v2 case there is no such coordinator, and the on-wire protocol specifies how the partial state is reconciled. This is the load-bearing constraint that shapes the rest of the design. A natural-looking answer is to add a distributed-consensus protocol between the data servers and have them agree on which write committed. That answer is rejected here. Distributed consensus is operationally expensive, introduces a synchronisation cost on every write, and makes the data servers themselves stateful peers in a way that closes off the simpler implementations the protocol is designed to accommodate. Instead, this draft uses two narrowly-scoped primitives that together provide just enough on-wire reconciliation: the chunk_guard4 compare-and-swap (CAS) and the CB_CHUNK_REPAIR callback. Every CHUNK_WRITE carries a chunk_guard4 -- a 32-bit per-chunk generation counter and a 32-bit owning-client short-id -- and the data server performs a per-chunk CAS on receipt. If two writers race for the same chunk, exactly one wins on each data server, the loser receives NFS4ERR_CHUNK_GUARDED for that chunk, and the chunks the loser intended to write are left unchanged. No data server needs to consult its peers; the CAS is local. The cost on the metadata server is bounded by the 8-byte chunk_guard4 header per chunk plus a 32-bit per-layout client identifier (, ). Independent collisions on different chunks resolve independently; there is no file-wide lock and no global ordering across writes. When the per-chunk CAS detects that a stripe ended up non-atomic -- some shards under writer A's guard, others under writer B's, or a writer crashed mid-fan-out -- the metadata server selects a repair client via CB_CHUNK_REPAIR () and that client drives the repair: it acquires CHUNK_LOCK on the affected range, reads the surviving shards, decodes through the erasure transform, writes the reconstructed shards via CHUNK_WRITE_REPAIR, and clears the errored state via CHUNK_REPAIRED. The repair client is exactly one actor holding a chunk-range lock; the data servers still do not coordinate among themselves. The repair-client selection rule is given in . These two primitives -- the per-chunk CAS and the callback-driven repair -- replace what would otherwise require a distributed-consensus protocol. The CAS handles the common case at local cost, where independent writers racing for different chunks resolve independently and concurrent writers on the same chunk get a clean win/loss decision. CB_CHUNK_REPAIR handles the rare case of partial-failure non-atomicity with a single coordinator selected per repair episode. The cost model is asymmetric on purpose: the hot path pays for an 8-byte header and a local CAS; the cold path pays for a selected actor and a small number of round-trips. The CHUNK_* operation set in this draft is the minimum sufficient to drive the chunk state machine () plus the repair flow above. CHUNK_WRITE places PENDING content; CHUNK_FINALIZE signals that the writer is done with a generation; CHUNK_COMMIT promotes that generation to durable, globally visible state; CHUNK_READ retrieves it; CHUNK_HEADER_READ provides the fast probe that lets repair coordinators and recovering writers inspect chunk metadata without reading payloads. CHUNK_LOCK, CHUNK_UNLOCK, CHUNK_ERROR, CHUNK_REPAIRED, CHUNK_WRITE_REPAIR, and CHUNK_ROLLBACK together drive the repair flow. Each operation does one well-scoped job; the complexity is in the state machine the operations drive, not in the operation set itself. Each of these primitives closes a specific gap in the lifecycle or the repair path. The detailed treatment of the operation set is in . The same design discipline shapes the rest of the specification. The protocol describes wire format and server obligations; it does not pin a data-server backend, a control protocol between metadata server and data server, a checksum algorithm, or a file-attribute representation on the data server. Different implementations resolve these choices differently and remain conformant. TRUST_STATEID () is one such control protocol that this draft defines; storage devices with their own established control protocols are conformant without implementing it. The tagged checksum4 () lets the metadata server pick any registered checksum algorithm per file. The authorization-outcome parity rule () lets data servers that do not expose a POSIX file namespace satisfy the tight-coupling requirements without materialising POSIX uid/gid bits. A protocol-level consequence of placing erasure coding at the client is that the layout is able to describe a file's storage shape over its full lifetime -- including the transition windows when the file is being assimilated from a non-erasure-coded source, re-encoded from one encoding to another, or recovered from a correlated encoding failure through a mirror under a different encoding. This draft allows a single file's layout to contain mirrors under different encodings. The heterogeneous-mirror capability is not a steady-state expectation; most files have one encoding most of the time. It is the protocol shape that lets transitions happen while the file remains readable. The deployment cases that drive the allowance are catalogued in . Scope note: the consistency goal of flexible file v2 layout is RAID consistency across the shards that make up an encoded stripe, not POSIX write ordering across arbitrary application writes. The protocol does not attempt to make overlapping application writes from different clients atomic; that is the province of file locking ( Section 12) and of application-level coordination. What the protocol does guarantee is that the shards comprising a given stripe agree on which write produced them -- expressed on the wire as agreement on the chunk_guard4 value of every chunk that carries those shards -- so that readers and repair clients never observe a half-applied stripe. Readers who need cross-write ordering beyond a single stripe MUST use the existing NFSv4 locking primitives.

Use Cases The protocol is designed around three workload classes. The labels below reflect the relative frequency of each class in installations that choose flexible file v2 layout for its combination of integrity and performance; individual deployments may diverge.

Single writer, multiple readers (the common case):: A file written by one client and subsequently read by many. Examples include artifacts deposited by batch jobs, container images, and media files. The protocol is optimized for this case; see .
Multiple writers without sustained contention (occasional):: Files with multiple concurrent writers where races on the same chunk are rare. Examples include shared-directory append-only logs and distributed builds. The chunk_guard4 CAS primitive and per-chunk locking cover this case without penalizing the common single-writer path.
Multiple writers, disjoint regions (rare):: High-performance computing (HPC) checkpoint workloads, in which many ranks write disjoint regions of the same file in lockstep. The protocol relies on block alignment to keep per-chunk contention rare despite overall high writer count. Contention that does occur is resolved via the deterministic tiebreaker rule defined in .

Scale targets include multi-thousand-client deployments (on the order of tens of thousands of concurrent clients for HPC checkpointing), parallel-filesystem replacements, and multi-rack shared-storage clusters. The repair protocol (see ) is designed to let such deployments tolerate data-server failures and concurrent-writer races without blocking the critical path for the first two workload classes.

Definitions

block:: the application's view of file data. A block is a unit of file content as observed by an NFS client at the POSIX layer or by the local file system. A chunk's payload, after any decoding the client performs, is presented to the application as one or more blocks.
shard:: the encoding's view. A shard is a single piece of an encoded stripe produced by an erasure-coding (or replication) transformation. A stripe of k data shards plus m parity shards is the unit an encoding encodes and decodes. The word "shard" only has meaning while the encoding is reasoning about a stripe; once a shard is at rest on a data server it is, by virtue of having been transmitted, the payload of a chunk.
chunk:: the protocol's unit of file data on the wire, carrying an envelope that distinguishes it from a block: a compare-and-swap guard (chunk_guard4 -- atomicity, see ), a checksum (per-chunk integrity), a provenance identifier (chunk_owner4, see ), a lifecycle state (PENDING / FINALIZED / COMMITTED via the chunk state machine, see ), and per-chunk locking that survives stateid revocation through lock escrow. A chunk is the addressable unit named in the CHUNK_* operations defined in this document and durably persisted by a data server. A chunk's payload may be a block (mirrored layout) or a shard (erasure-coded layout); the wire protocol does not distinguish. The chunk size MAY differ from the size of the block or shard it carries. See for the load-bearing role each envelope property plays in the protocol's consistency story.

The three terms describe the same data at three different layers and should be used accordingly. The encoding transforms blocks into shards; the wire protocol transmits shards as chunk payloads; the data server persists chunks. On read the path reverses. A protocol-internal note: the chunk state machine () and several CHUNK_* operations refer to the per-chunk-offset state records as "blocks" (PENDING / FINALIZED / COMMITTED / errored). This is a finer-grained use of the word, internal to the data server's chunk metadata, and should not be confused with the application-layer "block" defined above. Where ambiguity matters, this document writes "chunk-state block" or relies on context (operation names, state names) to disambiguate.

control communication requirements:: the specification for information on layouts, stateids, file metadata, and file data that must be communicated between the metadata server and the storage devices. There is a separate set of requirements for each layout type.
control protocol:: the particular mechanism that an implementation of a layout type would use to meet the control communication requirement for that layout type. This need not be a protocol as normally understood. In some cases, the same protocol may be used as a control protocol and storage protocol.
client-side mirroring:: a feature in which the client, not the server, is responsible for updating all of the mirrored copies of a layout segment.
data block:: A block (as defined above) in the client's cache for a file.
data file:: The data portion of the file, stored on the data server.
replication of data:: Data replication is making and storing multiple copies of data in different locations.
erasure coding:: A data protection scheme where a stripe of data is encoded into shards (k data shards and m parity shards) so that the original content can be reconstructed from any sufficient subset of the shards. Shards are transmitted as the payload of CHUNK operations and stored on different data servers.
client-side erasure coding:: Erasure coding in which the encode and decode transforms are performed on the pNFS client, and encoded shards are written directly to the data servers (rather than being computed at a server-side coordinator). This is the deployment shape Flexible File Version 2 Layout Type is designed for; the chunk substrate and CHUNK_* operations that make it safe are specified in and . Contrast: server-side erasure coding, where a coordinator holds the entire stripe and produces parity, is out of scope for this document.
compare-and-swap (CAS):: an atomic primitive from concurrent programming in which an update is conditional on a prior observed value: the operation succeeds only if the current value matches an expected prior value, and otherwise fails so the caller can retry. In this document, the chunk_guard4 mechanism (see ) implements CAS at the chunk level; the "expected prior value" is the chunk_guard4 the writer observed at read time, and the "fail" outcome is NFS4ERR_CHUNK_GUARDED.
(file) data:: that part of the file system object that contains the data to be read or written. It is the contents of the object rather than the attributes of the object.
data server (DS):: a pNFS server that provides the file's data when the file system object is accessed over a file-based protocol.
escrow (lock escrow, MDS-escrow):: a state in which a chunk lock is held by the metadata server on behalf of an as-yet-unselected future owner. When the metadata server revokes a client's stateid while the client still holds chunk locks, the locks are not dropped (which would expose the chunks to concurrent writers) but are transferred to the metadata server itself, marked by the reserved cg_client_id value CHUNK_GUARD_CLIENT_ID_MDS (see ). The metadata server holds the locks in escrow until a repair client adopts them via CHUNK_LOCK with CHUNK_LOCK_FLAGS_ADOPT (driven by CB_CHUNK_REPAIR). An "MDS-escrow owner" is the metadata server acting in this placeholder role; "in escrow" describes a lock in this state. Escrow preserves the lock-continuity invariant across stateid revocation: at no point during the revocation sequence is a chunk simultaneously locked and unowned.
fencing:: the process by which the metadata server prevents the storage devices from processing I/O from a specific client to a specific file.
file layout type:: a layout type in which the storage devices are accessed via the NFS protocol (see Section 5.12.4 of ).
gid:: the group id, a numeric value that identifies to which group a file belongs.
layout:: the information a client uses to access file data on a storage device. This information includes specification of the protocol (layout type) and the identity of the storage devices to be used.
layout iomode:: a grant of either read-only or read/write I/O to the client.
layout segment:: a sub-division of a layout. That sub-division might be by the layout iomode (see Sections 3.3.20 and 12.2.9 of ), a striping pattern (see Section 13.3 of ), or requested byte range.
layout stateid:: a 128-bit quantity returned by a server that uniquely defines the layout state provided by the server for a specific layout that describes a layout type and file (see Section 12.5.2 of ). Further, Section 12.5.3 of describes differences in handling between layout stateids and other stateid types.
layout type:: a specification of both the storage protocol used to access the data and the aggregation scheme used to lay out the file data on the underlying storage devices.
loose coupling:: when the control protocol is a storage protocol.
(file) metadata:: the part of the file system object that contains various descriptive data relevant to the file object, as opposed to the file data itself. This could include the time of last modification, access time, EOF position, etc.
metadata server (MDS):: the pNFS server that provides metadata information for a file system object. It is also responsible for generating, recalling, and revoking layouts for file system objects, for performing directory operations, and for performing I/O operations to regular files when the clients direct these to the metadata server itself.
mirror:: a copy of a layout segment. Note that if one copy of the mirror is updated, then all copies must be updated.
non-systematic encoding:: An erasure coding scheme in which the encoded shards do not contain verbatim copies of the original data. Every read requires decoding, even when no shards are lost. The Mojette non-systematic transform is an example.
proxy server (PS):: a peer of the metadata server, defined in , that admits client I/O on the metadata server's behalf -- either as a translator for clients that cannot speak the file's native encoding, or as a proxy-mediated data path during whole-file move and repair operations. A proxy server may additionally act as a data server.
recalling a layout:: a graceful recall, via a callback, of a specific layout by the metadata server to the client. Graceful here means that the client would have the opportunity to flush any WRITEs, etc., before returning the layout to the metadata server.
revoking a layout:: an invalidation of a specific layout by the metadata server. Once revocation occurs, the metadata server will not accept as valid any reference to the revoked layout, and a storage device will not accept any client access based on the layout.
resilvering:: the act of rebuilding a mirrored copy of a layout segment from a known good copy of the layout segment. Note that this can also be done to create a new mirrored copy of the layout segment.
rsize:: the data transfer buffer size used for READs.
stateid:: a 128-bit quantity returned by a server that uniquely defines the set of locking-related state provided by the server. Stateids may designate state related to open files, byte-range locks, delegations, or layouts.
storage device:: the target to which clients may direct I/O requests when they hold an appropriate layout. See Section 2.1 of for further discussion of the difference between a data server and a storage device.
storage protocol:: the protocol used by clients to do I/O operations to the storage device. Each layout type specifies the set of storage protocols.
systematic encoding:: An erasure coding scheme in which the first k of the k+m encoded shards are identical to the original k data blocks. A healthy read (no failures) requires no decoding -- the data shards are read directly. Decoding is triggered only when data shards are missing. Reed-Solomon Vandermonde and Mojette systematic are examples.
tight coupling:: an arrangement in which the control protocol is one designed specifically for control communication. It may be either a proprietary protocol adapted specifically to a particular metadata server or a protocol based on a Standards Track document. The specific tight-coupling variant defined by this document, in which the control protocol is the TRUST_STATEID family, is referred to as trusted-stateid tight coupling (see ).
trusted-stateid tight coupling:: the specific tight-coupling control protocol defined in this document, consisting of the operat