Lossless Ethernet for storage: DCB, PFC, ECN

Đội ngũ của chúng tôi đang thực hiện bản dịch. Bài viết này tạm thời chưa có bản tiếng Việt và đang được hiển thị bằng tiếng Anh.

Ethernet drops frames when it is congested; storage protocols built for Fibre Channel or RDMA assume nothing is ever dropped. Data Center Bridging (DCB) closes the gap with per-priority flow control and bandwidth allocation, and RoCE adds congestion signalling on top. The result is only as lossless as its worst link: a marginal optic that drops one frame in a million is a storage incident. This page explains the mechanisms and translates them into physical-layer requirements.

The DCB toolkit

Standard	Name	What it does
IEEE 802.1Qbb	PFC — priority-based flow control	pause frames per traffic class (0–7): a congested receiver stops one class without stopping the port
IEEE 802.1Qaz	ETS — enhanced transmission selection	guaranteed bandwidth shares per class (e.g. 50 % storage, 50 % LAN)
IEEE 802.1Qaz	DCBX — DCB exchange	LLDP-based negotiation of PFC/ETS settings between switch and NIC
IEEE 802.1Qau	CN — congestion notification	rarely deployed
RFC 3168 + DCQCN	ECN marking + rate control	the switch marks packets when queues build; RoCE NICs slow the flow before PFC is needed

How RoCE stays lossless

Storage/RDMA traffic is tagged with a dedicated priority (often 3 or 4) via DSCP or 802.1p.
PFC is enabled only for that priority on every port along the path, with buffer headroom sized for the link speed and cable length (in-flight bytes).
ECN thresholds mark packets early; NICs run DCQCN to reduce their rate, so PFC fires only as a last resort.
ETS guarantees the storage class its bandwidth under LAN load.
DCBX keeps NIC and switch settings consistent; mismatches silently break losslessness.

FC and InfiniBand achieve the same with credits at every link by default; Ethernet needs it configured, verified and monitored (FC protocol basics, InfiniBand transport).

What can go wrong

Problem	Effect	Detection
PFC not enabled on one hop / wrong priority	drops under load; RDMA retransmits; latency spikes	pause counters zero where they should not be; drops on the class
PFC storm	a stuck receiver pauses upstream indefinitely; congestion spreads fabric-wide	pause frames flooding; watchdog counters
PFC deadlock	cyclic buffer dependency in Clos fabrics; all traffic on the class stops	requires deadlock detection / watchdog to break
Headroom too small for long cables/optics	drops despite PFC	drops with pause counters active
ECN thresholds wrong	too early: throughput loss; too late: PFC floods	ECN mark counters, CNP counters on NICs
Physical errors on a link (CRC, FEC uncorrectable)	frames lost that no flow control can recover	port error counters, FEC counters — VDM & FEC metrics

iSCSI without DCB

iSCSI runs over TCP and survives loss, but pays for it in latency and retransmission. Best practice regardless of DCB:

dedicated VLANs/subnets per fabric, two fabrics, MPIO on hosts;
jumbo frames (MTU 9000) end to end;
no oversubscription between initiators and targets; 25G/100G uplinks sized for the array;
FEC on for every 25G+ link; CRC counters at zero;
optionally PFC on the iSCSI class to smooth micro-bursts (many arrays recommend it).

Physical-layer requirements

Requirement	Why	Practice
FEC enabled and matched on every 25G+ link	RoCE cannot tolerate the raw BER of unprotected 25G/50G lanes	FEC & link training
CRC and uncorrectable FEC = 0	each lost frame stalls a queue	alert on any increment
Rx power inside the window with margin	marginal optics produce bursts of errors under temperature	Rx power & link budget
Consistent speed/duplex/AN	DAC needs AN + link training both ends	Copper & DAC
Cable length within headroom assumptions	headroom is computed from in-flight bytes	prefer DAC in rack, SR/AOC in row, DR/LR between rooms
Identity accepted by NIC and switch	rejected optics fall back to no link or reduced speed	Vendor lock

Monitoring checklist

Per port: pause frames sent/received per priority, ECN marks, drops per class, CRC, FEC corrected/uncorrectable, link flaps.
Per NIC: CNPs sent/received, retransmissions, out-of-sequence, timeouts.
Per optic: Rx/Tx power and temperature trends (Monitoring).
Correlate storage tail-latency spikes with the above — the physical layer is guilty more often than the array.

In CodingBox

A lossless fabric is built from ordinary optics that are simply held to a higher standard. Incoming inspection on the bench — identity, checksums, live DDM against typical values, baseline stored — is how marginal modules are kept out of a fabric where one bad link costs an application its latency SLA (Check transceiver, DDM).

Where PFC and ECN sit among the other forwarding mechanisms, and how physical-layer errors surface in them: Forwarding basics.

Nếu bạn phát hiện thông tin chưa chính xác hoặc lỗi trong bài viết này, hãy chọn đoạn liên quan và nhấn Ctrl+Enter để .