[Discuss] Help with I/O errors on RAID array?

[dsr at randomstring.org (Dan Ritter)]

Bill Bogstad wrote: 
> On Sun, Oct 12, 2025 at 8:08?PM Daniel Barrett <dbarrett at 
> blazemonger.com> wrote:
> 
> I've never wrapped my head around exactly how NVME SSDs work, but they
> seem to be way more complicated then the essentially magnetic drive
> emulation that SATA SSDs use.   You might get some ideas by digging
> into NVME specs and specialized tools.   There is something called
> "nvme-cli" that might be helpful.


Bear with me on a history lesson:

Disks were expensive and came with expensive controllers that
only dealt with their own disks. This made everyone except drive
manufacturers unhappy.

Integrated Drive Electronics, IDE, took a ribbon cable running
off the 8-bit PC/XT bus and connected it to a controller mounted
on the drive itself, which meant that the drive manufacturers
had more competition but also a larger market.

ATA, later PATA, was IDE updated to the 16-bit wide PC/AT (yes,
286 era) bus. 

People wanted to connect other devices, like optical and tape
drives, so ATAPI was invented. ATAPI encapsulates SCSI commands
over the (parallel) ATA bus.

SATA is the ATA/ATAPI protocol running over a high-speed differential
serial protocol. 

SATA cabling is three grounds surrounding a transmit pair and a receive pair. 

The SATA cabling can also support a SCSI-only mode, called SAS -- SATA
Attached SCSI. This is purely a marketing-driven differentiator, as the
cost difference between the hardware for a SAS controller versus a SATA
controller is a few cents. SAS disks and controllers sell for more money.

NVMe is a multiqueued protocol for reading and writing blocks of data,
which doesn't care what it's transported over or most details of the
storage device at the other end. It performs the same function as SCSI,
though the details are different.

The most common transport is a PCIe bus. Each PCIe "lane" is a high-speed
differential serial transmit pair and receive pair.  (Sound familiar?)
M.2 and U.2 and U.3 are just physical ways of getting some PCIe lanes
into a compact, pluggable interface.

An NVMe host expects an NVMe target device controller on the other end of
the transport, which will translate to the specific storage medium. One
could connect rotating disks to an NVMe target controller, but I don't
think anyone ever has.

I say "an NVMe host", but that doesn't really exist in hardware: the
target controller does all of the work, and the host is just whatever
CPU and RAM system is connected to the transport, which needs to have
software to attach, identify and command targets.

Practically, that means that if someone comes out with an NVMe 1.x
revision, once the Linux kernel has drivers updated, all the new devices
will be expected to work on existing hosts. And, in fact, that
happened this August. 

The nvme-cli utility is mostly of interest to people running NVMe over
something other than a PCIe bus -- generically, "fabrics" --  because
which target is the host supposed to contact? That's where that gets
configured.

One thing it can do, though, is retrieve target device logs,
which might be useful here in figuring out what's going on.

-dsr-