Managed with AI assistance. Human-supervised; each guide shows its review status.
🛠️ Real incident, fixed with AI

Recovering Proxmox from RTL9210 UAS Transport Failures

A Proxmox host kept aborting its EXT4 journal because a USB-to-NVMe bridge was unstable under UAS; disabling UAS for that exact device fixed it.

Recovering Proxmox from RTL9210 UAS Transport Failures

Symptoms

Environment

Credentials, serial numbers, backup hashes, private addresses, local paths, and raw logs are omitted.

Evidence

The recovery started by treating the original image as immutable. It was attached read-only, and repairs were rehearsed in QCOW2 overlays:

qemu-img info <master-image>
qemu-img create \
  -f qcow2 \
  -F raw \
  -b <master-image> \
  <repair-overlay.qcow2>

qemu-img create \
  -f qcow2 \
  -F qcow2 \
  -b <repair-overlay.qcow2> \
  <disposable-boot-test.qcow2>

qemu-img info --backing-chain <disposable-boot-test.qcow2>

Non-modifying checks found limited metadata damage consistent with unclean shutdowns. No completed check reported bad blocks:

e2fsck -f -n <filesystem-device>
mount -o ro,noload <filesystem-device> <mountpoint>

Repairs performed in an overlay passed complete read-only checks, and a disposable child overlay booted to the normal Proxmox login prompt. This proved that the repair plan was internally consistent without modifying the source image.

The physical SSD was then inspected from a RAM-resident recovery environment. Its health data showed:

smartctl --scan-open
smartctl -x <bridge-exposed-device>
nvme smart-log <native-nvme-device>
nvme error-log <native-nvme-device>

These results did not prove the entire storage path was healthy, but they strongly contradicted active NAND failure. After the physical filesystems were repaired and verified offline, aborted journals still returned during a real boot. That recurrence shifted the investigation from on-disk damage to the live transport path.

The bridge was identified by its public USB vendor and product ID:

lsusb
lsusb -t

Linux supports a device-specific usb-storage quirk that disables UAS for an exact USB ID:

usb-storage.quirks=0bda:9210:u

The u flag maps to US_FL_IGNORE_UAS. Testing the quirk from the RAM-resident recovery environment caused this bridge to bind to usb-storage instead of uas, while the physical link remained negotiated at 10 Gb/s.

Root cause

The best-supported cause was instability in the RTL9210 bridge’s UAS transport path, not SSD media failure or remaining filesystem damage.

The evidence established that:

The evidence did not isolate whether bridge firmware, cable quality, power delivery, or a particular kernel interaction was the lowest-level trigger. It also does not imply that every RTL9210 enclosure requires this workaround.

Recovery

1. Preserve recoverability

Do not repair or boot the only trusted backup directly. Keep the source read-only, test repairs in overlays or a clone, and verify each modified filesystem afterward:

e2fsck -f <filesystem-device>
e2fsck -f -n <filesystem-device>

Only run repair commands against positively identified, unmounted filesystems. Device names such as /dev/sda are not stable identities.

2. Test the exact bridge without UAS

From a RAM-resident recovery environment with the target filesystems unmounted, set the quirk and discover the active USB interface from sysfs:

printf '%s\n' '0bda:9210:u' \
  > /sys/module/usb_storage/parameters/quirks

readlink -f /sys/bus/usb/devices/*/driver
lsusb -t

Rebind only the matching interface:

printf '%s\n' '<usb-interface-id>' \
  > /sys/bus/usb/drivers/uas/unbind

printf '%s\n' '<usb-interface-id>' \
  > /sys/bus/usb/drivers/usb-storage/bind

Rebinding can immediately remove a mounted root disk. Do not perform this from the installed system while that disk is in use.

Confirm the selected driver and quirk:

lsusb -t
dmesg -T | grep -Ei '0bda|9210|uas|usb-storage|quirk'

3. Apply the quirk during early boot

Because the root filesystem was behind the bridge, configuring the quirk after boot would be too late. Add it to both the kernel command line and the initramfs module configuration:

cat >/etc/default/grub.d/99-rtl9210-usb-storage.cfg <<'EOF'
GRUB_CMDLINE_LINUX="$GRUB_CMDLINE_LINUX usb-storage.quirks=0bda:9210:u"
EOF

cat >/etc/modprobe.d/rtl9210-usb-storage.conf <<'EOF'
options usb-storage quirks=0bda:9210:u
EOF

update-initramfs -u -k <known-good-kernel-version>
update-grub

Inspect the generated artifacts before rebooting:

lsinitramfs /boot/initrd.img-<known-good-kernel-version> \
  | grep rtl9210-usb-storage.conf
grep -n 'usb-storage.quirks=0bda:9210:u' /boot/grub/grub.cfg

Use a kernel already validated with the host’s other hardware and out-of-tree drivers.

Verification

A clean boot alone was not treated as proof. A temporary logical volume was created only from confirmed free space. A random pattern generated outside the storage under test was written with direct I/O, read back, and compared:

lvcreate -L 4G -n <temporary-test-lv> <volume-group>

dd if=/dev/urandom of=<ram-backed-pattern> \
  bs=1M count=256 status=progress
sha256sum <ram-backed-pattern>

dd if=<ram-backed-pattern> of=<temporary-lv-device> \
  bs=4M oflag=direct conv=fsync status=progress

dd if=<temporary-lv-device> bs=4M iflag=direct count=64 status=none \
  | sha256sum

The pattern was repeated across the full temporary LV. All segments matched, with no USB resets, kernel I/O errors, new SMART errors, media errors, or hash mismatches. The temporary LV was then removed:

lvremove <temporary-lv-device>
lvs -a

After reboot, verify the early-boot configuration, active transport, storage health, and current-boot logs:

uname -r
cat /proc/cmdline
lsusb -t
systemctl --failed
pvesm status
lvs -a -o +data_percent,metadata_percent
journalctl -k -b | grep -Ei \
  'ext4|journal|I/O error|buffer I/O|usb|uas|scsi|nvme|reset|abort'

Expected results were:

Prevention

References