The error pointed at the GPU. The culprit was the HBA. — a Proxmox passthrough recovery - Outros Tutoriais - Download Free Software Gratis e Completos Webmaster Mocambique

The error pointed at the GPU. The culprit was the HBA. — a Proxmox passthrough recovery

Tópico: The error pointed at the GPU. The culprit was the HBA. — a Proxmox passthrough recovery
Categoria: Tutoriais | Programação & Tecnologia
Idioma Principal: Português (Conteúdo de Tecnologia)

Descrição do Conteúdo / Informações:
-------------------------------------------------------------------------
The plan looked textbook. PCIe passthrough an NVIDIA RTX A4000 from a Proxmox 8 host into an Ubuntu 24.04 VM, giving a Filecoin sealing worker direct GPU access for the compute phases that need it. VT-d on the Xeon. IOMMU enabled at the kernel line. OVMF BIOS on the VM. vfio-pci bound to the device IDs. Machine type set to q35 for PCIe support. Standard sequence, followed by every guide, ordered exactly as those guides ordered it.

Then qm start returned, and the VM did not come up.

Not a kernel panic. Not a host lockup. Not a "device not found" error surfaced to the operator. QEMU exited from the host's perspective and left nothing behind — no running guest, no console output, no obvious pointer at what had failed. The Proxmox web UI showed the VM as stopped, exactly as it had before I hit start.

The errors were there. They were just in a different terminal. dmesg -w on the host, running in a second window, produced a burst of IRQ allocation failures and vfio interrupt remapping errors scrolling past at VM startup, then silence. The host itself was completely fine throughout. Nothing was wrong with the box. Only the VM layer was broken, and it was broken in a way that read more like a misconfigured VM than a hardware conflict — which is exactly why the real cause took three sessions to identify.

The segmentation piece covered the network work that came before this. This piece is one of the workload-layer stories that followed — the GPU passthrough that took two evenings longer than it should have, because the error pointed at the GPU and the culprit was somewhere else entirely.

Two things this piece is not. It is not a step-by-step passthrough guide — there are plenty of those, and they are all fine until they aren't. It is the story of what happens when the checklist runs clean and the passthrough still refuses to work.

// what the GPU was actually there to do

Filecoin sealing — the process of turning raw storage into provable, verifiable data on the network — is not one workload. It's two compute phases with fundamentally different characteristics, and only one of them benefits from a GPU.

The first phase is SDR: Sequential Data Replication. It's designed to be resistant to parallelisation — memory-latency bound, CPU-heavy, and immune to GPU acceleration by design. On this hardware, SDR takes three to four hours per sector regardless of what else is in the box. No GPU speeds it up. That is intentional at the protocol level, not a limitation of the implementation.

The second phase is where the GPU matters. Tree building — specifically the TreeRC computations that produce the Merkle tree structure used in Proof-of-Replication — is GPU-eligible. Without acceleration, TreeRC runs on CPU and adds another two to three hours per sector. With the A4000 handling it, the same work completes in roughly fifteen to twenty minutes.

The framing that gets used in Filecoin marketing — "GPU acceleration makes sealing faster" — is technically true and operationally misleading. The GPU doesn't make sealing fast. It makes the second half of sealing fast. A storage provider without GPU acceleration is bottlenecked on tree computation as badly as on SDR, and the observable sealing throughput is roughly the SDR time plus the CPU tree time, sector by sector.

Where the GPU actually earns its place is in pipeline parallelism at scale. With the CPU pipeline running SDR on one sector while the GPU pipeline runs TreeRC on the previous one, sealing throughput becomes gated by SDR alone. That's the operator-relevant payoff — not making any single sector faster, but decoupling the two phases so they can run against different sectors concurrently. For a small storage provider, that is the difference between sealing two sectors a day and sealing five or six.

That was what the A4000 was there to do. Which is why the VM not coming up was a problem.

// the diagnostic that didn't converge

The investigation stretched across three sessions over about a week, mostly limited by the time I had rather than by the problem itself. Each session ran ninety minutes or so, and each one eliminated one correct-looking explanation without producing the actual cause.

Session one focused on the obvious: was the GPU actually bound to vfio-pci, and was the VM config correct?

lspci -nnk -d 10de:24b0

That output showed Kernel driver in use: vfio-pci — the host had surrendered the card, exactly as intended. No nouveau, no nvidia. The VM's PCI passthrough config referenced the correct BDF for the A4000, machine type was q35, OVMF was enabled, memory ballooning was off. Nothing in the config was wrong. That was session one.

Session two went down the driver blacklist and IOMMU group mapping path. /etc/modprobe.d/vfio.conf had the right device IDs listed. /etc/modprobe.d/blacklist.conf blocked nouveau, nvidia, nvidiafb, and snd_hda_intel. update-initramfs -u -k all had been run, and the host had been rebooted since. All correct.

Then the IOMMU group listing:

find /sys/kernel/iommu_groups/ -type l | sort -V

The A4000 at 03:00.0 appeared in one group with its HDMI audio companion at 03:00.1 — expected. But the same group also contained an IBM SAS2008 HBA. The HBA was managing the host's scratch storage, which meant it couldn't be passed through to the VM. And in an IOMMU group, all devices are the atomic unit of passthrough. You pass through the entire group or none of it.

That was where the pointer to the SAS2008 first appeared. But at that moment, I did not yet understand what shared-group membership specifically causes to fail. Group sharing is a widely documented passthrough concern, and the standard advice ("try the ACS override kernel patch") felt like a hack for a build that would need to run reliably long-term. I closed session two aware the SAS2008 was somehow involved but not yet certain how.

Session three was the crystallisation. Not a eureka moment — a slow narrowing. Once every other explanation was eliminated, the IOMMU group problem was the only candidate left. But the specific mechanism — why exactly a non-passthrough device in the same group breaks the passthrough of the target device — needed one more step of understanding before the physical fix made sense.

// FLR: what makes an IOMMU group problem actually a passthrough blocker

IOMMU group membership alone doesn't break passthrough. The Linux community's shorthand — "everything in the group has to be passed through together" — is correct but incomplete. It describes the safety constraint, not the failure mode. Understanding what actually goes wrong requires knowing what QEMU does at VM startup.

When the VM starts and takes ownership of a passed-through PCI device, QEMU has to reset it — bring it to a known state before the guest OS initialises. The reset mechanism it relies on is Function Level Reset (FLR), a PCIe capability that allows a device to be reset in isolation without affecting anything else on the bus. FLR is the clean, safe reset. It's what QEMU wants.

The A4000 supports FLR. The SAS2008 does not. And here is the mechanism: when QEMU tried to perform FLR on the A4000, it could not do so cleanly because the SAS2008 was in the same IOMMU group and could not itself be reset. The interrupt remapping infrastructure that isolates devices during passthrough could not correctly separate the GPU's IRQ handling from the HBA's, because from the IOMMU's perspective the two devices were treated as a single reset domain. The IRQ allocation failed. The VM didn't start.

The error pointed at the GPU because the GPU was the device QEMU was trying to bring up. The culprit was the HBA because the HBA was the device that couldn't cleanly get out of the way.

FLR support is directly checkable from the host. For any PCI device, the reset_method sysfs file lists the reset mechanisms the kernel considers available:

cat /sys/bus/pci/devices/0000:XX:XX.X/reset_method

For the SAS2008, that file did not list flr among its supported methods. That was the confirmation the mechanism I was reasoning about actually applied to this hardware. Once confirmed, the fix stopped being a question of ACS overrides or kernel patches. It became a question of physical topology.

This is the class of understanding most passthrough guides skip. They tell you what to configure. They don't tell you what to check when the configuration is right and the passthrough still fails. FLR-vs-non-FLR devices sharing an IOMMU group is the silent blocker at the heart of this failure mode.

// moving the HBA to a slot that mattered

The fix was physical. The SAS2008 had to move to a different PCIe slot — one that was not just physically distant from the A4000 but sat behind a genuinely separate PCIe root port. That distinction is critical and easy to miss.

Not every PCIe slot on a server board sits behind its own root port. Slot layout on the motherboard reflects physical trace routing, not IOMMU topology. Two adjacent slots may share a root port; two slots on opposite ends of the board may or may not. Moving the HBA one slot over on the same root complex would have done nothing — the IOMMU group membership would have been identical after reboot, and the failure would have repeated.

The target slot had to be chosen carefully. Two sources helped. The board's manual documents the PCIe topology — which slots connect to which CPU root complex, and how the lanes are allocated across the PCH. The existing IOMMU group listing also carries information: any two devices that had shown up in different groups before must, by definition, be behind different root ports. A slot known to be safe for the HBA was a slot whose currently-installed device (or its adjacent devices) had shown up in a group separate from the A4000's.

The physical work itself was straightforward. Full shutdown of the host — no live PCIe hotswap for this class of card. Chassis open. HBA out of its original slot. SAS cables temporarily disconnected (they had to be re-plugged after the move — a small operational risk if any cable connector had degraded, but nothing did). HBA into the target slot on the other root complex. Chassis closed. Boot.

Confidence going into that boot was moderate, not certain. PCIe slot physical location does not guarantee separate root complex membership — the only real proof is what the IOMMU group listing shows after boot. Moving the card is a testable hypothesis, not a guaranteed fix. If the target slot had turned out to be on the same root complex as the A4000 anyway, I would have been back at square one, chassis open again.

The fix confirmed itself in layers.

Structural first. After boot, the IOMMU group listing showed the SAS2008 in its own group, and the A4000 with only its audio companion at 03:00.1. That was the proof the physical move had produced the isolation it needed to produce. Before I even started the VM, I knew the group problem was resolved at the topology layer.

Functional second. VM started. QEMU didn't exit. Guest OS came up. nvidia-smi inside the VM reported the A4000 with its full 16 GB of memory and no error state. CUDA initialised. The card was talking to its driver.

Ecological third. The sealing worker initialised, picked up a queued sector, and the first GPU-accelerated TreeRC task completed in fifteen minutes instead of three hours. That was the confirmation the whole endeavour had been about.

Three checks, each validating a different layer, each necessary. The structural check proved the topology; the functional check proved the driver stack; the ecological check proved the workload actually benefited. Skipping any one of them would have left the fix incompletely verified.

// the second time was clean

The Filecoin project was later scrapped for reasons unrelated to the GPU passthrough — project constraints outside this story. The A4000 came back into the general hardware pool. When MMX came online later as a farming workload, the passthrough pattern needed to run again on a different guest OS.

MMX is a Chia fork with an important twist relevant to storage layout: it supports both HDD plots and SSD plots, and the SSD plots are compressed. Farming compressed plots requires GPU-accelerated decompression at every proof challenge — the harvester has to expand the plot data on the fly to check whether it holds a winning proof. Without a GPU, compressed SSD plotting is effectively non-viable at scale. With one, the decompression happens fast enough that the harvester keeps up with challenges in real time.

MMX and Chia have always run on Windows VMs on this network — a convention that predates this build, driven purely by the maturity of the Windows GUI tooling around Chia's ecosystem rather than any technical requirement. So the second passthrough build targeted a Windows guest instead of Ubuntu.

The build was clean. IOMMU group listing checked before touching physical hardware — target slot known-safe. Device IDs identified. VFIO binding confirmed. Windows installed with only the emulated VGA present, drivers applied after boot, GPU attached in a second pass to avoid the display blackout risk when the NVIDIA driver takes over primary output. Two extra CPU flags on the VM: hidden=1 to mask the hypervisor and avoid the notorious Error 43, and +pcid for TLB performance under memory-heavy workloads.

One clean build. No gotchas. Roughly one session, no diagnostic sessions after.

The pattern only reads as clean because the checks became routine. That is what the Filecoin project's expensive lesson bought.

// what I'd make instinctive

Four things went into the runbook after this.

Check IOMMU groups before touching physical hardware. The listing takes seconds to produce and tells you exactly what the passthrough constraints will be before you start. Every hour of build effort saved by checking the group listing first is an hour that doesn't need to be spent later, chassis open, moving cards under time pressure.

Non-FLR devices in the same group are silent blockers. IOMMU group membership on its own doesn't fail the passthrough. It's the combination of shared group and one device without FLR support that produces the specific failure mode that took me three sessions to identify. If your target device is in a group with a device that doesn't support FLR, that group needs to be broken up before passthrough will work. The reset_method sysfs check is thirty seconds and catches this class of problem definitively.

Physical slot choice is a topology question, not a proximity question. Adjacent slots may or may not sit behind separate root ports. Distant slots may or may not either. The board manual and the existing IOMMU group listing between them tell you which slots are actually independent. Guessing based on physical distance is worse than guessing at random, because it feels like a plan.

Layered confirmation matters — structural, functional, ecological. A working passthrough is not the same as a working workload. The structural check (IOMMU group listing) proves the topology. The functional check (nvidia-smi or equivalent) proves the driver stack. The ecological check (the actual workload running against real work) proves the whole system does what it was built to do. Skip any layer and the fix is only partly verified.

None of these are theoretical. Every one of them was learned by getting the diagnostic order wrong the first time.

// closing

The GPU passthrough pattern extends far beyond either of the workloads that surfaced in this piece. Any hardware acceleration that a VM needs — an AI or ML lab running CUDA training jobs, a transcoding server that needs hardware video encoders, a gaming VM that needs a discrete GPU, or a media server that needs GPU-accelerated encoding for streaming to remote clients — hits exactly the same setup questions, the same IOMMU group constraints, and the same class of neighbour-check that this piece is built around.

The unified lesson from this piece is small and load-bearing: the error pointed at the GPU. The culprit was the HBA. Check the neighbours before you blame the card.

If you're planning GPU passthrough on a Proxmox host, do this before you touch any physical hardware. Enumerate the IOMMU groups on the current configuration. Identify what shares a group with your target device. Check the reset_method sysfs entry for each of those neighbours. If any of them lacks FLR support, plan the physical move now — not later, when the VM refuses to start and the error message is pointing everywhere except at the actual problem.

The checklist runs clean. Until it doesn't. This is the check that catches when it doesn't.