Skip to content

[Bug]: Segfault in VBoxDD.so (NVMe#0 worker) when VD layer cancels timed-out I/O request under host write load #853

Description

@jeremyakers

Version

7.2.14

Host OS Type

Linux

Host OS name + version

Ubuntu 22.04 LTS, kernel 5.15.0-190-generic

Host Architecture

x86

Guest OS Type

Linux

Guest Architecture

x86

Guest OS name + version

Home Assistant OS 18.2 (haos_ova / qemux86-64 image)

Component

Storage and Virtual Disks

What happened?

The entire VBoxHeadless process crashed (segfault) about 2.5 minutes into a host-side I/O load test, while the guest's VDI was attached to an NVMe storage controller (VM configured with EFI firmware, ICH9 chipset):

kernel: NVMe#0[2861276]: segfault at 40 ip 00007fd31e0bcdfa sp 00007fd31c355cb0 error 4 in VBoxDD.so[7fd31df38000+310000]

The final lines of VBox.log show dozens of AsyncCompletion: Task 0x… completed after 10 seconds, then VD#0: Trying to cancel request 0x3000, then the process dies (no further output). Full VBox.log from the crashed run: https://gist.github.com/jeremyakers/36c34e852a9b48479a8b1889f49e97df

Expected: degraded I/O performance under load. Got: a null-pointer dereference (segfault at 40) in the NVMe#0 worker thread inside VBoxDD.so, killing the whole VM process.

Additional context: this looks like the NVMe-path manifestation of the cancel-path defect family tracked in #15374 / #9975 (AHCI "Port x reset" after canceled writes/flushes). On this same host, an earlier 7.2.x point release (package installed 2026-08-13) with the disk attached via SATA/AHCI twice produced that AHCI wedge: VD#0: Flush request was active for 26 secondsAHCI#0: Port 0 reset loop → guest block layer ends up with corrupted disk geometry ("access beyond end of device") and the guest wedges. After migrating the same VDI to an NVMe controller on 7.2.14, the same underlying trigger crashes the whole emulator instead of wedging the guest.

Host storage hardware is exonerated: per-second iostat captured during the crash window shows max flush latency of 269 ms across six Samsung 850/860 PRO SSDs (md RAID10 behind an LSI SAS3008 HBA) — no multi-second stalls at the block layer.

How can we reproduce this?

  1. Linux host. VM with an NVMe-attached VDI, host I/O cache off (default), guest under normal load.
  2. Generate heavy concurrent host I/O against the same filesystem that holds the VDI:
    • dd if=<the.vdi> of=/dev/null bs=1M iflag=direct (30 GB direct read of the live VDI)
    • dd if=/dev/zero of=<same-fs>/load.bin bs=1M count=10240 conv=fdatasync oflag=direct (10 GB synced write)
    • and inside the guest: dd if=/dev/zero of=/data/test.bin bs=1M count=10240 conv=fsync (10 GB)
  3. Within ~2–3 minutes: VBox.log shows a burst of AsyncCompletion: Task … completed after 10 seconds, then VD#0: Trying to cancel request 0x3000, then the segfault above.

Reproduced 1/1 on 7.2.14r174565 (host: 24 CPU / 94 GB RAM, Ubuntu 22.04, kernel 5.15.0-190-generic).

Control run: with VBoxManage storagectl "Home Assistant" --name NVMe --hostiocache on and host vm.dirty_bytes=2147483648 / vm.dirty_background_bytes=536870912 set, the identical load runs 5+ minutes with zero AsyncCompletion/cancel/error lines in VBox.log — consistent with a stall → timeout → cancel → crash chain.

No core dump was captured (process ulimit -c 0); happy to re-run with core dumps enabled and provide one if useful — the reproduction is quick and reliable on this host.

Did you upload all of your necessary log files, screenshots, etc.?

Yes — full VBox.log from the crashed run is linked via gist above; the kernel segfault excerpt is inline.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions