r/VFIO•
Solar_Sails

Nvidia Driver v376.33 Fails to Install - Windows Reverts back to Generic Display Driver

Decided to make another topic, as it wouldn't have been relevant to the last one I made.

System Specs:

  • CPU: Intel i7 6700k
  • RAM: 16GB DDR4
  • HDD: 1TB (Host) | 500GB Virtual QCOW for Guest
  • GPU: Intel HD 530 (Host) | EVGA GTX 970 (Guest)
  • Motherboard: MSI B150M Mortar
  • Host OS: Manjaro w/Kernel Version 4.4.39-1-MANJARO
  • Guest OS: Windows 10 Pro

Configuration:

  • iGPU for Host/GTX 970 for Guest
  • IOMMU Enabled
  • VFIO Enabled/Virtio Drivers installed on Guest
  • libvirt 2.4.0
  • virt-manager 1.4.0
  • Created virtual interface w/NAT in Virt-Manager for network to Guest
  • OVMF for UEFI capability on Guest
  • No ACS Patch installed on first attempt | ACS Patch being installed for 2nd attempt
  • Nvidia workarounds implemented in the /etc/libvirt/qemu/win10.xml file

I am running libvirt 2.4.0, virt-manager 1.4.0 and VFIO. VFIO drivers are installed on the Guest VM.

The issue I'm having at the moment is installing the 376.33 Nvidia drivers on Windows 10, Version 1607, Build 14393.576

The Nvidia installer doesn't throw up the usual errors that it would make if you didn't properly use the workaround. It installs just fine and then I get a notification on Windows 10 saying:

"Display Driver Updated: Please restart your system for full functionality"

I restarted, and it still uses the default, while the GPU is still showing up with an exclamation point in Device Manager. Has anyone encountered this or know a workaround for this? I will try playing with older driver versions, but I'm starting to think I am missing something on the host side. I've also checked Event Viewer, but I am finding no entries on failures from the Nvidia installer or from Windows about this issue.

For clarification, I am outputting from the GPU on the virtual machine, but it's stuck at 800x600 resolution without the ability to change the resolution at all (options are greyed out).

==========================================================================

Image of the Nvidia installer after it's finished installing the drivers (Notification about Display Driver Updated appears): http://imgur.com/oK2imyF

System Information: http://imgur.com/46f8X3v

Device Manager after driver install: http://imgur.com/Zl0rnAf

Display Driver Updated: http://imgur.com/NmuIKRl

Resolution Locked: http://i.imgur.com/sTIYst1.png

==========================================================================

EDIT: I looked at some settings on my host side on Manjaro, and found that there's a hybrid driver for Intel/Nvidia installed. Do you think this may be the culprit? http://imgur.com/iH2Zfi6

==========================================================================

EDIT: EventLogs recorded Nvidia driver install from start to finish https://u.teknik.io/Oc6Us.zip

I have also re-created the VM with host-passthrough flags for the CPU. Nothing has changed. Starting to wonder if I -really- need the ACS patch. Windows keeps over-riding the Nvidia installer with its own drivers.

========================================================================== EDIT I noticed that there are 3 devices in the IOMMU group for the GPU. Going to compile the ACS kernel patch and configure the bootloader.

IOMMU Group 1 00:01.0 PCI bridge [0604]: Intel Corporation Skylake PCIe Controller (x16) [8086:1901] (rev 07)
IOMMU Group 1 01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM204 [GeForce GTX 970] [10de:13c2] (rev a1)
IOMMU Group 1 01:00.1 Audio device [0403]: NVIDIA Corporation GM204 High Definition Audio Controller [10de:0fbb] (rev a1)

ACS PATCH FIXED THIS ISSUE! Explanation: Host still had control of the GPU with the PCIe Root device being in the same group as the GPU and HD Audio device, hence, it would NOT allow the drivers to take control of the GPU despite the Guest OS recognizing the GPU is there. In order to fix this, you NEED the ACS patch installed to segregate the PCIe Root from the same group as the GPU and its HD Audio Device.

Thank you all for the help!

Finished product: http://i.imgur.com/45ffYb2.png

Posted Jan 6, 2017, 9:50 AM UTC · edited

319
simcop2387•

It installed fine on my system, not sure what exactly is going to be different between us.

###Software Debian Sid
QEMU 2.8
libvirt 2.5.0

###System specs i5-4960K, overclocked to 4GHz
32B Ram
Geforce 1070
Windows 10 64bit.

I've got no drivers loaded of any kind on the linux side (nouveau or proprietary). You should be able to check if that driver is doing anything by doing lspci -v and looking for the driver in use to be vfio-pci (this is with the VM running)

02:00.0 VGA compatible controller: NVIDIA Corporation GP104 [GeForce GTX 1070] (rev a1) (prog-if 00 [VGA controller])
        Subsystem: eVga.com. Corp. GP104 [GeForce GTX 1070]
        Flags: bus master, fast devsel, latency 0, IRQ 33
        Memory at f6000000 (32-bit, non-prefetchable) [size=16M]
        Memory at e0000000 (64-bit, prefetchable) [size=256M]
        Memory at f0000000 (64-bit, prefetchable) [size=32M]
        I/O ports at d000 [size=128]
        Expansion ROM at f7000000 [disabled] [size=512K]
        Capabilities: <access denied>
        Kernel driver in use: vfio-pci
        Kernel modules: nouveau

02:00.1 Audio device: NVIDIA Corporation GP104 High Definition Audio Controller (rev a1)
        Subsystem: eVga.com. Corp. GP104 High Definition Audio Controller
        Flags: bus master, fast devsel, latency 0, IRQ 34
        Memory at f7080000 (32-bit, non-prefetchable) [size=16K]
        Capabilities: <access denied>
        Kernel driver in use: vfio-pci
        Kernel modules: snd_hda_intel

My XML file for libvirt, I do most of this through virt-manager so I don't know what some of it does exactly but at the least it might help you spot something. Take special care looking at 02:00:* in mine as those are the graphics card and it's hdmi audio being passed through. You might also need to check that you've got the vendor id set in <hyperv>

<!--
WARNING: THIS IS AN AUTO-GENERATED FILE. CHANGES TO IT ARE LIKELY TO BE
OVERWRITTEN AND LOST. Changes to this xml configuration should be made using:
  virsh edit win10-clone
or other application using the libvirt API.
-->

<domain type='kvm'>
  <name>win10-clone</name>
  <uuid>112939e7-96d8-4446-91ca-c723a2fda6df</uuid>
  <memory unit='KiB'>12582912</memory>
  <currentMemory unit='KiB'>12582912</currentMemory>
  <vcpu placement='static'>4</vcpu>
  <os>
    <type arch='x86_64' machine='pc-q35-2.6'>hvm</type>
    <loader readonly='yes' type='pflash'>/usr/share/OVMF/OVMF_CODE.fd</loader>
    <nvram>/var/lib/libvirt/qemu/nvram/win10_VARS.fd</nvram>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
      <vendor_id state='on' value='fucknvidia'/>
    </hyperv>
    <kvm>
      <hidden state='on'/>
    </kvm>
  </features>
  <cpu mode='host-passthrough'>
    <topology sockets='1' cores='2' threads='2'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='hypervclock' present='yes'/>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <pm>
    <suspend-to-mem enabled='yes'/>
    <suspend-to-disk enabled='yes'/>
  </pm>
  <devices>
    <emulator>/usr/bin/qemu-kvm</emulator>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/media/virts/Win10_1607_English_x64.iso'/>
      <target dev='sda' bus='sata'/>
      <readonly/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/mnt/media/virts/win10-clone.qcow2'/>
      <target dev='sdc' bus='scsi'/>
      <boot order='1'/>
      <address type='drive' controller='0' bus='0' target='0' unit='2'/>
    </disk>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pcie-root'/>
    <controller type='pci' index='1' model='dmi-to-pci-bridge'>
      <model name='i82801b11-bridge'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/>
    </controller>
    <controller type='pci' index='2' model='pci-bridge'>
      <model name='pci-bridge'/>
      <target chassisNr='2'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </controller>
    <controller type='sata' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </controller>
    <controller type='scsi' index='0' model='virtio-scsi'>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x01' function='0x0'/>
    </controller>
    <interface type='network'>
      <mac address='52:54:00:86:6e:e1'/>
      <source network='vmnetwork'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x04' function='0x0'/>
    </interface>
    <interface type='bridge'>
      <mac address='52:54:00:28:3f:6f'/>
      <source bridge='br0'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x05' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target port='0'/>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='spice' autoport='yes'>
      <listen type='address'/>
    </graphics>
    <sound model='ich6'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </sound>
    <video>
      <model type='cirrus' vram='16384' heads='1' primary='yes'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/>
    </video>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x02' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x02' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x03' function='0x0'/>
    </hostdev>
    <redirdev bus='usb' type='spicevmc'>
      <address type='usb' bus='0' port='2'/>
    </redirdev>
    <redirdev bus='usb' type='spicevmc'>
      <address type='usb' bus='0' port='3'/>
    </redirdev>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </memballoon>
  </devices>
</domain>
1
Solar_SailsOP•

lspci -v reports the following:

01:00.0 VGA compatible controller: NVIDIA Corporation GM204 [GeForce GTX 970] (rev a1) (prog-if 00 [VGA controller])
Subsystem: eVga.com. Corp. Device 3973
Flags: fast devsel, IRQ 16
Memory at de000000 (32-bit, non-prefetchable) [size=16M]
Memory at c0000000 (64-bit, prefetchable) [size=256M]
Memory at d0000000 (64-bit, prefetchable) [size=32M]
I/O ports at e000 [size=128]
Expansion ROM at df000000 [disabled] [size=512K]
Capabilities: <access denied>
Kernel driver in use: vfio-pci
Kernel modules: nouveau

01:00.1 Audio device: NVIDIA Corporation GM204 High Definition Audio Controller (rev a1)
Subsystem: eVga.com. Corp. Device 3973
Flags: fast devsel, IRQ 17
Memory at df080000 (32-bit, non-prefetchable) [size=16K]
Capabilities: <access denied>
Kernel driver in use: vfio-pci
Kernel modules: snd_hda_intel

So it seems as if it did bind correctly to VFIO. Where are you pulling your XML from btw? I've seem to have forgotten where the hell that's located. virsh cannot find it either when I use the list command.

I am also not sure how to make out the IOMMU groups to determine if they're not the way they're supposed to be or if it's okay the way it is.

IOMMU Group 0 00:00.0 Host bridge [0600]: Intel Corporation Skylake Host Bridge/DRAM Registers [8086:191f] (rev 07)
IOMMU Group 10 00:1f.0 ISA bridge [0601]: Intel Corporation Sunrise Point-H LPC Controller [8086:a148] (rev 31)
IOMMU Group 10 00:1f.2 Memory controller [0580]: Intel Corporation Sunrise Point-H PMC [8086:a121] (rev 31)
IOMMU Group 10 00:1f.3 Audio device [0403]: Intel Corporation Sunrise Point-H HD Audio [8086:a170] (rev 31)
IOMMU Group 10 00:1f.4 SMBus [0c05]: Intel Corporation Sunrise Point-H SMBus [8086:a123] (rev 31)
IOMMU Group 1 00:01.0 PCI bridge [0604]: Intel Corporation Skylake PCIe Controller (x16) [8086:1901] (rev 07)
IOMMU Group 1 01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM204 [GeForce GTX 970] [10de:13c2] (rev a1)
IOMMU Group 1 01:00.1 Audio device [0403]: NVIDIA Corporation GM204 High Definition Audio Controller [10de:0fbb] (rev a1)
IOMMU Group 2 00:02.0 VGA compatible controller [0300]: Intel Corporation HD Graphics 530 [8086:1912] (rev 06)
IOMMU Group 3 00:08.0 System peripheral [0880]: Intel Corporation Skylake Gaussian Mixture Model [8086:1911]
IOMMU Group 4 00:14.0 USB controller [0c03]: Intel Corporation Sunrise Point-H USB 3.0 xHCI Controller [8086:a12f] (rev 31)
IOMMU Group 4 00:14.2 Signal processing controller [1180]: Intel Corporation Sunrise Point-H Thermal subsystem [8086:a131] (rev 31)
IOMMU Group 5 00:15.0 Signal processing controller [1180]: Intel Corporation Sunrise Point-H Serial IO I2C Controller #0 [8086:a160] (rev 31)
IOMMU Group 5 00:15.1 Signal processing controller [1180]: Intel Corporation Sunrise Point-H Serial IO I2C Controller #1 [8086:a161] (rev 31)
IOMMU Group 6 00:16.0 Communication controller [0780]: Intel Corporation Sunrise Point-H CSME HECI #1 [8086:a13a] (rev 31)
IOMMU Group 7 00:17.0 SATA controller [0106]: Intel Corporation Sunrise Point-H SATA controller [AHCI mode] [8086:a102] (rev 31)
IOMMU Group 8 00:1d.0 PCI bridge [0604]: Intel Corporation Sunrise Point-H PCI Express Root Port #9 [8086:a118] (rev f1)
IOMMU Group 8 00:1d.1 PCI bridge [0604]: Intel Corporation Sunrise Point-H PCI Express Root Port #10 [8086:a119] (rev f1)
IOMMU Group 8 03:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 15)
IOMMU Group 9 00:1e.0 Signal processing controller [1180]: Intel Corporation Sunrise Point-H Serial IO UART #0 [8086:a127] (rev 31)
1
simcop2387•

/etc/libvirt/qemu is where my xmls are living

1
Solar_SailsOP•

Okay, here is my setup for the VM.

<!--
WARNING: THIS IS AN AUTO-GENERATED FILE. CHANGES TO IT ARE LIKELY TO BE
OVERWRITTEN AND LOST. Changes to this xml configuration should be made using:
  virsh edit win10
or other application using the libvirt API.
-->

<domain type='kvm'>
  <name>win10</name>
  <uuid>b01dad8a-cfb0-410c-bb44-eb01e15a77c8</uuid>
  <title>Windows 10</title>
  <description>GPU passthrough testing</description>
  <memory unit='KiB'>8388608</memory>
  <currentMemory unit='KiB'>8388608</currentMemory>
  <vcpu placement='static' current='1'>4</vcpu>
  <os>
    <type arch='x86_64' machine='pc-i440fx-2.8'>hvm</type>
    <loader readonly='yes' type='pflash'>/usr/share/ovmf/x64/ovmf_x64.bin</loader>
    <nvram>/var/lib/libvirt/qemu/nvram/win10_VARS.fd</nvram>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
	<vendor_id state='on' value='nvidiacansuckmydick'/>
    </hyperv>
	<kvm>
		<hidden state='on'/>
	</kvm>
    <vmport state='off'/>
  </features>
  <cpu mode='custom' match='exact'>
    <model fallback='allow'>Skylake-Client</model>
    <topology sockets='1' cores='4' threads='1'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
    <timer name='hypervclock' present='yes'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/sbin/qemu-system-x86_64</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/libvirt/images/win10.qcow2'/>
      <target dev='hda' bus='ide'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/var/lib/libvirt/images/virtio-win-0.1.126.iso'/>
      <target dev='hdb' bus='ide'/>
      <readonly/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='ide' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <interface type='network'>
      <mac address='52:54:00:5f:80:7a'/>
      <source network='NET_NAT_192'/>
      <model type='rtl8139'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target port='0'/>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <sound model='ich6'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </sound>
    <hostdev mode='subsystem' type='usb' managed='yes'>
      <source>
        <vendor id='0x045e'/>
        <product id='0x0768'/>
      </source>
      <address type='usb' bus='0' port='1'/>
    </hostdev>
    <hostdev mode='subsystem' type='usb' managed='yes'>
      <source>
        <vendor id='0x046d'/>
        <product id='0x0a29'/>
      </source>
      <address type='usb' bus='0' port='2'/>
    </hostdev>
    <hostdev mode='subsystem' type='usb' managed='yes'>
      <source>
        <vendor id='0x093a'/>
        <product id='0x2521'/>
      </source>
      <address type='usb' bus='0' port='3'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x01' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </hostdev>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </memballoon>
  </devices>
</domain>

In virt-manager, am I also supposed to add the PCI device through there or do I just add it as a Display with VFIO?

1
simcop2387•
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x01' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </hostdev>

So I see your host device in there, that looks good. But I also see that you've got your CPU defined differently.

  <cpu mode='custom' match='exact'>
    <model fallback='allow'>Skylake-Client</model>
    <topology sockets='1' cores='4' threads='1'/>
  </cpu>

Try setting it to 'host-passthrough' in virt-manager and see if that makes a difference? otherwise try calling it a core2duo while installing. I know that the windows 10 anniversary update has an issue with QEMU during install only if you don't call it a core2duo (why, i have no idea). The topology shouldn't matter, what you've got set will look like a quadcore with hyperthreading disabled.

1
Solar_SailsOP•

Also adding the image of the display issue here and the OP.

http://i.imgur.com/sTIYst1.png

EDIT After doing some research on the IOMMU groups, I noticed that the PCI Express controller is somehow getting thrown into the same group as the GPU and its HD Audio device. Could this also be an issue?

EDIT2 Going to install the ACS patch. This seems to point to the issue with the PCIe Root being in the same group, as the host still controls the GPU rather than the VM, which is also the cause of the driver never really being -installed- per say.

1
Mooiweer16•

I know you've already fixed it but, tried this?

Also the PCI bridge being in the IOMMU group is normal (and is required!). as stated here I have a near identical setup as yours(6700k, GTX 1070) but I didn't need to apply the ACS patch.

1
Solar_SailsOP•

I've pre-applied this patch.

1
kwhali•

Out of curiousity why were you running Bumblebee? I thought that was only intended for optimus laptops and served little benefit on desktop machines? Your issue sounds a bit odd with your solution.

Why NVIDIA drivers didn't install properly for my Win10 but fine for my Win8.1

TL;DR: Use the latest official ISO from Microsofts website.
Anyway, out of interest for you or others. I've had a similar issue specifically with Win 10, 8.1 worked perfectly. Turned out that the original ISO I had(Win 10 with updates pre-applied) had something shady about it and wouldn't install properly on UEFI VM. Tried an ISO of the Win 10 media from release which installed and updated fine but still refused my GPU drivers. Found out that Microsoft actuall supplies a download of their Win 10 OS as an ISO officially on their website, an updated version from their original release media. Drivers installed on this official/updated ISO no problem.


You didn't share how you handled the pass-through but mentioned that you had trouble with the host claiming your GPU which required ACS to let your VM attach it. Not sure how you approached this prior, thought I'd share what I do as an alternative that probably works just as well without ACS.

3 Simple steps to passthrough on the host side

  • Update grub with line below(with your hardware ids instead of course)
  • Uninstall NVIDIA drivers
  • Switch your primary GPU in BIOS

/etc/default/grub:

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash modprobe.blacklist=nouveau,nvidia,nvidia_drm intel_iommu=on iommu=pt modules-load=vfio,vfio_iommu_type1,vfio_pci_vfio_virqfd vfio-pci.ids=10de:1b81,10de:10f0"

That single line will blacklist drivers on my Manjaro host, enable IOMMU and handle the VFIO binding. No additional files required. It's important to remove the nvidia drivers from the host and replace them with the Intel iGPU drivers instead(Manjaro handles this by default by uninstalling NVIDIA drivers from memory). Other than that to switch I just need to change my primary GPU in the BIOS at boot.


Going to take a wild guess that your approach was one of the more outdated ways(I'm not sure if any guides actually mention the way I do it with a single line) or you made a slight mistake somewhere(did you passthrough both the GPU and it's audio device, or only the GPU?).

1
psyblade42•

Out of curiousity why were you running Bumblebee? I thought that was only intended for optimus laptops and served little benefit on desktop machines?

It is useful to (time)share the use of the dedicated GPU between the Host and the Guest.

1
kwhali•

I wasn't aware you could use bumblebee like that to allow GPU on host without reconfiguring/restarting :) It allows dGPU for games on host while monitor is plugged into iGPU? I thought it only worked like that on Laptops, good to know can use like this for desktops, will have to try it sometime.

1
psyblade42•

Yes.

Normally I run all programs completely on the iGPU. Demanding games can be rendered on the dGPU via bumblebee. They display on the iGPU output like any other program rendered there. No restarting X or any such nonsense. Alternatively I can spin up a Windows VM that uses the dGPU (output via dGPU unfortunately). While I can't use bumblebee and VM at the same time there is no reconfiguring or rebooting needed to switch. Simply quit one and start the other.

Of course this won't work with all the blacklisting/uninstalling/binding you do. Bumblebee needs the nvidia driver installed and loaded. Thus the only related kernel parameter I use is "intel_iommu=on" (well and "iommu=pt" to work around a broken pcie-sata-controller). Instead I let libvirt do its thing and take automatic care of all the vfio stuff whenever the VM is started/stopped.

1
kwhali•

Are you sure the driver needs to be loaded? I thought it was handled via the 2nd x server that bumblee uses for the dGPU. From memory with my laptop when trying out different setups, Bumblebee managed the NVIDIA driver, I couldn't have it installed when using my iGPU as the libGL packages conflicted, Bumblebee avoided that issue.

Pretty cool though, I've got alot of virtualization things to try out but hope to give this a look some day :)

1
Solar_SailsOP•

Anyway, out of interest for you or others. I've had a similar issue specifically with Win 10, 8.1 worked perfectly. Turned out that the original ISO I had(Win 10 with updates pre-applied) had something shady about it and wouldn't install properly on UEFI VM. Tried an ISO of the Win 10 media from release which installed and updated fine but still refused my GPU drivers. Found out that Microsoft actuall supplies a download of their Win 10 OS as an ISO officially on their website, an updated version from their original release media. Drivers installed on this official/updated ISO no problem.

This was the strange thing about it. After updating from Version 1511 to 1607, I was still having the same issue with the Nvidia driver not being installed, even though it explicitly would show up in Device Manager. After I removed the Bumblebee driver, I had tried it again and it was still producing the same issue. It wasn't until I installed the ACS patch to segregate the IOMMU groups that made it work as intended.

For example, before the ACS patch:

IOMMU Group 1 00:01.0 PCI bridge [0604]: Intel Corporation Skylake PCIe Controller (x16) [8086:1901] (rev 07)
IOMMU Group 1 01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM204 [GeForce GTX 970] [10de:13c2] (rev a1)
IOMMU Group 1 01:00.1 Audio device [0403]: NVIDIA Corporation GM204 High Definition Audio Controller [10de:0fbb] (rev a1)

After the ACS patch:

IOMMU Group 12 01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM204 [GeForce GTX 970] [10de:13c2] (rev a1)
IOMMU Group 12 01:00.1 Audio device [0403]: NVIDIA Corporation GM204 High Definition Audio Controller [10de:0fbb] (rev a1)
...
IOMMU Group 1 00:01.0 PCI bridge [0604]: Intel Corporation Skylake PCIe Controller (x16) [8086:1901] (rev 07)

I'm still not exactly sure why I had to have the PCI-e X16 controller in a separate IOMMU group for this to work.

Going to take a wild guess that your approach was one of the more outdated ways(I'm not sure if any guides actually mention the way I do it with a single line) or you made a slight mistake somewhere(did you passthrough both the GPU and it's audio device, or only the GPU?).

I used the guide from the Arch Wiki. https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF

Both the GPU and the audio device were passed through to the VM.

1
kwhali•

That guide doesn't blacklist at all, not sure if that would have made a difference for you. You do not need to split the IOMMU group like you did, that's why I think there is something odd and you didn't need to do the ACS patch. I have not tried with Bumblebee as I wasn't aware that could work.

My IOMMU group for my GPU that passed through to Win10 VM fine with drivers working:

IOMMU Group 1 00:01.0 PCI bridge [0604]: Intel Corporation Skylake PCIe Controller (x16) [8086:1901] (rev 07)
IOMMU Group 1 01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP104 [GeForce GTX 1070] [10de:1b81] (rev a1)
IOMMU Group 1 01:00.1 Audio device [0403]: NVIDIA Corporation GP104 High Definition Audio Controller [10de:10f0] (rev a1)

Did you use the official ISO(not sure what version it provides) from the MS website? If you sourced it elsewhere or the ISO is not recent it could be the cause. Not sure how you installed the driver, the official one found it when doing updates, I didn't have to download the driver from NVIDIA myself. I did need to have my 2 workarounds(kvm off and vendor id spoof), pretty sure until the driver is installed it shows up in Device Manager with the error triangle.

It was my third attempt/ISO. I also installed the OS from the passedthrough GPU on external monitor from the start this time, usually I use VNC and don't pass the GPU through until I've completed initial installation. No idea if that made a difference. Was using Q35 for chipset too.

1
muteor•

I had similar problems getting a EVGA GTX 660 ti working, turned out I needed to flash the card with EFI rom. Once this was done it worked right away.

I used rom-parser from here to determine that my card needed to be flashed: http://vfio.blogspot.co.uk/2014/08/does-my-graphics-card-rom-support-efi.html

Before:

root@bumblebee:~/rom-parser# ./rom-parser BACKUP.rom
Valid ROM signature found @400h, PCIR offset 190h
    PCIR: type 0 (x86 PC-AT), vendor: 10de, device: 1183, class: 030000
    PCIR: revision 0, vendor revision: 1
    Last image

After:

root@bumblebee:~/rom-parser# ./rom-parser BACKUP_NEW.rom
Valid ROM signature found @400h, PCIR offset 190h
    PCIR: type 0 (x86 PC-AT), vendor: 10de, device: 1183, class: 030000
    PCIR: revision 0, vendor revision: 1
Valid ROM signature found @10000h, PCIR offset 1ch
    PCIR: type 3 (EFI), vendor: 10de, device: 1183, class: 030000
    PCIR: revision 3, vendor revision: 0
        EFI: Signature Valid, Subsystem: Boot, Machine: X64
    Last image

You can dump the bios using NVFlash found here (also useful to backup): https://www.techpowerup.com/downloads/Utilities/BIOS_Flashing/NVIDIA/

./nvflash --index=0 --save BACKUP.rom

I had to enable UEFI settings in my bios after I had flashed it, without that the card wouldn't show up as a pci device in lspci.

Hopefully this helps someone else googling :)

1