Source
r/VFIO•
bubblethink

Short report (WIP): Got the NVIDIA GPU to passthrough on an optimus laptop

Since there haven't been any successful posts regarding this setup, I'm sharing this so that others can try and improve this. This should work on at least a few thinkpads.

Laptop: Thinkpad W530, Card: Quadro K2000M

What works (so far): Linux host, Linux guest with nouveau driver

Setup: Pretty standard. Luckily, the IOMMU group is isolated with only the GPU and a bridge in it. GPU can be passed through without too much trouble.

Main challenges:

  • GPU vbios isn't easily accessible in this setup since it's not on the card. It's a part of the system bios, which the driver is supposed to read through ACPI calls.
  • There are a few ways to extract it. I tried nouveau's debugfs interface first. It has the useful stuff that nouveau needs, but it is truncated. Eventually, a full bios with passing signatures will be needed. I've got the full rom too from a couple of other tools. PM me if you need the ROM.
  • Since the guest cannot read the rom on its own, it needs to be passed as a file. This can be done as the romfile option, or by baking it into initramfs on the guest, and adding a kernel boot arg "nouveau.config=NvBios=path-to-file". This is all that is needed to make passthrough work.
  • Next thing I tired was installing nvidia drivers on the guest. This fails even if you pass the pci rom because nvidia drivers know that this is an optimus card, and only try the acpi method to get the bios. They error out with a message like "Failed to load vbios into system memory" .

Next steps:

  • The BIOS+ACPI need to be patched so that the nvidia drivers on the guest can load properly. Coreboot has some documentation and builds regarding this stuff. I think that should be helpful.
  • Windows guest. I haven't tried this yet, but I highly doubt that it will work unless the linux guest with nvidia drivers works first.

Other quibbles:

  • libvirt on Ubuntu 16.04 has a bug that doesn't allow the rom file option. It errors out with a permissions error no matter what the permissions of the file are. Calling qemu directly works.
  • I haven't been able to get my mouse + keyboard to work in this setup. I'm pretty new to this stuff. What's the standard way of doing this ? I tried passing my external mouse and keyboard, but that didn't work. I was able to do some basic testing with synergy, which was crap. I'll probably never use that again.

Acknowledgements:

Thanks to imirkin_ on the nouveau irc for all the help.

Imgur

Posted Jul 28, 2017, 10:38 PM UTC · edited

3219
ermockler•

I got as far as GPU vbios, and didn't think it was worth gambling all the brain damage getting the rom on maybe having a system that probably wouldn't take drivers. Is this pretty repeatable if I were to get a laptop with a k200m? I tried on a Dell E6520, k200m sounds familiar. I don't see any thinkpad w30's on ebay.....

You could pass the usb devices individually, I'm used to doing this in xml & using libvirt.

If you were to install a new VM, booting from the install cd, do you see the output on the lcd, or is this only after you load drivers?

I'd be interested in the rom & a more detailed walkthrough.

This is very cool , good job!!

2
bubblethinkOP•

Sorry, there were a couple of typos. The laptop is W530 and the card is K2000M. Yes, Dell and HP also had a couple of models with the same card. This is an optimus setup with a hardware mux. So you can operate the laptop as integrated only, optimus, or discrete only. Modern laptops don't have the integrated only option. So I'm not sure if the GPU can be sufficiently detached that they can be passed through in newer laptops. This should be repeatable if you get a w530 with a k2000m. I would advise waiting though for more reports. I tried passing individual devices for keyboard and mouse in virt-manager, but that didn't work. However, my peripherals were connected to the dock. Maybe that plays a role. Since this was just proof of concept, I didn't spend too much time on it.

2
ermockler•

Those I saw on ebay. I think the dell had only 2 options in the bios. Does the vm play out of laptop lcd or do you need the monitor?

That's an old ass keyboard.

1
bubblethinkOP•

Yes, the keyboard is an IBM model M from the 80s. The vm needs the monitor. This laptop's wiring is a bit peculiar. The igpu is connected to the lcd. All the external ports are connected to the nvidia gpu. If you want to use the discrete gpu on the lcd, it writes to the igpu's framebuffer, which then shows it on the lcd. This has been a source of headache for years on linux. Things are getting a bit better now though. For the passthrough case, you completely decouple the igpu and the discrete gpu. So igpu only drives the lcd, and the discrete one only drives external ports.

3
IRanOutOfNicknames•

Oh shit, so Nouveau works. I got as far as the nvidia driver on a Windows VM, extracted my vbios and passed it as -romfile but gave up when I couldn't get it to work no matter what. I'm on a non-quadro, will try nouveau when I get home.

Some things I tried: 1. GVT-g. When everything is set up properly, you still can't install the driver, and unpacking and manually installing it doesn't work either because you can't load it. Also, the code is utter brain damage. 2. Unpacking the driver and manually installing it on a standard passthrough configuration. Doesn't load. 3. Unpacking the driver and manually installing in a configuration in which both Intel and Nvidia GPUs are passed through. Didn't do this with romfile=. Doesn't load.

Question: Did you have to fiddle with nouveau in any way to get it working? If so, what exactly did you do?

1
bubblethinkOP•

No changes to nouveau. You'll have to look at dmesg output to see if something isn't working. The quadro part isn't very important for linux and nouveau. Even for windows, only desktop quadros make some difference regarding driver checking I guess. If nouveau isn't able to read the vbios, it'll say so in dmesg. If everything works, make sure to remove all other vga adapters like spice and qxl. That troubled me for a while. Nouveau was working all along, but monitor wouldn't light up because some other adapter was the primary.

3
dwe11er•

libvirt on Ubuntu 16.04 has a bug that doesn't allow the rom file option. It errors out with a permissions error no matter what the permissions of the file are. Calling qemu directly works.

Probably AppArmor is blocking access to that file. Try putting it inside /var/lib/libvirt/images or something accessible for protected process.

1
bubblethinkOP•

I thought so too, but it doesn't look like an apparmor issue. I tried quite a few things, including disabling apparmor and making the file 777. I think it's a bug.

1
kwhali•

Windows guest. I haven't tried this yet, but I highly doubt that it will work unless the linux guest with nvidia drivers works first.

So no real progress then as that's what I believe everyone who's attempted this has been trying to do :P I have one of the first threads on this sub about doing just this, had my vBIOS and all that but Windows would be reluctant to install the driver still. Sounds like some interesting finds you came across though!

1
bubblethinkOP•

Windows should work too once you patch the BIOS to include the vbios rom. The path is pretty straightforward. There's one hook where the nvidia drivers look for the rom. If you make that call succeed, they should work. I'll attempt it some other weekend. Whether windows works or not is kinda secondary though. The hardware allows pass-through, which I don't think was a known thing. I don't think there are too many laptops where this would work. Whether nvidia drivers intentionally break this is something beyond our control. I'm less interested in circumventing their checks.

1
jscinoz•

Hey, I know this is a rather old post, but I've been having a go at all this myself over the past week, and I've gotten as far as getting the Linux binary Nvidia driver to work in the guest, by way of custom ACPI table I wrote that implements the _ROM method reading from the PCI option ROM.

I've not yet managed to get it to work with a Windows guest (Windows seems to be clearing the PCI XROMBAR, so by the time the _ROM method is called, I get a nonsense address causing a BSOD), nor with an OVMF/EFI Linux guest (SeaBIOS works okay), but it does at least seem to prove the general principal. I'm not too sure what exactly to try next (other than patching OVMF to implement the _ROM method itself and to keep PCI option ROMs around in memory somewhere), but I thought I'd share what I'd found so far. There's more details over here, along with the ASL source for the custom ACPI table I mentioned.

Here's hoping someone more skilled in low level stuff than I can pick up the torch :)

2
bubblethinkOP•

Hey. It's great that you got binary drivers to work on linux. I didn't get a chance to revisit this after the original post. Which laptop did you try this on ? I saw your github link, and it mentions muxless laptops. The laptop in this thread (w530) has a mux, for which now there are more guides, including the one you linked on your github page. I'm not sure how those guides deal with the issue of passing the rom file to the binary drivers. That part shouldn't depend on mux v/s muxless, right ?

1
edbch•

Hi jscinoz

I wonder if you could help me to get working the ASL source for the ACPI table you provided in the github.

I have a XPS 15Z L511Z laptop with an Nvidia GT525m

01:00.0 VGA compatible controller [0300]: NVIDIA Corporation 
GF108M [GeForce GT 525M] [10de:0df5] (rev a1)

that has no support for uefi.

Valid ROM signature found @0h, PCIR offset 188h
PCIR: type 0 (x86 PC-AT), vendor: 10de, device: 0df5, class: 
030000
PCIR: revision 0, vendor revision: 1
Last image

Apparently the proprietary driver for this board relies only on acpi to get the rom as you described. No matter what rom I pass through kvm rom file, it always gives the same error in dmesg

NVRM: failed to copy vbios to system memory.

And

cat /proc/driver/nvidia/gpus/0000\:02\:06.0/information
Model:          GeForce GT 525M
IRQ:            30
GPU UUID:      GPU-????????-????-????-????-????????????
Video BIOS:      ??.??.??.??.??
Bus Type:      PCI
DMA Size:      36 bits
DMA Mask:      0xfffffffff
Bus Location:      0000:02:06.0
Device Minor:      0

, but for the nouveau it works (with the correct rom).

I would like to have cuda working inside the guest, that is all.

I saw by the guest's xml that you posted in the github that you use the same pci address from the host in the guest. I read somewhere, I can not find the reference right now, that it is not a good idea. Do you have a specific reason for this? Finally, when I try to compile the asl file you provided I get

make
iasl ssdt1.asl

Intel ACPI Component Architecture
ASL+ Optimizing Compiler version 20160108-64
Copyright (c) 2000 - 2016 Intel Corporation

Compiler aborting due to parser-detected syntax error(s)
ssdt1.asl     47:             for (Local2 = 0, Local2 < Local0,                 Local2++) {
Error    6126 -                             syntax error, unexpected '{' ^

ssdt1.asl     51:             Return (Local1)
Error    6126 -                   ^ syntax error, unexpected PARSEOP_RETURN

ssdt1.asl    196:             For (Local4 = 0, Local4 < Local2, Local4++) {
Error    6126 -                             syntax error, unexpected '{' ^

ssdt1.asl    212:
Error    6126 - syntax error, unexpected PARSEOP_METHOD,         
expecting $end and premature End-Of-File

ASL Input:     ssdt1.asl - 212 lines, 8464 bytes, 121 keywords

Compilation complete. 4 Errors, 0 Warnings, 0 Remarks, 0                     
Optimizations
Makefile:7: recipe for target 'ssdt1.aml' failed
make: *** [ssdt1.aml] Error 255

I have zero experience with ACPI source language, so could you please point me out to the relevant documentation?

Thanks

Ps: sorry my terrible english.

1
jscinoz•

If you grab the latest SSDT from my repository, it should compile just fine. You will need to change the name of the rom file within (https://github.com/jscinoz/optimus-vfio-docs/blob/master/asl/ssdt1.asl#L252) to match that of the one you're providing to qemu.

I have also noticed that you need rombar to be off, while still providing the romfile option, to cause qemu to include it in the genroms directory in fw_cfg.

I used the same PCI address in the guest as the host, as I figured I should try match the host layout as closely as possible. What have you read suggesting against that? Can you please provide a link?

Here's the ACPI spec which specifies ASL too - this is what I worked off to learn enough to get to where we are now.

1
kwhali•

I'll attempt it some other weekend.

If you have success with it, do share here :)

The hardware allows pass-through, which I don't think was a known thing. I don't think there are too many laptops where this would work.

I didn't try with a linux guest, but made about the same progress as you describe with windows, just no patching so driver install fails. I had passed the vbios rom and such though.

Windows should work too once you patch the BIOS to include the vbios rom.

That's the VM bios to patch right not the host systems? Interesting find :)

1
bubblethinkOP•

just no patching so driver install fails.

How is your laptop wired ? Is the nvidia card directly connected to external ports ? That's really important for having a practical solution. All laptops have different wiring for optimus. If the nvidia card isn't directly connected to the external port, even if you pass it through, you won't be able to connect it to a monitor.

That's the VM bios to patch right not the host systems? Interesting find :)

Yes, the BIOS and the ACPI calls that you present to the guest need to be patched so that when the driver looks for the rom, it finds them.

1
kwhali•

That's really important for having a practical solution. All laptops have different wiring for optimus.

Not sure, thought they all routed output through the iGPU? I remember trying all the outputs. I think Pascal 1xxx and newer in laptops are no longer using mobile variants so newer laptops might not do the whole iGPU shared framebuffer thing? My laptop is from 2014 MSI with an 860m and 4th gen(Haswell?) Intel i7, obviously not enough to go by I guess :P It cannot do vGPU with Intel sadly.

Yes, the BIOS and the ACPI calls that you present to the guest need to be patched so that when the driver looks for the rom, it finds them.

Is that something generic? Or could be made generic that maybe it could be patched upstream if it doesn't affect non-optimus users.

1